The fundamental purpose of testing—and, for that matter, of all software quality assurance (QA) deliverables and processes — is to tell you just what you’ve built and whether it does what you think it should do. This is essential, because you can’t inspect a software program the same way you can inspect a house or a car. You can’t touch it, you can’t walk around it, you can’t open the hood or the bedroom door to see what’s inside, you can’t take it out for spin. There are very few tangible or visible clues to the completeness and reliability of a software system — and so we have to rely on QA activities to tell us how well built the system is.
Furthermore, almost any software system developed nowadays for production is vastly more complex than a house or car — it’s more on the same order of complexity of a large petrochemical processing and storage facility, with thousands of possible interconnections, states, and processes. We would be (rightly) terrified if, say, Exxon build such a sprawling oil refining complex near our neighborhood and then started up production having only done a bare minimum of inspection, testing, and trial operations before, during and after construction, offering the explanation that they would wait until after the plant went into production and then handle problems as they crop up. Yet too often that’s just how large software development projects are run, even though the system in development may well be more complex (in terms of connections, processes, and possible states) than such a petrochemical factory. And while most inadequately tested software systems won’t spew pollutants, poison the neighborhood, catch fire, or explode, they can cripple corporate operations, lose vast sums of money, spark shareholder lawsuits, and open the corporation’s directors and officers to civil and even criminal liability (particularly with the advent of Sarbanes-Oxley).
And that presumes that the system can actually go into production. The software engineering literature and the trade press are replete with well-documented case studies of “software runaways”: large IT re-engineering or development projects that consume tens or hundreds of millions of dollars, or in a few spectacular (government) cases, billions of dollars, over a period of years, before grinding to a halt and being terminated without ever having put a usable, working system into production. So it’s important not to skimp on testing and the other QA-related activities.
Bruce F. Webster, “Obamacare and the Testing Gap”, And Still I Persist…, 2013-10-31
November 4, 2013
QotD: Software quality assurance
Living in a Surveillance State: Mikko Hypponen at TEDxBrussels
November 1, 2013
Let’s hope badBIOS is an elaborate Halloween hoax
Dan Goodin posted a scary Halloween tale at Ars Technica yesterday … at least, I’m hoping it’s just a scary story for the season:
In the intervening three years, Ruiu said, the infections have persisted, almost like a strain of bacteria that’s able to survive extreme antibiotic therapies. Within hours or weeks of wiping an infected computer clean, the odd behavior would return. The most visible sign of contamination is a machine’s inability to boot off a CD, but other, more subtle behaviors can be observed when using tools such as Process Monitor, which is designed for troubleshooting and forensic investigations.
Another intriguing characteristic: in addition to jumping “airgaps” designed to isolate infected or sensitive machines from all other networked computers, the malware seems to have self-healing capabilities.
“We had an air-gapped computer that just had its [firmware] BIOS reflashed, a fresh disk drive installed, and zero data on it, installed from a Windows system CD,” Ruiu said. “At one point, we were editing some of the components and our registry editor got disabled. It was like: wait a minute, how can that happen? How can the machine react and attack the software that we’re using to attack it? This is an air-gapped machine and all of a sudden the search function in the registry editor stopped working when we were using it to search for their keys.”
Over the past two weeks, Ruiu has taken to Twitter, Facebook, and Google Plus to document his investigative odyssey and share a theory that has captured the attention of some of the world’s foremost security experts. The malware, Ruiu believes, is transmitted though USB drives to infect the lowest levels of computer hardware. With the ability to target a computer’s Basic Input/Output System (BIOS), Unified Extensible Firmware Interface (UEFI), and possibly other firmware standards, the malware can attack a wide variety of platforms, escape common forms of detection, and survive most attempts to eradicate it.
But the story gets stranger still. In posts here, here, and here, Ruiu posited another theory that sounds like something from the screenplay of a post-apocalyptic movie: “badBIOS,” as Ruiu dubbed the malware, has the ability to use high-frequency transmissions passed between computer speakers and microphones to bridge airgaps.
October 31, 2013
A garage of historical significance
In The Register, a remarkably blasé report on the designation of the house where Jobs and Wozniak created the first Apple computers:
The house where Steve Jobs built his first computers has been added to a list of historic buildings in Los Altos.
The Los Altos Historical Commission voted unanimously to add the home at 2066 Crist Drive as a historic resources, since its hallowed garage was where Jobs made his first computers and co-founded Apple, the San Jose Mercury News reported.
The commission’s report said that it had been reviewing the property for potential designation for the past two years due to its “association with an event and an individual of historic significance”.
From other discussion on the topic, this will require the current owner of the property (Patricia Jobs, the sister of the late Steve Jobs) to get the commission’s advance permission to do any kind of work on the house … including ordinary maintenance. No funds from the municipality go along with this designation: once your house has been so designated, you no longer exercise full rights of ownership, but you still are required to pay for any work the commission deems necessary or desirable. Ms Jobs apparently still has a right to appeal, but I don’t know what her chances of success might be.
The drug-running submarine squadron
Strategy Page reports that the set of almost-complete submarines built by a drug cartel in Colombia were much more sophisticated and capable than first thought:
The leader (Mauner Mahecha) of the project was a guy in his early 30s with no boat building experience but excellent organizational and leadership skills.
Mahecha had a huge budget and used it to find and hire men with the needed skills or experience with submarines. Mahecha also quickly recruited additional specialists as needed and obtained whatever materials the builders called for. His project built three submarines, and the project was shut down because one of the men recruited (an experienced engine mechanic working for the Colombian Navy) managed to tip off the Colombian Navy intelligence and then the U.S. about the project.
[…]
The Mahecha submarines, when closely examined by experts, turned out to be more sophisticated than first thought. The outer hull was made out of strong, lightweight, Kevlar/carbon fiber that was sturdy enough to keep the sub intact but very difficult to detect with most sensors. The hull could not survive deep dives but this boat didn’t have to go deep to get the job done. The diesel-electric power supply (up to two-hundred and forty-nine lead-acid batteries), diving and surfacing system, and navigational systems of captured subs were all in working order. Those who built these boats apparently borrowed much from recreational subs. The sub builders also had impressive knowledge of the latest materials used to build exotic boats.
The three fiberglass/Kevlar submarines were obviously built to transport cocaine to North America and the existence of a building effort had been detected by intel agencies. For several years before the submarine boat yard was discovered the U.S. Navy, in cooperation with some Central and South American navies, have been looking for these subs, at sea and on land. While these submarines didn’t run very deep (less than twenty meters/sixty-two feet), they are invisible to most sensors when completely submerged. These subs were designed to run on batteries for up to eighteen hours, before having to surface and recharge. When they are at sea, they usually operate their diesel engines. These are noisy. Sonar can pick up this noise over a long distance. By capturing these subs it was possible to run the engines and get a sound profile of this type of boat and equip American sonar systems with this data. These subs had a range (on internal fuel) of about twelve-thousand kilometers. Thus, the boat could get from Colombia to southern California and back. These drug gangs spent over two million on each of these subs.
The most potent weapon the U.S. Navy has against these tiny (less than thirty-four meters/one-hundred foot long) subs is heat sensors, but even that may have had limited effectiveness. That’s because one of the subs captured had a snorkel type device (a tall structure extending from the conning tower, which contained pipes allowing diesel exhaust to escape and fresh air to be brought into the submerged boat.) It’s this heat that airborne sensors can detect. All surface (or semi-submerged) ships at sea display this kind of “heat signature”, and capturing working examples of these cocaine smuggling subs makes it possible to get a better idea of what the airborne heat sensors should be looking for. A snorkel, however, puts out less heat that a sub running on the surface would and is harder to detect. When running on batteries there is no heat to detect.
October 29, 2013
What happens when you challenge hackers to investigate you?
Adam Penenberg had himself investigated in the late 1990s and wrote that up for Forbes. This time around, he asked Nick Percoco to do the same thing, and was quite weirded out by the experience:
It’s my first class of the semester at New York University. I’m discussing the evils of plagiarism and falsifying sources with 11 graduate journalism students when, without warning, my computer freezes. I fruitlessly tap on the keyboard as my laptop takes on a life of its own and reboots. Seconds later the screen flashes a message. To receive the four-digit code I need to unlock it I’ll have to dial a number with a 312 area code. Then my iPhone, set on vibrate and sitting idly on the table, beeps madly.
I’m being hacked — and only have myself to blame.
Two months earlier I challenged Nicholas Percoco, senior vice president of SpiderLabs, the advanced research and ethical hacking team at Trustwave, to perform a personal “pen-test,” industry-speak for “penetration test.” The idea grew out of a cover story I wrote for Forbes some 14 years earlier, when I retained a private detective to investigate me, starting with just my byline. In a week he pulled up an astonishing amount of information, everything from my social security number and mother’s maiden name to long distance phone records, including who I called and for how long, my rent, bank accounts, stock holdings, and utility bills.
[…]
A decade and a half later, and given the recent Edward Snowden-fueled brouhaha over the National Security Agency’s snooping on Americans, I wondered how much had changed. Today, about 250 million Americans are on the Internet, and spend an average of 23 hours a week online and texting, with 27 percent of that engaged in social media. Like most people, I’m on the Internet, in some fashion, most of my waking hours, if not through a computer then via a tablet or smart phone.
With so much of my life reduced to microscopic bits and bytes bouncing around in a netherworld of digital data, how much could Nick Percoco and a determined team of hackers find out about me? Worse, how much damage could they potentially cause?
What I learned is that virtually all of us are vulnerable to electronic eavesdropping and are easy hack targets. Most of us have adopted the credo “security by obscurity,” but all it takes is a person or persons with enough patience and know-how to pierce anyone’s privacy — and, if they choose, to wreak havoc on your finances and destroy your reputation.
H/T to Terry Teachout for the link.
Obamacare’s technical issues
A comment at Marginal Revolution deservedly has been promoted to being a guest post, discussing the scale of the problems with the Obamacare software:
The real problems are with the back end of the software. When you try to get a quote for health insurance, the system has to connect to computers at the IRS, the VA, Medicaid/CHIP, various state agencies, Treasury, and HHS. They also have to connect to all the health plan carriers to get pre-subsidy pricing. All of these queries receive data that is then fed into the online calculator to give you a price. If any of these queries fails, the whole transaction fails.
Most of these systems are old legacy systems with their own unique data formats. Some have been around since the 1960′s, and the people who wrote the code that runs on them are long gone. If one of these old crappy systems takes too long to respond, the transaction times out.
[…]
When you even contemplate bringing an old legacy system into a large-scale web project, you should do load testing on that system as part of the feasibility process before you ever write a line of production code, because if those old servers can’t handle the load, your whole project is dead in the water if you are forced to rely on them. There are no easy fixes for the fact that a 30 year old mainframe can not handle thousands of simultaneous queries. And upgrading all the back-end systems is a bigger job than the web site itself. Some of those systems are still there because attempts to upgrade them failed in the past. Too much legacy software, too many other co-reliant systems, etc. So if they aren’t going to handle the job, you need a completely different design for your public portal.
A lot of focus has been on the front-end code, because that’s the code that we can inspect, and it’s the code that lots of amateur web programmers are familiar with, so everyone’s got an opinion. And sure, it’s horribly written in many places. But in systems like this the problems that keep you up at night are almost always in the back-end integration.
The root problem was horrific management. The end result is a system built incorrectly and shipped without doing the kind of testing that sound engineering practices call for. These aren’t ‘mistakes’, they are the result of gross negligence, ignorance, and the violation of engineering best practices at just about every step of the way.
October 28, 2013
Mark Steyn on the Obamacare software
Mark Steyn’s weekend column touched on some items of interest to aficionados of past government software fiascos:
The witness who coughed up the intriguing tidbit about Obamacare’s exemption from privacy protections was one Cheryl Campbell of something called CGI. This rang a vague bell with me. CGI is not a creative free spirit from Jersey City with an impressive mastery of Twitter, but a Canadian corporate behemoth. Indeed, CGI is so Canadian their name is French: Conseillers en Gestion et Informatique. Their most famous government project was for the Canadian Firearms Registry. The registry was estimated to cost in total $119 million, which would be offset by $117 million in fees. That’s a net cost of $2 million. Instead, by 2004 the CBC (Canada’s PBS) was reporting costs of some $2 billion — or a thousand times more expensive.
Yeah, yeah, I know, we’ve all had bathroom remodelers like that. But in this case the database had to register some 7 million long guns belonging to some two-and-a-half to three million Canadians. That works out to almost $300 per gun — or somewhat higher than the original estimate for processing a firearm registration of $4.60. Of those $300 gun registrations, Canada’s auditor general reported to parliament that much of the information was either duplicated or wrong in respect to basic information such as names and addresses.
Sound familiar?
Also, there was a 1-800 number, but it wasn’t any use.
Sound familiar?
So it was decided that the sclerotic database needed to be improved.
Sound familiar?
But it proved impossible to “improve” CFIS (the Canadian Firearms Information System). So CGI was hired to create an entirely new CFIS II, which would operate alongside CFIS I until the old system could be scrapped. CFIS II was supposed to go operational on January 9, 2003, but the January date got postponed to June, and 2003 to 2004, and $81 million was thrown at it before a new Conservative government scrapped the fiasco in 2007. Last year, the government of Ontario canceled another CGI registry that never saw the light of day — just for one disease, diabetes, and costing a mere $46 million.
But there’s always America! “We continue to view U.S. federal government as a significant growth opportunity,” declared CGI’s chief exec, in what would also make a fine epitaph for the republic. Pizza and Mountain Dew isn’t very Montreal, and on the evidence of three years of missed deadlines in Ontario and the four-year overrun on the firearms database CGI don’t sound like they’re pulling that many all-nighters. Was the government of the United States aware that CGI had been fired by the government of Canada and the government of Ontario (and the government of New Brunswick)? Nobody’s saying. But I doubt it would make much difference.
October 25, 2013
The glamour of big IT projects
Virginia Postrel on the hubris of the Obamacare project team:
The HealthCare.gov website is a disaster — symbolic to Obamacare opponents, disheartening to supporters, and incredibly frustrating to people who just need to buy insurance. Some computer experts are saying the only way to save the system is to scrap the current bloated code and start over.
Looking back, it seems crazy that neither the Barack Obama administration nor the public was prepared for the startup difficulties. There’s no shortage of database experts willing to opine on the complexities of the problem. Plenty of companies have nightmarish stories to tell about much simpler software projects. And reporting by the New York Times finds that the people involved with the system knew months ago that it was in serious trouble. “We foresee a train wreck,” one said back in February.
So why didn’t the administration realize that integrating a bunch of incompatible government databases into a seamless system with an interface just about anyone could understand was a really, really hard problem? Why was even the president seemingly taken by surprise when the system didn’t work like it might in the movies?
We have become seduced by computer glamour.
Whether it’s a television detective instantly checking a database of fingerprints or the ease of Amazon.com’s “1-Click” button, we imagine that software is a kind of magic — all the more so if it’s software we’ve never actually experienced. We expect it to be effortless. We don’t think about how it got there or what its limitations might be. Instead of imagining future technologies as works in progress, improving over time, we picture them as perfect from day one.
October 13, 2013
Stross – Microsoft Word delenda est
As a writer, Charles Stross hates, hates, hates, hates, hates Microsoft Word and wants it to DIE:
Microsoft Word is a tyrant of the imagination, a petty, unimaginative, inconsistent dictator that is ill-suited to any creative writer’s use. Worse: it is a near-monopolist, dominating the word processing field. Its pervasive near-monopoly status has brainwashed software developers to such an extent that few can imagine a word processing tool that exists as anything other than as a shallow imitation of the Redmond Behemoth. But what exactly is wrong with it?
I’ve been using word processors and text editors for nearly 30 years. There was an era before Microsoft Word’s dominance when a variety of radically different paradigms for text preparation and formatting competed in an open marketplace of ideas. One early and particularly effective combination was the idea of a text file, containing embedded commands or macros, that could be edited with a programmer’s text editor (such as ed or teco or, later, vi or emacs) and subsequently fed to a variety of tools: offline spelling checkers, grammar checkers, and formatters like scribe, troff, and latex that produced a binary page image that could be downloaded to a printer.
These tools were fast, powerful, elegant, and extremely demanding of the user. As the first 8-bit personal computers appeared (largely consisting of the Apple II and the rival CP/M ecosystem), programmers tried to develop a hybrid tool called a word processor: a screen-oriented editor that hid the complex and hostile printer control commands from the author, replacing them with visible highlight characters on screen and revealing them only when the user told the program to “reveal codes”. Programs like WordStar led the way, until WordPerfect took the market in the early 1980s by adding the ability to edit two or more files at the same time in a split screen view.
October 11, 2013
Creating an “air gap” for computer security
Bruce Schneier explains why you’d want to do this … and how much of a pain it can be to set up and work with:
Since I started working with Snowden’s documents, I have been using a number of tools to try to stay secure from the NSA. The advice I shared included using Tor, preferring certain cryptography over others, and using public-domain encryption wherever possible.
I also recommended using an air gap, which physically isolates a computer or local network of computers from the Internet. (The name comes from the literal gap of air between the computer and the Internet; the word predates wireless networks.)
But this is more complicated than it sounds, and requires explanation.
Since we know that computers connected to the Internet are vulnerable to outside hacking, an air gap should protect against those attacks. There are a lot of systems that use — or should use — air gaps: classified military networks, nuclear power plant controls, medical equipment, avionics, and so on.
Osama Bin Laden used one. I hope human rights organizations in repressive countries are doing the same.
Air gaps might be conceptually simple, but they’re hard to maintain in practice. The truth is that nobody wants a computer that never receives files from the Internet and never sends files out into the Internet. What they want is a computer that’s not directly connected to the Internet, albeit with some secure way of moving files on and off.
He also provides a list of ten rules (or recommendations, I guess) you should follow if you want to set up an air-gapped machine of your own.
October 7, 2013
CSEC’s sudden media prominence … in Brazil
If you haven’t heard of CSEC before, you’re certainly not alone. The signals intelligence service known as Communications Security Establishment Canada has been eager not to be in the public eye, but allegations are being made that CSEC has been spying on the Brazilian government’s mining and energy ministry:
The impact for Canada of these revelations could be equally grave: they come at a time when Brazil has become a top destination for Canadian exports, when a stream of delegations from the oil and gas industries are making pilgrimages to Rio de Janeiro to try to get a piece of the booming offshore oil industry, and when the Canadian government is eager to burnish ties with Brasilia. Foreign Affairs Minister John Baird visited Brazil in August, and spoke repeatedly about the country as a critical partner for Canadian business.
[…]
While CSEC’s role in conducting economic espionage has been alluded to before, how it does this job has not. The significance of the documents obtained by Globo in Brazil is that they speak to how “metadata” analysis by CSEC can be used to exploit a rival country’s computer systems.
The CSEC-labeled slides about the “Olympia” program describe the “Brazilian Ministry of Mines and Energy” as a “new target to develop” despite “limited access/target knowledge.”
The presentation goes on to map out how an individual’s smartphone — “target’s handset” — can be discerned by analysis, including by cross-referencing the smartphone’s Sim card with the network telephone number assigned to it and also to the handset’s unique number (IMEI).
The “top secret” presentation also refers to attacks on email servers.
“I have identified MX [email] servers which have been targeted to passive collection by the Intel analysts,” one slide says, without explaining who the speaker is.
October 5, 2013
Reckoning military strength is more complex than counting tanks, ships, or noses
Sir Humphrey is back from his honeymoon and posts about the unfailing media habit of nostalgically looking back at the military of the Cold War era and contrasting it with the much smaller military of today:
When one looks back over the last 150 years, the possession of large military forces by the UK has been somewhat of an aberration. If you ignore WW1 & WW2, then the only period in which large forces were sustained was from 1945 until the end of the Cold War. This could only be done by relying firstly on large numbers of conscripts, then having to provide very low pay after the end of National Service. It is telling that once military wages began to catch up with, then overtake civilian roles, manpower quickly became increasingly unaffordable. Similarly it is easy to forget that this period is one of the very few in UK history where there was a clearly defined opponent, where UK forces had a clear role to play (e.g. maintain BAOR, defend the home base, conduct ASW) as well as support wider non NATO commitments. It is much easier to justify the retention of larger armed forces when you have a specific role in mind for them, and not just being held at readiness as a contingency.
In the UK we are perhaps guilty of looking back on the Cold War period as halcyon era where we had large armed forces, while forgetting that they existed to do very specific roles, and also encourage other nations to pull their weight too. The post Cold War era wasn’t some wonderful period where UK forces roamed the globe in glorious isolation emulating Palmerston’s views, but a period when the UK had to contribute to an international coalition and work with our partners against a common enemy. This is important to remember, for the argument that 30 years ago we had X frigates, Y jets and Z tanks compared to today’s paltry number is actually misleading. In reality much of this equipment was fully committed to NATO forces, and wasn’t easily available to support wider UK national interests beyond the NATO area. So yes, the UK had capabilities, but they were borne to meet a specific external threat, and not a general role.
Similarly, if one looks at availability, it becomes clear that in real terms UK capability for purely national tasks now isn’t far off what it was at the end of the Cold War. Speaking to a Naval friend who joined in the late 1980s, he pointed out that of the 47 escorts when they joined, nearly a third were usually tied up in refit. Add to this the tasking and working up of escorts for things like NATO commitments, and support to the South Atlantic, and suddenly that’s the best part of another 15 escorts committed. At best there would be a margin of some 10-15 hulls available for national discretionary deployments — not much more than is available today.
Yes, yes, but what about tiny Obscuristan with their 500 tanks? Britain is much bigger than Obscuristan, shouldn’t the British army have more tanks than them? And Fantasia has more ships in their navy than the Royal Navy does!
It is also important to realise when looking at these sorts of papers that nations have very different defence requirements. It is one thing to say we have less soldiers than say, South Korea, but we forget that we do not have a nuclear armed neighbour on our border with a leader who is not always a completely rational actor. It is entirely logical that some nations will have more military personnel than the UK — they have direct ground threats, or their need for manpower for other jobs means it is politically helpful to keep a large army to hand. For instance many states still conscript their troops, meaning on paper their army is vastly larger than the British Army, but this is only achieved through a ready pool of manpower who can be paid a pittance and employed on duties which are often as much about support local agriculture by working on farms, or support public order as it is about being a military force.
There are also many nations out there who on paper have large stockpiles of equipment (particularly in the Middle East) and this can easily be turned into a headline about how a tiny nation has more tanks than the UK. The reality though is that these purchases are little more than an insurance policy designed to coax the nations into feeling an obligation to support the purchaser in a real crisis. If one views defence sales to the Middle East as a means of these nations buying support through economic largesse then that’s probably not far off the mark. Many of these equipment buys are in fact often stored in the desert and left to rust without ever being used. The author has heard many tales of armouries full of weapons never removed from packing crates, or trained on and often forgotten about. On paper this is a capability, and in reality it is little more than a box of life expired spare parts. One difference between the UK and many other nations is that the UK is willing to genuinely use and ‘sweat’ its assets to get the most from its equipment purchases. Just because some nations have impressive arsenals does not equate to a genuine ability to use them to best effect.
