Software Testing Blog

My last car had a really irritating rattle. Something in the dashboard would vibrate as the car drove – never loud enough that I could say it was a real problem but never soft enough for me to ignore it either. It was just the right level to irritate me as I drove along, except for when I forgot it was there.

Wait, when I forgot that it was there? Well, of course there were lots of times when the sound of the rattle would just disappear in the background noise, or perhaps it would get lost in the noise of the radio. I forgot about it a lot, and it really only bugged me when I remembered and noticed it. And then for the next several days, the sound would grate on my nerves until I forgot about it again.

Some bugs are like that. They’re bugs in the traditional sense – the feature doesn’t work as expected. But they’re never so important that anyone wants to rip apart the software (or the dashboard of a car) to find and fix the problem.

So does that mean that these kinds of bugs are unimportant?

Read more…

Who wouldn’t like the idea of cracking the lottery? Just figure out the code, and incredible riches can be yours! But the lottery is unbreakable – audited by governments, contractors, corporations, and independent agencies; or at least that’s what they want you to think.

A professional statistician named Mohan Srivastava managed to discover a flaw in certain kinds of scratch-off lottery games that  allow a player to get a winning edge by doing some simple math. Wired has the whole story, and it’s well worth reading. The summary is this:

Scratch-off lottery tickets aren’t totally random. A computer prints the tickets so that a certain number are guaranteed to win – thus meeting the odds requirements set by the laws of different states. That means that a computer program has to spit out both winning and non-winning scratch-off lottery tickets. The game that Mr. Srivastava cracked had two components – a visible grid of numbers and a scratch-off section with more numbers. You play the game by scratching off the hidden section and looking for for tic-tac-toe patterns in the grid.

What Mr. Srivastava realized is that the winning tickets had a slightly different statistical distribution of data in the grid section than non-winning tickets. Knowing this, he could pick out winning tickets with 90% certainty, all without scratching a single lottery ticket.

What are some lessons for testers?

Read more…

Pardon my American “gun-loving” nature while I start this blog post with an analogy about guns. Let’s say that I’m interested in learning how to shoot a gun, so I go visit my local shooting range for instruction. The people who run the shooting range (which is fully licensed and very reputable) give me extensive safety training before finally issuing me a gun with some ammunition. While standing in front of the target, I carefully load the gun, point the barrel at my foot, and pull the trigger.

So whose fault is it that I just shot myself in the foot? Is it the shooting range’s for not issuing me special bullet proof boots or for not training me even more thoroughly about the dangers of shooting my extremities? Or is it mine, for shooting myself in the foot?

A recent Windows “bug” has put Microsoft in the position of the shooting range. I put “bug” in quotes because it’s hardly the shooting range’s fault that I shot myself in the foot, but it’s still their problem in the end. Same with Microsoft who must now deal with an annoying flaw in their DLL loading behavior that isn’t their fault but has become their problem. Ars Technica explains the problem very well, but here’s a quick summary of the issue:

Read more…

This month’s installment of ‘This Week In Testing‘ takes us waaaay back to 1962 when the Mariner I space probe, America’s first planetary flyby that was supposed to go to Venus, went completely off course and had to be immediately destroyed — a mere 293 seconds after launch.

The Cost? $18.2 million (in 1962!)

The Bug? Omission of a single overbar

The Mariner I was the first spacecraft of the NASA Mariner program that “launched a series of robotic interplanetary probes designed to investigate Mars, Venus and Mercury (Wikipedia).”

The bug that brought the mission to its speedy end was carried out by a programmer, who while transcribing a handwritten (in pencil no less) formula into code, missed one single overbar (or as it’s less-technically known: the hyphen).

NASA’s public account of the software glitch is written as follows:

The Mariner 1 Post Flight Review Board determined that the omission of a hyphen in coded computer instructions in the data-editing program allowed transmission of incorrect guidance signals to the spacecraft. During the periods the airborne beacon was inoperative the omission of the hyphen in the data-editing program caused the computer to incorrectly accept the sweep frequency of the ground receiver as it sought the vehicle beacon signal and combined this data with the tracking data sent to the remaining guidance computation. This caused the computer to swing automatically into a series of unnecessary course corrections with erroneous steering commands which finally threw the spacecraft off course.

Fortunately, the mission was successfully completed by Mariner 2 five months later, but it’s hard to ignore the significant costs brought about by a mere hyphen. Do you have any bug stories like this one? Has a missing bar (or something equivalent) ever led you to a messy debacle?

In this month’s installment of This Week In Testing, the date was May 1996 and the setting was the First National Bank of Chicago (insert dramatic pause here). The gist? Software “glitches” caused the bank accounts of 823 customers of the major US bank to be credited with a total of $924,844,208.32 each.

According to The American Bankers Association, all of $763.9 billion — more than six times the total assets of First Chicago NBD Corp. — was the largest error in US banking history.

And the reason given? Inadequate testing of course! The bank updated its ATM transaction software with new message codes. The message codes were unfortunately not tested on all ATM protocols, which resulted in some ATMs interpreting the codes as huge increases to customer balances.

This isn’t the first time we bring up banking bugs. You might remember Software Bugs: You Win Sum, You Lose Sum, the post about a man in Orlando who while making a routine bank transfer was shocked to see his balance at $88,888,888,888.88.

What other bugs have you recently heard or read about with such huge financial implications? Any mobile banking bugs?

Before I get into the story of this fascinating bug, I wanted to take a moment to introduce you to T.W.I.T. We liked the “bug-iversary” concept so much here at uTest that we decided to make it a recurring column, called T.W.I.T. or This Week In Testing (also noting the happy coincidence that the word “twit” is synonymous with “fool” and “dope,” words that characterize many of these bug follies ).

But I digress! So, this week in testing brings us an interesting heart device bug discovered March 12, 2008.

A team of computer security researchers were able to gain wireless access to a combination heart defibrillator and pacemaker. According to the New York Times,

[The researchers] were able to reprogram it to shut down and to deliver jolts of electricity that would potentially be fatal. The researchers said they had also been able to glean personal patient data by eavesdropping on signals from the tiny wireless radio embedded in the implant as a way to let doctors monitor and adjust it without surgery.

Full report and more after the bump!

Read more…

SCMagazine reported this week that researchers in Malta have discovered a decade-old vulnerability, present in all versions of Windows since 2000.  This bug can cause PCs to crash instantaneously and without warning, as well as reeling the compromised machine into a distributed denial-of-service (DDoS) attack.  This exploit is only dangerous if the user is duped into running an app with the malicious code (according to Paul Gafa, CTO of 2X Software).

The bug was discovered while Gafa was writing a software testing app:

“You can be the least privileged user on the system and still crash it,” Gafa said. “I believe it is very easy for Microsoft to sort it out. They just need to validate arguments passed to Windows APIs.” (source: SC Magazine)

Microsoft is currently aware of the defect and responded with this insight:

“Our initial assessment of the report is that malicious code would have to already be running or a user would have to be able to run a specially crafted application to cause the system to crash. In either case, the system has already been compromised or the user has rights to logon to the system.”

I’m curious to hear if anyone has other stories of old bugs causing new problems or vulnerabilities?

With our testing community currently hammering away in the “Bug Battle of the TV Networks” this week, it’s time to take a moment to reflect on our February bug-iversary.

On February 11, 2007, during its very first overseas deployment to Okinawa, Japan, six F-22 Raptors flying from Hawaii experienced multiple computer crashes, including navigation, communication and fuel system crashes, when crossing the International Date Line.

Read more…

Bug-iversary Alert! Tomorrow is the 20-year anniversary of the “crash” of the AT&T Long Distance Network. On January 15, 1990 faulty software was installed on the AT&T Electronic Switching System (Number 4 ESS): a one-line bug incapacitated the entire system, disabling switches throughout half the network.

Known as one of the most serious telecom bugs in history, more than 75 million calls were not connected during 9 hours, an estimated $60 million loss.

Dennis Burke of California Polytechnic said it best: “The Jan. 1990 incident showed how bugs in self-healing software can bring down healthy systems, and the difficulty of detecting obscure load- and time-dependent defects in software.”

Speaking of “load defects,” AT&T — after signing up to be exclusive U.S. provider of iPhone service — has recently come under fire for the quality of its network coverage. Businessweek‘s top headlines read:

• An AT&T Mystery: Abrupt New York iPhone Shutdown
• AT&T Mulls Plans to Deal with iPhone Data Demand
• Can AT&T Meet iPhone Network Demands?

In light of this bug-iversary, I can’t help but wonder if more testing should have been done before AT&T took on the massive data demands of modern 3G smartphones? What do you think?

When the dust settled on our Battle of the E-Tailers last month, Amazon.com performed exceptionally well – scoring high marks for its usability, feature set and a host of other categories. In fact,  74% of our testers said they trusted Amazon most for their holiday shopping in the post-Bug Battle survey.

But despite the fact that hundreds of testers were scouring their site for flaws, at least one MAJOR software bug went unnoticed: a $3 billion validation error, as recently reported by NetworkWorld.

Apparently, a Californian software engineer in the market for a Discovery Channel ‘Cells’ CD-ROM was able to find an edition on Amazon. No surprises here. The only problem (aside from the fact it only runs on Windows 98), was that the price tag for the item was about $3 billion. Plus $3.99 shipping and handling, of course. Seeing the price, and knowing full well that Amazon would settle the issue in a fair and timely manner, he decided to purchase the item on a lark. Here’s his first hand account:

Read more…