Software Testing Blog

Our Testing the Limits guest this month is Ben Simo. Known as the “Quality Frog” on Twitter, Ben is one of the most insightful and entertaining testers in the business. A proponent of the context-driven school, Ben has more than 19 years of experience testing software and developing testing tools. He currently lives in Colorado with his wife, two children, two dogs, five cats and fourteen – count ‘em – fourteen goldfish. For the full Ben Simo experience, go to his blog.

In part I of our interview, we get his thoughts on the Worst Bug Ever; his testing philosophy; what it means to be a defensive pessimist; testing certifications, the state of the industry and more. Be sure to check tomorrow for part II.

**************

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

Read more…

Our guest blogger this month is Atul Angra. A resident of India, Atul is one of our more accomplished testers (a Gold Tester in fact), with over six years of professional experience. He’s a photographer at heart, but a tester by trade, with domain expertise in healthcare and finance. He’s also a former Bug Battle winner, a guest judge, a Tester of the Year, a Forums junkie, a crash course author and he’s here today to discuss how interpersonal skills can make or break a tester’s career. Enjoy!

*******

Let’s take a scenario where a tester follows the rules and reports 100 bugs. Some of these bugs were traced to non-documented requirements that are implicit in nature, such as a drop-down list not populating alphabetically and things of that nature. These bugs are quite common and usually end up in conflict, as development teams reject them based on the argument that it’s not a defined requirement.

Here, both the developer and tester are not ready to close this issue – and they are both correct. The traditional way these issues are resolved is by involving someone from management to intervene and make a decision. The time spent in escalation and argument is much greater than what it would have taken to actually fix the issue.

At a high level, we could blame the team which collected requirement, but this may not be the case when it comes to implicit requirements. Many of these situations could be resolved if the tester demonstrates interpersonal skills.

Read more…

Take that Oracle! You just let Apple capture the lead in the 2010 Bug Marathon, otherwise known as Secunia’s Half Year Report (PDF). Worth the read, the 20-page report identifies the ten largest vendors with the most vulnerabilities (in all their products) and ranks them for the first half of 2010 – great entertainment for those who like to track bugs and keep score.

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

Read more…

This month’s installment of ‘This Week In Testing‘ takes us waaaay back to 1962 when the Mariner I space probe, America’s first planetary flyby that was supposed to go to Venus, went completely off course and had to be immediately destroyed — a mere 293 seconds after launch.

The Cost? $18.2 million (in 1962!)

The Bug? Omission of a single overbar

The Mariner I was the first spacecraft of the NASA Mariner program that “launched a series of robotic interplanetary probes designed to investigate Mars, Venus and Mercury (Wikipedia).”

The bug that brought the mission to its speedy end was carried out by a programmer, who while transcribing a handwritten (in pencil no less) formula into code, missed one single overbar (or as it’s less-technically known: the hyphen).

NASA’s public account of the software glitch is written as follows:

The Mariner 1 Post Flight Review Board determined that the omission of a hyphen in coded computer instructions in the data-editing program allowed transmission of incorrect guidance signals to the spacecraft. During the periods the airborne beacon was inoperative the omission of the hyphen in the data-editing program caused the computer to incorrectly accept the sweep frequency of the ground receiver as it sought the vehicle beacon signal and combined this data with the tracking data sent to the remaining guidance computation. This caused the computer to swing automatically into a series of unnecessary course corrections with erroneous steering commands which finally threw the spacecraft off course.

Fortunately, the mission was successfully completed by Mariner 2 five months later, but it’s hard to ignore the significant costs brought about by a mere hyphen. Do you have any bug stories like this one? Has a missing bar (or something equivalent) ever led you to a messy debacle?

In this month’s installment of This Week In Testing, the date was May 1996 and the setting was the First National Bank of Chicago (insert dramatic pause here). The gist? Software “glitches” caused the bank accounts of 823 customers of the major US bank to be credited with a total of $924,844,208.32 each.

According to The American Bankers Association, all of $763.9 billion — more than six times the total assets of First Chicago NBD Corp. — was the largest error in US banking history.

And the reason given? Inadequate testing of course! The bank updated its ATM transaction software with new message codes. The message codes were unfortunately not tested on all ATM protocols, which resulted in some ATMs interpreting the codes as huge increases to customer balances.

This isn’t the first time we bring up banking bugs. You might remember Software Bugs: You Win Sum, You Lose Sum, the post about a man in Orlando who while making a routine bank transfer was shocked to see his balance at $88,888,888,888.88.

What other bugs have you recently heard or read about with such huge financial implications? Any mobile banking bugs?

You – the second person English pronoun.  You are the one reading this article. You were Time Magazine’s Person of the Year in 2006. You are special. You rock. Our company name is all about you and testing.

You have also been very naughty. Check out this Twitter entry written by you:

I kill people who nudge me

Wait, that wasn’t written by you? It was written by someone else named You? Oh, our mistake. And apparently it was Twitter’s mistake too according to this article on TechCrunch.

Twitter likes to tell you who is doing what and when at the bottom of each tweet. For example, a post description might tell you that it was retweeted by a friend.  Or if you were the one doing the retweeting, then the post description should say that it was retweeted by “you”.  But what happens when a buggy hyperlinking algorithm decides that anything after the words “Retweeted by” should link to a Twitter profile?

“Retweeted by you” becomes “Retweeted by you” – as in twitter.com/you. And you sounds cranky.

There are a lot of good lessons here for testers and developers, but I want to highlight a few particular:

Read more…

In light of Toyota’s recent quality issues, the number of formal consumer complaints has risen above the norm. To make matters worse, Toyota has had an extremely difficult time making sense of all this new feedback.

Why? Well, if you are an experienced QA professional, you know exactly why.

A recent article about how to write a useful NHTSA (National Highway Traffic Safety) complaint should strike a chord with software testers. The complaint template is very similar to the bug reports we all know and love. In fact, they both serve the same purpose: defect reporting.

Consumers can learn a few lessons from software testers – and vice versa – by taking a look at some key excerpts from the article:

Include data that will help the manufacturer better understand the problem:

• Facts about your vehicle and maintenance records
• What you did and how the vehicle responded
• Evidence and extra details

Read more…

If you lived in the United States during the 1980s, then you probably remember the famous Your Brain on Drugs ad campaign.  Created by the government to combat drug abuse, the ad compares the damaging effects of using drugs to frying an egg.

So what about bugs, as in software bugs?  More than just a lame rhyme, it turns out that bugs may have a negative effect on our brains as well – if you believe the Extended Mind hypothesis.  Stick with me here.

The Extended Mind hypothesis says that our minds are more than what is contained inside our skulls.  When we create or use tools, then we are effectively creating extensions of ourselves.  For example, that would mean that there’s no difference between remembering the capital of the state of Kentucky and looking it up on Wikipedia.  (Here’s a link to help you remember.)

A recent study suggests that there may be some validity to this, a fact discovered by creating a simple software bug and seeing how people respond.  From a recent article in Wired:

An empirical test of ideas proposed by Martin Heidegger shows the great German philosopher to be correct: Everyday tools really do become part of ourselves.

The findings come from a deceptively simple study of people using a computer mouse rigged to malfunction. The resulting disruption in attention wasn’t superficial. It seemingly extended to the very roots of cognition.

Read more…

Before I get into the story of this fascinating bug, I wanted to take a moment to introduce you to T.W.I.T. We liked the “bug-iversary” concept so much here at uTest that we decided to make it a recurring column, called T.W.I.T. or This Week In Testing (also noting the happy coincidence that the word “twit” is synonymous with “fool” and “dope,” words that characterize many of these bug follies ).

But I digress! So, this week in testing brings us an interesting heart device bug discovered March 12, 2008.

A team of computer security researchers were able to gain wireless access to a combination heart defibrillator and pacemaker. According to the New York Times,

[The researchers] were able to reprogram it to shut down and to deliver jolts of electricity that would potentially be fatal. The researchers said they had also been able to glean personal patient data by eavesdropping on signals from the tiny wireless radio embedded in the implant as a way to let doctors monitor and adjust it without surgery.

Full report and more after the bump!

Read more…

SCMagazine reported this week that researchers in Malta have discovered a decade-old vulnerability, present in all versions of Windows since 2000.  This bug can cause PCs to crash instantaneously and without warning, as well as reeling the compromised machine into a distributed denial-of-service (DDoS) attack.  This exploit is only dangerous if the user is duped into running an app with the malicious code (according to Paul Gafa, CTO of 2X Software).

The bug was discovered while Gafa was writing a software testing app:

“You can be the least privileged user on the system and still crash it,” Gafa said. “I believe it is very easy for Microsoft to sort it out. They just need to validate arguments passed to Windows APIs.” (source: SC Magazine)

Microsoft is currently aware of the defect and responded with this insight:

“Our initial assessment of the report is that malicious code would have to already be running or a user would have to be able to run a specially crafted application to cause the system to crash. In either case, the system has already been compromised or the user has rights to logon to the system.”

I’m curious to hear if anyone has other stories of old bugs causing new problems or vulnerabilities?