Software Testing Blog

I wanted to share a short but interesting video essay from Scrum co-founder Jeff McKenna on the subject of software bugs – mainly, how we think of them and how we act on them. Here a few items I found particularly intriguing, see if you agree:

• There’s virtually no difference between high or low priority bugs
• Fixing bugs is most often based on business factors (i.e. a small bug impacting a big customer)
• Bug triage is waste of time in terms of learning how to write less bugs
• Fixing bugs should be the highest priority
• Unit tests help, but they are not a cure-all
• Better to find bugs ASAP, to improve the learning

Here’s the full video:

Conscious Software Development: No More Bugs from Dymaxicon on Vimeo.

What do you think? Are we looking at bugs the wrong way? Are we more concerned with the defects themselves, as opposed to what we can learn from them? Let us know in the comment section.

We love studying bugs when they come up, and this past week we’ve seen a few big ones go by. When bugs happen, there’s always a lot we can learn from them. Here’s a quick roundup of four different bugs that were recently in the news:

Apple iPhone Tracking – First up, we learned last week that iPhones store their location in a file that never gets deleted, and then backup that file to iTunes each time the phone syncs. That means that anyone with access to a laptop belonging to an iPhone owner could see where they had been as long as they had owned their phone. (For the record, my iPhone says I spend a lot of time in Southborough, MA at the uTest headquarters.)

After a few days of silence on the issue, Apple announced that this was the result of a bug in iOS – three bugs actually. 1 – the iPhone keeps the location data for too long and should instead periodically purge it. 2 – this data is backed up to iTunes and should not be. 3 – the data is not deleted if a user disables location services. Apple has plans to fix all three bugs and to also begin encrypting the location file on the iPhone.

Why were they tracking this data at all? Apple uses this information (anonymously) to improve their location services and make it easier for iPhones to determine their location without having to resort to GPS (which is slow). But they only need a small amount of data at a time rather than the entire location history the iPhone was storing.

Do you have an iPhone? Are you curious to see where you’ve been? Here’s a clever app that will plot your location history on a map. If you’re into fancy statistical analysis, you can also use this add-on to plot your location using R.

Read more…

Back in 2009 when we relaunched the uTest website (who here remembers our old logo?), we chose Drupal 6 for our home page CMS. Back then, I was certain that Drupal 7 would launch “real soon now” and that we would be thinking about an upgrade pathway sometime in early 2010.

My prediction was a bit off (Drupal 7 actually launched in January, 2011). That’s not a bad thing because good software takes time, and our early evaluations of Drupal 7 have been very positive. We’re excited to begin migrating as soon as we can. Yet given that Drupal 7 took three years to develop, one has to wonder if we’ll see Drupal 8 before 2014.

With that in mind, I was a bit surprised to read Drupal founder Dries Buytaert’s plan for launching Drupal 8 on a more reasonable timescale: never have more than 15 critical bugs in the database. From this recent article in eCRMGuide:

“At no point in time will there be more than 15 critical bugs,” Buytaert said. “I will not pull in a big change if we know there are known bugs. This gives us the ability to do timely releases because we know at most the release is only 15 critical bugs away from being ready.”

So wait, is this practical?

Read more…

Who wouldn’t like the idea of cracking the lottery? Just figure out the code, and incredible riches can be yours! But the lottery is unbreakable – audited by governments, contractors, corporations, and independent agencies; or at least that’s what they want you to think.

A professional statistician named Mohan Srivastava managed to discover a flaw in certain kinds of scratch-off lottery games that  allow a player to get a winning edge by doing some simple math. Wired has the whole story, and it’s well worth reading. The summary is this:

Scratch-off lottery tickets aren’t totally random. A computer prints the tickets so that a certain number are guaranteed to win – thus meeting the odds requirements set by the laws of different states. That means that a computer program has to spit out both winning and non-winning scratch-off lottery tickets. The game that Mr. Srivastava cracked had two components – a visible grid of numbers and a scratch-off section with more numbers. You play the game by scratching off the hidden section and looking for for tic-tac-toe patterns in the grid.

What Mr. Srivastava realized is that the winning tickets had a slightly different statistical distribution of data in the grid section than non-winning tickets. Knowing this, he could pick out winning tickets with 90% certainty, all without scratching a single lottery ticket.

What are some lessons for testers?

Read more…

It’s often said that “bad press is better than no press at all.” Well, that might be true in some circles, but certainly NOT in software testing and development.

Case in point: SQS Software Quality Systems has put together a Top Ten list – a la David Letterman – of the worst software failures of 2010. If you don’t see your own software on this list, consider it a good year. Better yet, if someone didn’t make the list that should have (in your opinion), go ahead and call them out in the comment section.

And so without further delay, we bring you the Top Ten worst software bugs of 2010:

1. Car manufacturer – brake recall
Recall of two major car brands due to anti-lock brake system defect.

2. Wrong organs removed from donors
Faulty software led to the removal of the wrong organs from 25 donors in the UK. The error originated in faulty data conversion software that was used to upload information on donation preference.

3. Government department prevents completion of online tax returns
Hundreds of people are unable to complete their tax returns online in due to a software bug that has locked users out of their online accounts.

4. Stock exchange
A stock exchange suffered technical glitches during the first phase of its high-profile migration to new technology, trading on its alternative trading platform starting more than an hour late as a result of the problem.

5. Software glitch causes outage for thousands of GPS receivers
While installing software upgrades to ground control stations for a new fleet of GPS satellites, inspectors discovered a glitch in software compatibility that rendered up to 10,000 GPS receivers dark for at least two weeks.

Continue reading on net-security.org >>>

The Q4 uTest Bug Battle is officially underway. As we noted yesterday, this quarter’s competition will challenge testers to find bugs in three of world’s most prominent specialty e-tailers: eBay, Overstock and Zappos.

So which of these three sites will have the most robust feature set? Which will have the most intuitive search and product comparison features? Which one will have the most user friendly mobile application? And most important of all, which one will have the most bugs?

That’s for our community to determine. Once the testing phase of the competition is over (on Monday, November 8 at noon) we’ll be sending surveys to all participating testers, where they will compare the usability and feature set of these three applications.

At this point, I shouldn’t have to tell you about the many reasons why you should participate, but I will anyway. For one, a strong performance in the uTest Bug Battle can help you get recognized by our project managers, which means you’ll be invited to more private test cycles. It’s also a great way to hone your testing skills. Oh – and did we mention that we’re giving away $4000 in prizes across a broad range of categories?

So here’s your chance. Log into your uTest account, scour these apps for defects and report them in a clear, concise manner. And if you do it better than your peers, you could be named the Q4 Bug Battle winner and earn some big prize money for your time.

Need some pointers? This uTest Forums thread will show you how to increase your odds of winning.

Have more questions? Learn more about the Bug Battle basics.

Good luck everyone!

Today Computerworld (CW) has officially declared it national Debugging Day. It’s not usually formally celebrated (although it should be!), but the debugging tradition has been honored for more than 50 years now. From the very first bug to space bugs to horrible PR bugs to end-of-the-world bugs, today is the day to examine the most infamous bugs in history.

“It all began with a log entry from 1947 by Harvard University’s Mark II technical team. The now-classic entry features a moth taped to the page, time-stamped 15:45, with the caption ‘Relay #70 Panel F (moth) in relay’ and the proud boast, ‘First actual case of bug being found’ added by Grace Hopper.”

Since the tale of the moth, bugs have spanned from the benign to wreaking complete havoc. Many of the classic bugs of the past have been detailed in our TWIT (This Week In Testing) posts, but here are the top bugs that truly pay tribute to this day:

• The Mars Climate Orbiter doesn’t orbit (CW)
• The Most Expensive Hyphen In History
• Forty seconds of Ariane-5 (CW)
• Pentium chips fail math (CW)
• All Circuits Are Currently Busy — 20 Years After AT&T Crash
• Windows Genuine (Dis)advantage (CW)
• Black Monday’s dark secret (CW)
• International Date Line Bug Caused Fighter Aircraft Systems Crash

While we remember these bugs, today is also a day to celebrate and thank software testers all around the world who discover and prevent these disasters (and the smaller but crucial defects) every day. Thanks!

Our Testing the Limits guest this month is Ben Simo. Known as the “Quality Frog” on Twitter, Ben is one of the most insightful and entertaining testers in the business. A proponent of the context-driven school, Ben has more than 19 years of experience testing software and developing testing tools. He currently lives in Colorado with his wife, two children, two dogs, five cats and fourteen – count ‘em – fourteen goldfish. For the full Ben Simo experience, go to his blog.

In part I of our interview, we get his thoughts on the Worst Bug Ever; his testing philosophy; what it means to be a defensive pessimist; testing certifications, the state of the industry and more. Be sure to check tomorrow for part II.

**************

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

Read more…

Our guest blogger this month is Atul Angra. A resident of India, Atul is one of our more accomplished testers (a Gold Tester in fact), with over six years of professional experience. He’s a photographer at heart, but a tester by trade, with domain expertise in healthcare and finance. He’s also a former Bug Battle winner, a guest judge, a Tester of the Year, a Forums junkie, a crash course author and he’s here today to discuss how interpersonal skills can make or break a tester’s career. Enjoy!

*******

Let’s take a scenario where a tester follows the rules and reports 100 bugs. Some of these bugs were traced to non-documented requirements that are implicit in nature, such as a drop-down list not populating alphabetically and things of that nature. These bugs are quite common and usually end up in conflict, as development teams reject them based on the argument that it’s not a defined requirement.

Here, both the developer and tester are not ready to close this issue – and they are both correct. The traditional way these issues are resolved is by involving someone from management to intervene and make a decision. The time spent in escalation and argument is much greater than what it would have taken to actually fix the issue.

At a high level, we could blame the team which collected requirement, but this may not be the case when it comes to implicit requirements. Many of these situations could be resolved if the tester demonstrates interpersonal skills.

Read more…

Take that Oracle! You just let Apple capture the lead in the 2010 Bug Marathon, otherwise known as Secunia’s Half Year Report (PDF). Worth the read, the 20-page report identifies the ten largest vendors with the most vulnerabilities (in all their products) and ranks them for the first half of 2010 – great entertainment for those who like to track bugs and keep score.

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

Read more…