• There’s virtually no difference between high or low priority bugs
• Fixing bugs is most often based on business factors (i.e. a small bug impacting a big customer)
• Bug triage is waste of time in terms of learning how to write less bugs
• Fixing bugs should be the highest priority
• Unit tests help, but they are not a cure-all
• Better to find bugs ASAP, to improve the learning

Here’s the full video:

Conscious Software Development: No More Bugs from Dymaxicon on Vimeo.

What do you think? Are we looking at bugs the wrong way? Are we more concerned with the defects themselves, as opposed to what we can learn from them? Let us know in the comment section.

Apple iPhone Tracking – First up, we learned last week that iPhones store their location in a file that never gets deleted, and then backup that file to iTunes each time the phone syncs. That means that anyone with access to a laptop belonging to an iPhone owner could see where they had been as long as they had owned their phone. (For the record, my iPhone says I spend a lot of time in Southborough, MA at the uTest headquarters.)

After a few days of silence on the issue, Apple announced that this was the result of a bug in iOS – three bugs actually. 1 – the iPhone keeps the location data for too long and should instead periodically purge it. 2 – this data is backed up to iTunes and should not be. 3 – the data is not deleted if a user disables location services. Apple has plans to fix all three bugs and to also begin encrypting the location file on the iPhone.

Why were they tracking this data at all? Apple uses this information (anonymously) to improve their location services and make it easier for iPhones to determine their location without having to resort to GPS (which is slow). But they only need a small amount of data at a time rather than the entire location history the iPhone was storing.

Do you have an iPhone? Are you curious to see where you’ve been? Here’s a clever app that will plot your location history on a map. If you’re into fancy statistical analysis, you can also use this add-on to plot your location using R.

Amazon EBS Outage – I don’t think anyone will deny that last weekend’s Amazon outage sucked for everyone involved. On April 21, cloud services in Amazon’s East Coast data center were dramatically impaired, halting operations for dozens of internet companies who use Amazon for hosting. The outage was long, and Amazon took several days to bring their data center back online.

Today, Amazon released a fascinating and detailed post-mortem explaining what happened during the outage and why. This document should be required reading for any software tester interested in root cause analysis. Amazon not only identifies what specifically triggered the event (an accidental network misconfiguration) but also how that single event could cause such a massive loss of service. It turns out that many of their well planned redundancies could, in the right circumstances, all execute at once and bring an entire data center to a grinding halt. They concluded that the real issue was software that incorrectly handled data replication during a failure – something they will correct with a bug-fix and update.

Kudos to Amazon for their transparency and for such a well documented failure analysis. We can all learn something from this.

$23 Million Books – Another Amazon story, but one for which they’re entirely blameless. Recently you could buy a copy of The Making of a Fly (a biology text) for a whopping $23,698,655.93 from their used book section. These books were not for sale by Amazon but rather by affiliates who use Amazon as an e-commerce platform. It turns out that two of those affiliates were locked in a very silly battle involving buggy pricing software that just didn’t know when to quit.

Bordeebook and Profnath are two independent book sellers who appear to use computer based pricing for their books. Bordeebook always set their price at 1.270589 times that of Profnath, and then Profnath would raise its price to be 0.9983 of Bordeebook. This cycle would continue over and over again until the top price of the book reached a whopping $23 million. The trend was originally noticed by Michael Eisen – a biology researcher at UC Berkeley. Eventually the issue was caught, and Profnath lowered their price to a more reasonable $106.23.

Lesson learned: when dealing with random data you should set checks, caps, and limits to prevent chaos.

Bugs Thwart Pirates – Not every bug is unplanned or undesirable. Would you believe that some game designers are even including bugs as a mechanism to identify and thwart software pirates? A recent article describes how one developer intentionally included a bug that would cause pirated versions of his game to crash with a very specific error message. When any pirates eventually posted a request for help on a software forum (along with the error message), moderators immediately knew they were dealing with a pirate and would instead ban them from the forum entirely.

This isn’t the first time game developers have used this trick. The always irreverent Cracked recently described six more examples in a recent posting. They range from the subtle (Batman is a klutz in pirated versions of Arkham Asylum) to the overt (the game music in Michael Jackson: The Experience is replaced with vuvuzelas).

Lesson learned: pay for your video games.

My prediction was a bit off (Drupal 7 actually launched in January, 2011). That’s not a bad thing because good software takes time, and our early evaluations of Drupal 7 have been very positive. We’re excited to begin migrating as soon as we can. Yet given that Drupal 7 took three years to develop, one has to wonder if we’ll see Drupal 8 before 2014.

With that in mind, I was a bit surprised to read Drupal founder Dries Buytaert’s plan for launching Drupal 8 on a more reasonable timescale: never have more than 15 critical bugs in the database. From this recent article in eCRMGuide:

“At no point in time will there be more than 15 critical bugs,” Buytaert said. “I will not pull in a big change if we know there are known bugs. This gives us the ability to do timely releases because we know at most the release is only 15 critical bugs away from being ready.”

So wait, is this practical?

The answer is almost certainly “yes!” From Joel Spolsky’s 12 Steps to Better Code:

5. Do you fix bugs before writing new code?
In general, the longer you wait before fixing a bug, the costlier (in time and money) it is to fix.

In this light, Dries may be on to something. If developers have to fix Drupal’s code before they’re allowed to check-in anything else, then they have a huge incentive to clear out bugs before they become major issues. In other words, their new features won’t make it into the codebase until they’ve cleaned up a sufficient number of outstanding prior mistakes.

What remains to be seen is if the critical bugs list can be properly managed to make this practical. Project managers will have to be ruthless about defining what is and isn’t a “critical bug.” It doesn’t help anyone if the list of critical bugs is out of date, contains invalid information, or has bugs related to incomplete features that just haven’t been written yet.

But kudos to Dries and the Drupal team. I sincerely hope this works out and I can’t wait to see Drupal 8.

A professional statistician named Mohan Srivastava managed to discover a flaw in certain kinds of scratch-off lottery games that  allow a player to get a winning edge by doing some simple math. Wired has the whole story, and it’s well worth reading. The summary is this:

Scratch-off lottery tickets aren’t totally random. A computer prints the tickets so that a certain number are guaranteed to win – thus meeting the odds requirements set by the laws of different states. That means that a computer program has to spit out both winning and non-winning scratch-off lottery tickets. The game that Mr. Srivastava cracked had two components – a visible grid of numbers and a scratch-off section with more numbers. You play the game by scratching off the hidden section and looking for for tic-tac-toe patterns in the grid.

What Mr. Srivastava realized is that the winning tickets had a slightly different statistical distribution of data in the grid section than non-winning tickets. Knowing this, he could pick out winning tickets with 90% certainty, all without scratching a single lottery ticket.

What are some lessons for testers?

First: no system is honest or perfect, no matter what anyone claims. An audited system may have been checked for certain kinds of known flaws, but clever exploration can frequently reveal new bugs that weren’t previously known.

Second: there’s real money at stake here. The lottery generates billions of dollars in revenue for state and local governments. Having that money be put in doubt can be a serious problem. Wired also points out that a broken lottery can be a perfect system for money laundering.

Third: be honest. It’s always tempting to use your knowledge of bugs and flaws for evil, but Mr. Srivastava sets the right example for testers everywhere. He told the lottery commission about his find, and got the broken scratch-off game removed from the market.

Do you play the lottery?

Case in point: SQS Software Quality Systems has put together a Top Ten list – a la David Letterman – of the worst software failures of 2010. If you don’t see your own software on this list, consider it a good year. Better yet, if someone didn’t make the list that should have (in your opinion), go ahead and call them out in the comment section.

And so without further delay, we bring you the Top Ten worst software bugs of 2010:

1. Car manufacturer – brake recall
Recall of two major car brands due to anti-lock brake system defect.

2. Wrong organs removed from donors
Faulty software led to the removal of the wrong organs from 25 donors in the UK. The error originated in faulty data conversion software that was used to upload information on donation preference.

3. Government department prevents completion of online tax returns
Hundreds of people are unable to complete their tax returns online in due to a software bug that has locked users out of their online accounts.

4. Stock exchange
A stock exchange suffered technical glitches during the first phase of its high-profile migration to new technology, trading on its alternative trading platform starting more than an hour late as a result of the problem.

5. Software glitch causes outage for thousands of GPS receivers
While installing software upgrades to ground control stations for a new fleet of GPS satellites, inspectors discovered a glitch in software compatibility that rendered up to 10,000 GPS receivers dark for at least two weeks.

Continue reading on net-security.org >>>

So which of these three sites will have the most robust feature set? Which will have the most intuitive search and product comparison features? Which one will have the most user friendly mobile application? And most important of all, which one will have the most bugs?

That’s for our community to determine. Once the testing phase of the competition is over (on Monday, November 8 at noon) we’ll be sending surveys to all participating testers, where they will compare the usability and feature set of these three applications.

At this point, I shouldn’t have to tell you about the many reasons why you should participate, but I will anyway. For one, a strong performance in the uTest Bug Battle can help you get recognized by our project managers, which means you’ll be invited to more private test cycles. It’s also a great way to hone your testing skills. Oh – and did we mention that we’re giving away $4000 in prizes across a broad range of categories?

So here’s your chance. Log into your uTest account, scour these apps for defects and report them in a clear, concise manner. And if you do it better than your peers, you could be named the Q4 Bug Battle winner and earn some big prize money for your time.

Need some pointers? This uTest Forums thread will show you how to increase your odds of winning.

Have more questions? Learn more about the Bug Battle basics.

Good luck everyone!

“It all began with a log entry from 1947 by Harvard University’s Mark II technical team. The now-classic entry features a moth taped to the page, time-stamped 15:45, with the caption ‘Relay #70 Panel F (moth) in relay’ and the proud boast, ‘First actual case of bug being found’ added by Grace Hopper.”

Since the tale of the moth, bugs have spanned from the benign to wreaking complete havoc. Many of the classic bugs of the past have been detailed in our TWIT (This Week In Testing) posts, but here are the top bugs that truly pay tribute to this day:

• The Mars Climate Orbiter doesn’t orbit (CW)
• The Most Expensive Hyphen In History
• Forty seconds of Ariane-5 (CW)
• Pentium chips fail math (CW)
• All Circuits Are Currently Busy — 20 Years After AT&T Crash
• Windows Genuine (Dis)advantage (CW)
• Black Monday’s dark secret (CW)
• International Date Line Bug Caused Fighter Aircraft Systems Crash

While we remember these bugs, today is also a day to celebrate and thank software testers all around the world who discover and prevent these disasters (and the smaller but crucial defects) every day. Thanks!

In part I of our interview, we get his thoughts on the Worst Bug Ever; his testing philosophy; what it means to be a defensive pessimist; testing certifications, the state of the industry and more. Be sure to check tomorrow for part II.

**************

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

I’d like to be able to say that I carefully selected the name and that it signifies that I care about quality and I’m amphibious like a frog. I’d like to say something along the lines that I started life as a tadpole in the waters of programming and later grew legs to live on land and be a tester.  I could even say that as a Quality Frog, I now eat bugs for breakfast and help keep the waters clear. While such thoughts may have come to mind, the truth is that I came up with Quality Frog while pairing a variety of words with Quality in search of an available domain name. Frog came to mind as something that ate bugs and the domain name was available. Since then, I’ve continued to use it as a handle.

uTest: You’re a self-described “defensive pessimist”, which seem like good qualities for a tester to have. What other attributes come in handy in this line of work?

Simo: The term “defensive pessimist” comes from Dr. Julie Norem, a psychology professor at Wellesley College.  In her book, “The Positive Power of Negative Thinking”, Dr Norem describes defensive pessimists as people who typically perform worse when pressured to look on the bright side and be optimistic about things that concern them.  Rather than trying to think happy thoughts and only look at the positive, defensive pessimists imagine the worst case scenario; not to get depressed and become immobilized, but to develop solutions for what might go wrong in order to be better prepared. Defensive pessimists can make great testers, and are likely to annoy many optimists.

The third guiding principle of the Association for Software Testing states “AST views software testing as a cognitively complex activity that requires critical thinking, effective communication, and rapid self-directed learning.”  I fully agree with this.  Therefore, I find it essential that testers be critical thinkers, effective communicators, and self-educators.  Any one of these three things without the others will make us less effective as testers.

This doesn’t mean that every tester must master all three of these on their own.  My ideal test team would be comprised of people with a diversity of aptitudes, skills, and experience. I don’t want a team of clones.

uTest: Much like our previous Testing the Limits’ guests, you’re a critic of testing certifications. Yet some still see certifications as the only way to stand out from the “unskilled labor” crowd. Tell us a bit about why you’re a skeptic/critic of certs – and how they could be improved and made more relevant/useful/predictive.

Simo: I am not against certifications per se. I am against bad certifications.  I am against certifications that are presented to be something other than what they are. I am against certification bodies and trainers that prey upon people’s desire to stand out and tell people they can improve and certify their competence as testers with few days of training and a multiple-false test. In his keynote at the Conference of the Association for Software Testing (CAST) this year, Cem Kaner stated that If one can become certified in their profession in three days, they are a commodity, and don’t deserve much more pay than unskilled labor. I agree with Dr. Kaner. Rather than educate and help people stand out from the unskilled labor crowd, such certifications trivialize testing and encourage wrong thinking that testing is unskilled labor.  I want testers to be more than a commodity.

Many IT certifications, including testing certifications, are more about marketing than education.  These certifications are not good measures of skill, competency, professionalism, quality, or any of the many things those on the receiving end (of the marketing) care about. In my experience interviewing job candidates, tester certifications have not been an indicator of applicants’ testing abilities.

Software testing is a rich and diverse field. It is also a young field. Rather than feign maturity and simplicity where there is none, let’s embrace the diversity and youth. Let’s continue to learn. Let’s not lock in a set of context-free definitions and practices and make them a standard. Such standards will hurt the quality of software, not improve it.

Rather than pursue a certification, I encourage testers to get involved in a professional community. Find colleagues that challenge you and help you learn.  Seek real education that comes through interaction and doing over memorizing information useful in passing a multiple-false test. The Association for Software Testing offers a series of online software testing courses that facilitate deeper learning that you are likely to find in training focused on helping you pass a certification exam.

Editor’s note: We hope enjoyed part I of our interview with Ben Simo. Here’s part II.

Our guest blogger this month is Atul Angra. A resident of India, Atul is one of our more accomplished testers (a Gold Tester in fact), with over six years of professional experience. He’s a photographer at heart, but a tester by trade, with domain expertise in healthcare and finance.

*******

Let’s take a scenario where a tester follows the rules and reports 100 bugs. Some of these bugs were traced to non-documented requirements that are implicit in nature, such as a drop-down list not populating alphabetically and things of that nature. These bugs are quite common and usually end up in conflict, as development teams reject them based on the argument that it’s not a defined requirement.

Here, both the developer and tester are not ready to close this issue – and they are both correct. The traditional way these issues are resolved is by involving someone from management to intervene and make a decision. The time spent in escalation and argument is much greater than what it would have taken to actually fix the issue.

At a high level, we could blame the team which collected requirement, but this may not be the case when it comes to implicit requirements. Many of these situations could be resolved if the tester demonstrates interpersonal skills.

Testers play a challenging and critical role in the organization. They dare their developer counterparts and are constantly challenged by release managers, as they pose significant risk in delaying the release. This may even stop the organization from achieving a financial target on time. In other words, testers play the role of Devil’s advocate when it comes to improving quality of a deliverable.

As such, good testing skills and good interpersonal skills make the KILLER COMBO that suits this role. Critics are rarely appreciated for their work, but coaches are. Yet coaches and critics do the same thing: they point out your mistakes and give you a chance to perform better the next time around.

The difference is in their approach. An ideal tester should become a coach instead of a critic. Developers turn defensive when testers approach them with a statement such as “This is not working as intended.”

With good interpersonal skills, these discussions can become more effective. A good tester will put forth a scenario that makes the developer consider the impact if the bug is not fixed.

A tester who reports the maximum number of critical bugs with a low rejection ratio is considered efficient. Other expectations from testers are to meet deadlines, ensure process compliance, and practice good documentation with complete functional testing. They ignore a very important attribute here:  interpersonal skills.

Conclusion:

A tester might have all the required technical skills, but may still fail because of his/her interpersonal skills. A bug that could have been amicably solved turns into a management issue with leads to a lot of wasted time.

Testers should spend time in enhancing their interpersonal skills. People always like to have colleagues who are a good listeners and who love to share knowledge. One who shares information and resources, and is helpful by bringing conflicts to the surface and getting them resolved in an ethical and professional manner.

Editor’s note: If you would like to write for our Guest Blogger series, send your posts and ideas to me at mikeb@utest.com

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

• Since 2005, no significant up-, or downward trend in the total number of vulnerabilities in the more than 29,000 products covered by Secunia Vulnerability Intelligence was observed.
• A group of ten vendors, including Microsoft, Apple, Oracle, IBM, Adobe, and Cisco, account on average for 38 percent of all vulnerabilities disclosed per year.
• In the two years from 2007 to 2009, the number of vulnerabilities affecting a typical end-user PC almost doubled from 220 to 420, and based on the data of the first six months of 2010, the number is expected to almost double again in 2010 to 760.
• During the first six months of 2010, 380 vulnerabilities or 89% of the figures for all of 2009 has already been reached.

ZDNet’s Adrian Kingsley-Hughes made note of the products that are causing these vendors to rise or fall in the standings. The culprits, he says, are as follows:

• Apple - (iTunes, Quicktime)
• Microsoft – (Windows, Internet Explorer)
• Sun Microsystems – (Java, now part of Oracle)
• Adobe - (Acrobat Reader, Flash)

So testers, who do you see as the Bug Marathon winner (i.e. loser) at the end of the year? Not that we’re keeping score or anything.