Software Testing Blog

A number of years ago, I took a tour of a plate glass factory. Plate glass manufacturing is pretty simple: dirt pours in one end of a factory where it’s melted in a huge furnace. The melted dirt is then poured out as a thin sheet which then cools into glass as it rolls along a mile-long conveyor belt. The process is continuous – dirt constantly pours in and glass constantly flows out in a never ending ribbon. At the very end of the factory, away from the furnace, a lonely robot slices the ribbon into panes of glass for things like windows and doors.

Periodically, a technician will take one of those glass panes back to a lab where it is broken up, melted, dissolved with chemicals, and analyzed in fine detail under a microscope. That technician is a tester – one who is testing the production of the glass to make sure it matches quality requirements. His job is very different from that of a software tester, but surprisingly there are many things a software tester can learn from him.

That may sound bizarre because software isn’t manufactured. There is no real “production” in software – every copy of an application should be exactly the same. But production testing is about more than manufacturing. It’s about managing variability – and understanding variability should be incredibly important to software testers.

Read more…

After almost a year of being told that we “have to interview Kaner” by previous Testing the Limits guests and readers, we exercised our listening skills and sought him out. With us this month to share his unique brand of wit and wisdom is Dr. Cem Kaner – author, lawyer, speaker, professor and one of the most respected minds in the testing world.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

Read more…

So maybe it wasn’t AT&T’s fault after all.

Apple recently revealed that there is a fundamental flaw in their method for calculating how many signal bars to display.  And we have the iPhone 4 (and its “learn to hold your phone the right way” fiasco) to thank for bringing this software snafu to light.

CNN Money shares the following details from Apple:

“Upon investigation, we were stunned to find that the formula we use to calculate how many bars of signal strength to display is totally wrong,” Apple wrote in a statement posted on its website. “Our formula, in many instances, mistakenly displays 2 more bars than it should for a given signal strength.”

That means, for example, that iPhones sometimes display four bars when they should be displaying two. Apple said users reporting a significant drop in bars when they hold their iPhone 4 are probably in an area of “very weak signal strength” but were unaware of that because the phone displayed four to five bars.

“Their big drop in bars is because their high bars were never real in the first place,” the company said.

Perhaps most surprising, Apple disclosed that the problem is not confined to the iPhone 4.  The faulty formula has been present in every iPhone model since the 2007 original.  Questions remain about whether the issue is strictly software-related, or if it also involved hardware problems.  However, Apple has said it will release a free software update in the next several weeks to fix the glitch. It will use a new formula recommended by AT&T.

“They want to know if the 91-shot Saturn Missile Battery really fires 91 shots,” writes Cory Matteson of the Lincoln Journal Star. “And, they want to know if 96,000 firecrackers in a pile the size of a bean bag chair will leave the lawn looking like burnt toast if they’re set off all at once.”

Much like software users, fireworks enthusiasts (like the guy in the picture) want a product that functions as expected; that works safely and without any unnecessary complications, regardless of the environment it is being used in. In short, they want a product that has been tested extensively before it was launched released.

So, seeing that it’s Forth of July weekend here in The States, I thought I would direct your attention to some testing fireworks – literally, in the form of this manual from the Consumer Product Safety Commission, and figuratively, in form of these recent (and provocative) software testing blog posts. Enjoy.

How Challenging Each Other Helps the Craft – James Bach
“Regular readers know that I’m dissatisfied with the state of the testing industry. It’s a shambles, and will continue to be as long as middle managers in big companies continue to be fat juicy targets for scam-artists (large tool vendors, consulting firms, and certain “professional” organizations) and well-meaning cargo cultists (such as those who think learning testing is the same as memorizing definitions of words and filling in templates).

What I can do about it is to develop my personal excellence, and associate myself with others who wish to do likewise. Someday, perhaps we will attain a critical mass. Perhaps the studious will inherit the Earth.”

The Heart of a Tester – Pradeep Soundararajan
“In 1954, when software testing was just about taking birth, there were two groups that started to form. I was as curious as you are right now, to know what those two groups stood for. One of the groups christened as, “Kuzusu”, had a thought that good testing would reduce the number of billable hours to deliver a good enough product and hence had to be avoided. The other group christened, “Shidachi”, stood for good testing that can save a lot of stakeholders time and money to deliver a good enough product. Things started getting hostile. People from the two groups tried killing each other.”

Read more…

Quality (pun intended ) software testing events are hard to find, but we’ve not only attended and spoken at some fantastic conferences around the world, but we’ve also simply asked around and received some great feedback in order to compile the Top Ten Testing Events.

Much like our Top 20 Software Testing Tweeps post, we need your help in letting us know if we’ve accidentally missed any good ones. Here they are in order of occurrence:

1. QUEST-Quality Engineered Software & Testing Conference (Apr 19-23, 2010: Dallas, TX)
2. Rapid Software Testing-By DevelopSense (Jul 5-7, 2010: Amsterdam, NL)
3. STANZ-Software Testing Australia/New Zealand (Aug 23-34: NZ & Aug 26-27: AU)
   
4. CAST-Conference of the Association for Software Testing (Aug 2-4, 2010: Grand Rapids, MI)
5. STAREAST (passed) & STARWEST-Software Testing Analysis & Review (Sept 26-Oct 1, 2010: San Diego, CA)
6. iqnite events-Next one in UK-formerly Software & Systems Quality (Oct 4, 2010: London, UK)
7. STPCon-Software Test Professionals Conference (Oct 19-21, 2010: Las Vegas, NV)
8. GTAC-Google Test Automation Conference (Oct 28-29, 2010: Hyderabad, India)
9. Expo: QA (Nov 16-18, 2010: Madrid, Spain)
10. EuroSTAR (Nov 29-Dec 2, 2010: Copenhagen, Denmark)

Have we omitted any noteworthy testing conferences you’ve recently attended? Please add your recommendations in the comments and they’ll be placed in the running to join the top events list. Maybe we can make this list a Top 15!

UPDATE: So far, some really great recommendations from our community include O’Reilly Velocity, Bangalore Workshop on Software Testing and VISTACON 2010 (the first Vietnam International Software Testing & Automation Conference).

Considering that a World Cup ref is once again under police surveillance (next stop, Witness Protection) as the result of a blown call, perhaps it’s finally time for FIFA to consider a long-overdue technology upgrade. May I suggest instant replay.

Or, if you’re one of those soccer purist types, maybe you’d prefer a ball that lights up like a slot machine when it crosses the goal line. Since my eyes still hurt from the NHL’s FoxTrax puck, I’m inclined to opt for the former, but what do I know?

Apparently very little, because a Mexico City-based development team is hard at work on a ball that would do just that. Aside from the aforementioned light show, it would also be able to “beam out TV replays” to prevent further refereeing gaffes. They have a point. I mean, if you somehow miss a TV replay of a DISCO BALL lighting up behind the net, perhaps refereeing isn’t for you.

Of course, many players are not too thrilled with the current Jabulani ball, which has been “praised” for its supermarket-like quality, so one can only imagine how this one would go over. My guess: not so much.

But if such technology is introduced into the Beautiful Game, it will only be after extensive rounds of real-world software testing. Here’s what the dev team is up against:

Read more…

Is there anything worse than doing your taxes? Apparently, for most software developers the answer is ‘yes’… testing software. And this issue is costing companies…often to the tune of hundreds of thousands of dollars for some of the more severe bugs.

Most tech execs and CTO types will tell you that having your developers test their own code — and not investing in proper testing resources — is a recipe for disaster. And yet inexplicably, some companies still go down this path.

We’ve all heard the numerous arguments before:  developers are too valuable to spend their time testing (eg: an engineer in Boston comes fully loaded at $120k); developers make lousy testers; these two separate functions should each be left to the experts.

Well Chris Matyszczyk from CNET (@ChrisMatyszczyk) says that, in addition to those arguments, there’s another reason to invest in proper testing:  most developers would rather do their taxes than test code.  And he’s got the stats to back it up:

Developers seem to be increasingly bugged by the agony of ill-tested software. All but 11 percent of the respondents cited either design defects, problems in test execution, or simply insufficient time spent on testing on all platforms and targets. And 58 percent named the latter two as the greatest evils.

More than half declared that the last significant software bug cost their companies an average of $250,000. So now, even I, a regressive in so many ways, see just how painful developers’ lives really are.

However, this research doesn’t seem to account for all the depths and nuances of pain. It gave respondents the option of choosing only the dentist, the fender-bender, or the taxes when expressing their dissatisfaction on, say, discovering that management won’t be investing in proper software testing or that sorting out the bugs is down to the developer.

I’d love to hear from the testers and the developers out there.  Should testers be responsible for their own QA?  Should they own a small part of testing (eg: unit testing only)?  Or should development and testing be separated and left to the specialists?  What say you, oh wise ones?

In this month’s installment of This Week In Testing, the date was May 1996 and the setting was the First National Bank of Chicago (insert dramatic pause here). The gist? Software “glitches” caused the bank accounts of 823 customers of the major US bank to be credited with a total of $924,844,208.32 each.

According to The American Bankers Association, all of $763.9 billion — more than six times the total assets of First Chicago NBD Corp. — was the largest error in US banking history.

And the reason given? Inadequate testing of course! The bank updated its ATM transaction software with new message codes. The message codes were unfortunately not tested on all ATM protocols, which resulted in some ATMs interpreting the codes as huge increases to customer balances.

This isn’t the first time we bring up banking bugs. You might remember Software Bugs: You Win Sum, You Lose Sum, the post about a man in Orlando who while making a routine bank transfer was shocked to see his balance at $88,888,888,888.88.

What other bugs have you recently heard or read about with such huge financial implications? Any mobile banking bugs?

In the third and final installment of our Testing the Limits interview with Lanette Creamer, we cover the Seattle testing scene; why more women don’t enter the profession; mobile testing challenges; test automation; her favorite Nicholas Cage movie and more. In case you missed them, here’s part I and part II.

uTest: The Seattle area has spawned an inordinate number of top testers (Whittaker, Bach, Bach, Creamer, et al) – what’s the deal with that?  Is there something in the water or is just a result of the Microsoft ecosystem being nearby?
LC: If there was no James Bach there would be no interview with a crazy redheaded tester named Creamer, because I would have no testing blog. If James Bach wasn’t in Seattle, I may not have had the chance to see him speak so often. Cast 2007 was in Bellevue, WA, maybe partially because that is close to Microsoft, so I guess in a roundabout way, it could be the Microsoft ecosystem being nearby that made Bellevue the location for Cast at the right time.

I prefer to think of it as something special about Seattle that fosters a unique perspective and resilience. Maybe it’s all of the cloudy weather. The grunge movement started in Seattle, and much like Nirvana, Pearl Jam, and Soundgarden, there are some innovative contrarians who aren’t afraid to blaze new trails coming out of Seattle to this day – and flannel is in style again.

uTest: Numbers-wise, the software testing profession is clearly dominated by dudes. Why do you think that is? How do we change this trend? Does it matter, or is this topic completely overblown?
LC: When all of the women who have the talent, skills, and desire to be testing are appreciated for the value they offer, and the field is still dominated by dudes, then great! It is about having the opportunity, not about enforcing some gender ratio. Right now things are not equal and fair for female testers and I’d like to see that change in my lifetime. I don’t think male testers are the problem at all. After a few curious looks, once we start actually testing or talking about it, in my experience, most testers are supportive and eager to help each other learn regardless of gender. The problem is higher up in the companies where the value testers bring isn’t well understood and diversity isn’t valued for men or women. The top reason we should care about diversity in our testing teams is because the demographic of a computer user is more diverse than ever before.

Read more…

No bugs were harmed in the planning of this Bug Battle… but they will be — starting Friday at noon ET.  That’s right uTesters — it’s time for our 2nd quarter Bug Battle competition!

And this time, we’ll be checking out the top location-based check-in services: Foursquare vs. Gowalla vs. Brightkite.  We’ll be letting our community put these innovators to the test (their public-facing websites, their check-in services, their mobile apps and their integrations with Facebook and Twitter).

And after the testing phase is complete on Monday, May 24th, we’ll be sending a survey to all participating testers to compare the usability and feature set of these three leaders.

What’s in it for testers?  Well, bragging rights, sure.  But we’ll also be doling out nearly $4,000 in prize money to the testers who report the most severe/interesting bugs and provide the most insightful survey feedback.

So it’s time to do what you do best, uTesters!  Log into your uTest account and scour these apps for quality defects and report them in a clear, concise way.  And if you do it better than your peers, you could be named the Q2 Bug Battle winner and earn some big prize money for your time.