Software Testing Blog

Dr. James Whittaker – the man who wrote the book on software testing – will be joining us for another uTest webinar on Tuesday, March 31 (1PM – 2PM ET).  Join us to learn five insights that will revolutionize your QA, including:

• The key difference between “experience” code and
  “infrastructure” code
• Why testers should play a greater role in development
• Why testing sometimes sucks

As a bonus, everyone who registers will receive a free e-book James wrote just for us.  You’ll learn more tips about how to manage and improve your QA process – including some details that aren’t available anywhere else.

Register today for this great webinar from James Whittaker!

Barbara Liskov, professor of computer science at MIT, was recently awarded the Association for Computing Machinery’s A.M. Turing Award.  The Turing Award is much like the Nobel Prize for computing research, and in this case it was awarded for her “contributions to practical and theoretical foundations of programming language and system design, especially related to data abstraction, fault tolerance, and distributed computing.”

Professor Liskov was the first woman in the United States to receive a PhD in computer science, and she went on to develop new ideas in using abstraction in programming.  For software testers, however, she has also developed new techniques for improving fault tolerance in distributed computing systems.  From Dr. Dobb’s:

Her most recent research focuses on techniques that enable a system to continue operating properly in the event of the failure of some of its components. Her work on practical Byzantine fault tolerance demonstrated that there were more efficient ways of dealing with arbitrary (Byzantine) failures than had been previously known. Her insights have helped build robust, fault-tolerant distributed systems that are resistant to errors and hacking. This research is likely to change the way distributed system designers think about providing reliable service on today’s modern, vulnerable Internet.

Her ideas will no doubt influence how distributed, grid, and cloud computing systems are both developed and tested.  Congratulations to Professor Liskov for winning such an amazing prize!

Like many Americans, I’ve been watching housing prices very intently.  While looking at data on the website for the Massachusetts Association of Realtors, I came across a very common and ordinary web bug: a broken link.

(Click the thumbnail for a full sized version.)

Now broken links are incredibly common, and anyone with a website has had one happen to them at some point.  But I want to dig a little deeper.  Take a look at that screen shot and think about how many bugs do you see?

By my count, I see three very real bugs:

• There’s the broken link.  Looking at this error, I think this is a convoluted way of saying that the file I wanted wasn’t found.  Whoever uploaded the file didn’t set a folder correctly.
• The Folder is not set exception didn’t get handled.  Ideally, some kind of exception handler should have caught this exception and provided the user with a more coherent error message.
• The user was shown an incoherent error message.  An end user should never see this kind of error message in the first place.  It reveals information about the underlying system configuration that could be dangerous.  At the same time the user doesn’t receive any kind of friendly help about why the error occurred or how to resolve it.

Each of these is a very real bug, but each one is also more difficult to fix than the previous.  For example, fixing a broken link may be as simple as changing a file location in the content management system.  The person responsible for posting the content could take care of that.

Correcting the next bug requires help from the developers to write a proper exception handler to catch this exception. While fixing a broken link is something that could be handled in the field, writing a new exception handler usually has to be done during the development process.  This, of course, requires a solid QA process to be in place during development.

For most projects, the last bug is the most challenging to fix.  It usually requires careful planning from the earliest stages by product management, development, and QA to agree that reducing the complexity of bugs is both important and feasible.  Software testers will have to be involved early in the process so that they can understand how to discover and interpret bugs that are designed to be customer friendly or even invisible.

Even simple bugs, like broken links, can lead to deeper questions for testing and development teams.  Not every bug has to be fixed, but make sure to keep your customers and users in mind when planning your project.  Good planning could turn a boring broken link into something that’s both apologetic and helpful.

If you or someone you know is struggling to develop iPhone apps, this recent item from TechCrunch may provide relief:

At the recent Future of Web Apps conference in Miami, Y-Combinator-backed 280 North announced Atlas, a drag-and-drop visual editor for building desktop web applications with Cappuccino, 280 North’s Javascript-based framework…Atlas can wrap up iPhone Web Apps like native applications, granting them access to a significant portion of the iPhone API and allowing them to be sold through the App Store.

This lowers the barrier of entry for iPhone development substantially, allowing those with Javascript knowledge to create fully functional applications on the platform without requiring them to learn a whole new language. The same limitations that apply to Javascript apply here, presumably – in other words, don’t expect to be throwing down ultra-rich OpenGL-based 3D games, but mid-range apps (such as Twitter clients, RSS readers, etc.) should be completely doable.

I’ve gathered the significance of this from a developer’s perspective, but I’m curious as to what the testing community thinks, and three questions immediately come to mind:

1. Will this complicate the QA process for these apps, make it easier, or have no affect at all?

2. Besides Twitter clients and RSS readers, what other types of mid-range apps should I expect to see?

3. Will the lower barrier to entry result in lower quality apps, or just the opposite?

Leave a comment and let us know what you think.

On Slashdot there’s a discussion about how websites are actively trying to convert IE6 users to a more recent browser.  Figuring out how to convince Internet Explorer 6 users to upgrade to a more recent browser is a good topic because supporting IE6 is both expensive and time consuming given its many idiosyncrasies and poor adherence to standards.  I suspect more and more websites will try to help their users upgrade to IE7, IE8, or Firefox.

However, people looking to convert their own users should keep a few things in mind:

1.) Many people using IE6 don’t know how to upgrade.  This includes everyone who either doesn’t know how to run Microsoft Update or is simply too afraid to go download the right software.  I think of people like my grandmother who runs an out of date copy of Windows ME and refuses to upgrade to anything newer.  People like this need lots of hand holding, and for many of them IE7 may be too much for their out of date PCs.

2.) Other people may not be allowed to upgrade.  IE6 is a standard at many large companies, and overzealous IT departments may fear trying to standardize around IE7 or Firefox.  What’s worse is that many legacy web tools were written specifically for IE6, and the companies responsible may no longer exist making upgrading impossible.

Legacy web browsers will always exist, so web developers looking to make sure their products work for all of their customers will probably be doing cross browser testing for a long time to come.  Still, encouraging people to upgrade to a more recent and more secure web browser is a great goal.

Are you still running IE6?  I would love to know why – tell me in the comments below.

How important is a test plan for your software project?  Really important!  Whether your projects are small or large, having a well thought out test plan can be the most important tool to improve quality for your project.  Unfortunately, many companies overlook how important a test plan is to their testing process.

So what is a test plan?  Roughly, it’s the strategy behind your testing.  It contains a list of things to test, what not to test, who’s responsible for testing, pass and fail conditions, etc.  Depending on your organization, you may want to track costs, approval pathways, deliverables, and more.  In fact, there’s even an IEEE standard for test plans and documentation – IEEE 829.

For many engineers, test planning looks easy – so easy that they can track it all in their heads.  But solid testing is like solid engineering, and that means having good plans up front.  A good test plan tells you:

• Are you spending your time and resources testing the right things?
• What constitutes a bug?
• Which bugs are the most important to find?
• How much coverage do you need?

The payoff is even more important.  With a good testing plan, you can release your product with far more confidence than you could otherwise.  Consider the costs of missing out on bugs: if having a test plan makes it possible to find 10 extra bugs that you would have missed, and each of those bugs costs you $1,000 to fix after releasing your product, then you just saved $10,000!

Bugs are always embarrassing, so knowing that you’ve thought through your testing can help you be more confident about your product and sleep better at night.

It’s always fun to see how different kinds of engineers test what they do.  For example, acoustics engineers must test their concert halls – something that’s especially important for classical music halls where the performers are not equipped with microphones.  In New York City, the venerable Alice Tully Hall in Lincoln Center has been undergoing renovations for the past couple of years.  Now the construction is nearly complete, and this recent article from the New York Times describes some of the acoustics testing taking place.  It’s surprising how many parallels there are to software testing:

Functional Testing – The new hall has movable stages, remotely controlled noise dampeners, sophisticated noise shapers, and more.  Each of these needs to be tested to make sure they work in a variety of conditions including with multiple instruments, performers, and performer locations.

Beta Testing – It’s not enough for the acoustics engineers to say the hall works as expected.  Actual audience members must provide feedback to make sure the hall sounds great from every seat.  For good results, the audience needs to remember that they’re not just enjoying a performance, they’re there to help the engineers refine their design.  That means sitting through multiple stage adjustments and providing useful feedback at each step.

Load Testing – Load testing the acoustics of a performance hall may seem odd, but there are big differences between the acoustics of an empty hall and a full hall.  People – bodies that is – absorb sound by simply sitting in their chairs.  Good acoustics testing requires filling the hall to make sure it sounds just as great with a large crowd as it does with a small crowd.

Apparently the feedback has been outstanding for the new Alice Tully Hall.  Hopefully once it reopens, New Yorkers will be proud of their newly renovated concert hall.

Buggy products can be a real customer turn-off.  Witness the recent release of the BlackBerry Storm, Research In Motion’s response to Apple’s iPhone.  The Storm had tremendous promise as a great new touchscreen phone, but customer response has been limited because of early bugs in the device’s software.  Here’s a quote from one Storm customer in an article from the Wall Street Journal:

“I found myself wanting to throw it in the ocean due to my frustration with its overall usability,” said Steven Golub, a longtime Verizon customer from Morristown, N.J., who bought the Storm the day it was released, but returned it a few weeks later.

That’s pretty damning, but let’s stop and give RIM some benefit of the doubt.  Buggy software is a customer turn-off, and undoubtedly bad reviews will dampen customer enthusiasm.  But here’s a quote from the same Wall Street Journal article:

Verizon and RIM, determined to release the Storm in time for the holidays, rushed the device to market despite glitches in the stability of the phone’s operating system, according to people close to the launch.

RIM had a choice to make – release a buggy product in time for the holidays, taking bad reviews on the chin, or wait until the Storm worked better and try to gain market share during the traditionally slow Q1 made even slower by a bad economy.  We really can’t know for sure if RIM made the right decision, but there are some lessons other companies can learn here:

1.) Know Your Limitations - We all have limitations in our planning: not enough time, not enough people, or not enough testing.  Maybe your budget is limited or maybe you need to hit a holiday launch date.  Either way, it’s very important to be aware of your limitations early in the process.  If you have a tight schedule and a firm launch date, then you should make sure you have enough resources for last-minute development and testing.  If you don’t have enough people, then you should evaluate different contingencies for alternate launch dates.

2.) Work Smarter – You may feel like you have to cut corners to overcome your limitations, but the Internet can make it easy to expand your efficiency on-demand.  There are many companies that can help you add capacity to overcome bottlenecks and release a better product on time.  The uTest software testing service is perfect for helping companies improve their testing, but solutions exist for everything from coding to graphic design.

3.) Have Good Customer Data – If you’ve already passed the point of no return, then you will have to make a tough decision.  In this case, nothing helps more than really good customer data.  In RIM’s case, they had to decide between weaker customer demand because of timing or because of bugs.  Between the two, they chose to cast their lot with a buggier product over a late product.

RIM still sold 500,000 Storms over the holidays, and that’s not bad.  This is also their first touchscreen device and it holds tremendous design potential.  While Storm Version 1 had a rocky launch, Storm Version 2 may take the world by, well, storm.

SANS recently published a list of the 25 most dangerous programming errors.  The list is pretty comprehensive and covers things ranging from not validating input to using insecure cryptographic algorithms.  This list is interesting not because the bugs are common or difficult to test, but rather because each of these bugs can produce catastrophic consequences if left unfixed.

Any software tester should find this list interesting, but one might want to ask if this list is always important?  Bugs that impact security are relevant to Internet applications, but are they relevant to other things like games?  The answer is, as always, “it depends.”  However, poor security can be a problem in all kinds of places.  For instance, a bug could allow access to protected media in a game or make it easier to compromise other related applications.

Good testers should always look out for bad security.  Even if the danger is low in one particular product, testers can be very influential in helping developers improve their coding and protect against more serious bugs in other products.

On Mashable, there is a great post discussing the 10 Most Common Misconceptions About User Experience Design.  The writer is a user experience designer, and she makes the point that user experience is more than turning out a nice looking application – it’s a process that requires good focus by everyone involved from the developers to decision makers.

As testers, it becomes very important to understand the strategy behind the user experience as well as the actual interface itself.  Get to know what drives the user decision making behind an application and look for ways that the application could be improved from a strategic point of view rather than just a best practices point of view.  For example, a program written for the elderly might follow all of the Microsoft or Apple user interface best practices, but what it really needs is just larger fonts.

As uTesters, the great thing is that in many cases we are the customers for the software we test.  Or we live with the customers or we know the customers.  As a crowd, we know a large number of people that individually we may never meet.  From a testing point of view, I think that gives us a whole new range of insights into what makes for a good user experience.