Software Testing Blog

In the wake of the IMF’s recent hack, the world’s top cyber-security experts have come up with a sure-fire solution to prevent this from ever occurring again: bugless software.

Don’t laugh, this was the consensus that emerged from the third annual cyber defense conference hosted by NATO last week, where a top IT expert was promoting the idea that companies should “pay hackers who have detected program bugs.”

I would propose that companies pay testers, not hackers, but that’s beside the point. The real issue here is the notion of bugless software. More on that in a second, but first, a few details from the conference courtesy of The Montreal Gazette:

Bugless software is key to cyber security and global vendors should pay hackers who have detected program bugs and so helped pre-empt attacks, a top IT expert told a NATO cyber security meeting here Thursday.

“Software vulnerabilities enable breaches. If we want to make cyber space safer, we need to find a way to force vendors to produce more secure software,” Charlie Miller told some of the world’s top IT security experts attending NATO’s third annual cyber defence conference.

Renowned in IT circles for having detected bugs in Google’s Android software and being the first to find a critical bug in the MacBook Air, Miller has also worked as computer security specialist at the U.S. National Security Agency.

He charges that the reluctance of software vendors to pay hackers for weeding out program bugs is a factor contributing to online security breaches.

“Vendors don’t like to pay for (the detection of) bugs because of many reasons, among other things they’re afraid it will encourage people with good IT skills to find bugs, or it may make competitor’s software look more secure,” he said, noting that companies like Mozilla and Google have paid up to 3,000 dollars for information on software flaws.

Read more…

We’re happy to announce another new series to the uTest blog (in case you missed it, we introduced the Tester Challenge last week). This series will be devoted entirely to usability testing, and will be hosted by uTest’s very own UX expert, Inge De Bleecker. We’re still working on a series title, but for now, you can find it under the category of Usability Testing.

There are a number of different ways to run a usability test, but all usability tests should consist of the following 3 steps:

1. Test participant recruitment: a usability test is a test of the product or application using people that are representative of the target population for the application or product. For instance, an application that helps runners keep track of their progress should be tested by runners, and an application that allows mothers of young babies to log information about their babies should be tested by new mothers.
2. Task plan creation: a usability professional identifies the most important actions that users are expected to perform when using the application, and creates tasks that reflect these actions. For instance, on a web site that allows users to create photo collages, some of the actions include uploading pictures, choosing a collage template, organizing the pictures and saving the collage. Those actions are turned into tasks that are aggregated in a task plan. Each test participant is asked to complete all tasks in the task plan. After each task, the participants are asked some questions about the task. The questions are carefully crafted in order to gather as much relevant information from the participants as possible.
3. Data analysis: once all participants have completed the usability test, a usability professional draws on his or her background in interface and interaction design, cognitive science, and statistics to analyze and aggregate the data. The result is a usability report that includes a summary of the data as well as design recommendations.

As a tester, you may be asked to do usability testing. Unless you have a group of target users hanging around, and you yourself are a usability professional or you know one sitting around the corner, you’ll be hard pressed to run a true usability test.

It’s always a good idea to explain to those who ask you to do usability testing what usability testing entails. Chances are that a true usability test does not fit within the project parameters. Even so, you can still provide value and contribute to improving the usability of the application or product by doing some ‘exploratory usability testing’. This type of testing can bring to light at least some of the more prominent usability issues.

And now, some tips for exploratory usability testing:

Read more…

The Uruguay IT Chamber has launched a new university program for software testing.  This is big news for the world of software testing, as it not only legitimizes the professional tester but will help draw attention (and projects) to the importance of testing.

TestingReflections.com has a nice write up on the news release, but it’s particularly interesting to see that there are three tiers of testing, as follows:

Software Tester: The ideal program for entry-level, junior, and career-switching software testers.  Successfully completing this credential will give you the knowledge and experience most employers expect from testers with 1-2 years of on-the-job experience – effectively enabling you to start your career needing only to acclimate to the specific expectations of the employer and demonstrate your skills at work before being acknowledged as a mid-level tester.

Software Testing Professional: For testers with several years of testing experience on-the-job who are looking to make the jump from mid- to senior-level, this program is designed to teach high-quality individual contributors how to be effective technical leaders within their testing organization.  Effectively giving the tester the new skills they need, in addition to their existing hands-on testing knowledge, to prepare them for the additional responsibilities of a technical testing manager or of a manager of small to mid-sized testing projects.

Software Testing Leader: This final step in the program has been designed for senior-level testers who desire to be successful managers or directors of corporate testing programs.  Frequently the most challenging step for career software testers is to transition from being a technical leader to a management role focused on the interface between quality testing and executive-level business value.  Successful completion of this stage of the program will give the student the tools they need to make the jump from technical leader to manager – opening the door to further advancement to positions like “Director of Testing Services” or “VP of Software Product Quality”.

Full article after the break.

Read more…

Robert Scoble just published his list of “Must Have iPhone Apps” on Quora. It’s well over 100-apps long but it’s pretty exciting to see so many uTest customers listed (dare I say that it reads like a “Who’s Who of uTest Customers”?)

It’s pretty extensive but we know there are more “Must Have Apps” out there – for me it’s…

• Nike+ GPS – best run tracking app yet with a lot of social interaction (i.e. play “tag” against friends, get live cheering from Facebook friends). Plus it’s compatible with DailyMile.com which is my go-to exercise tracker.
• TWC (The Weather Channel) – It’s the best weather app I’ve seen yet.
• Weber’s On The Grill – its summer.  How can you not have this downloaded?
• ESPN’s FFL app – With the pending lockout this might be pretty pointless come fall but I’m optimistic that I’ll have a chance to take some friends money.
• HBO Go – Free access to all HBO content for existing HBO customers means I watched all Entourage episodes in a couple weeks on the iPad. Curb Your Enthusiasm is next.

Scoble’s list after the jump. You know you want to see it….

Read more…

Quick – what’s a URL? Most of you would point to that string of text at the top of your browser that defines the location for this page. But URLs represent a lot of things: references to pages, pointers to content, and the foundations of links. They’re the fiber of the web, and the entire notion of HTTP is about pages pointing to pages using URLs.

However, a new approach to building websites is threatening to turn this notion of URLs on its head. Two simple characters – #!  (called either a hashbang or a shebang) – are creating more trouble than anything seen in years. Adding those to a URL makes it something else entirely, but to understand why we need to first go over a couple of web fundamentals.

Read more…

Luminary: a person who has attained eminence in his or her field or is an inspiration to others

You’d be hard-pressed to find a profession with a wider range of ideas and personalities than that of software testing. This point is certainly not lost on our readers, as evidenced by the popularity of our Testing the Limits interview series. And it’s not lost on our good friends at Software Test Professionals, who have opened up nominations for the 2nd Annual Software Test Luminary Award.

More on the nomination process in a second, but first, a little bit about the award itself:

The Luminary award will honor any software testing and quality assurance professional who is determined, persistent, and committed to improving a process or methodology. They develop ideas, which when properly applied, have a positive impact on the end product, either by enhancing quality or performance or resulting in improved efficiencies for a particular process, team or organization. In addition, their contributions elevate the critical role of the software test profession within the software development process.

A luminary is someone who has inspired others by their actions and the results of those actions on the profession. They inspire others to pursue a software testing career. It is about how they have given back, and shared their knowledge and experience with others in order to advance the profession and improve the career paths of all practitioners. A luminary will typically be recognized and respected long after their days of practicing have ended.

If you recall, last year’s honor went to Gerald M, Weinberg, who edged out fellow nominees James Bach and Cem Kaner.

So who will be named this year’s Software Test Luminary? It’s your call. STP will gather nominations and submit the top 3 candidates for a final round of voting. The finalist will be announced at the Software Test Professionals Fall 2011 Conference, October 24-27 in Dallas, Texas.

Here’s a quick timeline of the events:

Read more…

With all the talk about the end of the world (hope you didn’t make plans this weekend), it seems fitting that we’d find an article proclaiming  the end of manual software testing. That’s right, a researcher from the University of Twente has developed a system that claims to eliminate the need for manual software testing. In other words, it eliminates the need for YOU! I’ll let that sink in….

The system is said to facilitate quick and accurate testing – and here’s how it supposedly works (via Forbes.com):

The testing phase for new software consists of three steps: developing the tests, running the tests and evaluating the results. These three steps are generally performed manually. Model-Based Testing is a method that automates all steps in the software testing process. When used properly, the method completely eliminates the need for manual software testing.

Model-Based Testing has a number of major advantages: it makes the software testing process faster, cheaper and more accurate. It is not uncommon for manual software testing to take anywhere from several months to years. Van der Bijl’s new system can significantly reduce the duration of the testing period and thus reduce costs. “We can reduce the duration of the testing phase by at least thirty percent. We were even able to reduce overall software development time for one of our customers by a factor of four.” Model-Based Testing is more accurate, because in principle there is no limit to the number of tests you run, says Van der Bijl. “If you want, you can even run a million tests.”

So is this really the end of manual software testing? I suspect you know the answer to that question. If not, here are a few great responses on this debate from StackOverFlow.com:

Read more…

“Technology is everywhere in every fabric of our lives. With technology, as ubiquitous as it has become, everybody will gain or fail in much the same broad way as the next person or company. Nobody is exempt.” – Jeff Papows, Author of Glitch: The Hidden Impact of Faulty Software

As Jeff said (in his Testing the Limits interview) nobody is exempt from the impact of faulty software. Not developers. Not testers. Not immigrants. Not gas station owners.

We spend most of our time on this blog discussing the impact of software bugs on testers and developers, but a few recent news stories show us how they affect the latter (i.e. immigrants and gas station owners). Take a look…

Computer glitch causes gas station to sell 8,000 gallons at $1.10 each

When residents of Wilmington – a neighborhood of Los Angeles, CA – got word that a local Valero fueling station was selling gas for the bargain price of $1.10 a gallon, almost immediately, vehicles lined up ’round the corner. This wasn’t one of those promo deals on fuel though. It was a computer glitch that lead to the station dispensing some 8,000 gallons of gas at just $1.10 each.

When station owner Kenny Nguyen got wind of his unintended act of kindness, he immediately shut down all of the pumps. All told, Nguyen estimates that he lost nearly $21,000 due to the glitch. Ouch.

Computer glitch voids green card lottery results

Tens of thousands of would-be immigrants to America are sure to be disappointed after a computer glitch prompted the State Department to invalidate results of the most recent green card visa lottery.

Read more…

Last week, I decided to go back in time to revisit a classic work of software testing theory by James Bach, on the subject of risk-based software testing. What I tried to show was that despite tremendous advances in terms of tools, techniques and technology, the fundamentals of software testing essentially remain the same. I hope that was conveyed in the post.

Anyway, in that same spirit, I’m going to quickly bring back another classic work of software testing theory for debate and discussion: Bug Advocacy by Dr. Cem Kaner.

If that name sounds familiar, it’s because Cem Kaner is one of the field’s leading thinkers, teachers and practitioners. He’s also appeared as a guest on our Testing the Limits interview series, which you can find here, here and here.

So what is “Bug Advocacy” all about? The title pretty much says it all (I hope), but let’s take a look at some key excerpts from this 100-page masterpiece to find out more. First the, the premise:

1. The point of testing is to find bugs.
2. Bug reports are your primary work product. This is what people outside of the testing group will most notice and most remember of your work.
3. The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.
4. Programmers operate under time constraints and competing priorities. For example, outside of the 8-hour workday, some programmers prefer sleeping and watching Star Wars to fixing bugs.

A bug report is a tool that you use to sell the programmer on the idea of spending her time and energy to fix a bug.

Motivating the Bug Fixer
Some things that will often make programmers want to fix the bug:

• Read more…

Here’s a funny (and honest) infographic on the love-hate relationship between software testers and developers. It was created by Bruce McLeod, author of  the Teknologika blog. He also wrote an accompanying blog post explaining this image in more detail.

Well done Bruce!