Software Testing Blog

Want to become a better tester? These ten steps will get you there – guaranteed!

1. Test.
2. Test more.
3. Test even more.
4. Test even more than that.
5. Test when you don’t want to.
6. Test when you do.
7. Test when there are lots of projects.
8. Test when there are not.
9. Test every day.
10. Keep testing.

This list was adapted from Brian Clark’s 10 Steps to Becoming a Better Writer (link). So what advice do YOU have for becoming a better tester? Let us know in the comment section.

In the second installment of our Usability Testing series, Inge De Bleecker offers some valuable advice for getting to know your users. Enjoy! 

There are many aspects involved in making a user-facing product successful; one of them is to know your user.

Companies developing a new user-facing product often aren’t sure who their target user is, or they attempt to target ‘everybody’. Companies think that by thinking of ‘everybody’ as their audience, the product will sell better, since there are more bodies to sell the product to.

When it comes to designing a solid product user experience however, ‘everybody’ is not a good target user. It is not feasible to design a product so that it is user-friendly, appealing, and easy to use for everyone. The result of such effort may very be a product that doesn’t appeal to anybody in particular; and this in turn may make the product unsuccessful.

Sometimes companies have a vague idea of their target user based on customer input, assumptions, or theoretical data. User research can be used to validate these vague ideas, and further define the details.

It’s always fun and engaging to do user research for a completely new product. Startups often have novel product ideas, yet knowledge about the end user is often based entirely on assumptions. The goal of user research is to identify any and all target users groups and learn about their behaviors and expectations so that the product functionality and design will be appealing to those users. On a number of occasions, I have found that user research has brought very interesting insights to the table. I’ll describe one example below.

A couple of years ago, a few smart people had an idea for an intelligent application that could help users with shopping. The application was thought to primarily be a voice-driven mobile client with a secondary web-based client.

We defined a number of assumed target user groups, and set out to do a round of interviews with people who fit those groups, as well as people who specifically didn’t fit those groups. We wanted to learn about their current behavior related to shopping, and wanted to see if they might be interested in using an application such as the one that was going to be developed.

Read more…

Readers of this blog know that we keep a watchful eye on software glitches in the news, as hardly a day goes by without some type of software malfunction wreaking havoc on unsuspecting users. Some weeks, the stories are worse than others – and this is one of those weeks.

That said, here are a few of the software bugs making headlines this Summer:

Glitch leaks private data from DNA company

Private information involving 800 Australians has been accessible online due to an apparent software glitch at one of Australia’s largest DNA testing companies. Adelaide-based Medvet Laboratories carries out paternity and drug tests and is owned by the South Australian Government.

The Government says an apparent software glitch on the company’s website has been leaking names and addresses since Friday. Experts say the lapse has the potential to open the company and the Government up to compensation claims. SA Health chief executive officer David Swan says the private details have now been removed and an investigation is underway.

Computer Shutdown Alarm Wakes Shuttle Crew

The shuttle Atlantis crew was fast asleep last night when a computer shutdown triggered an alarm, jarring the weary astronauts out of their sleeping bags. Shuttle commander Chris Ferguson and pilot Doug Hurley worked for about 45 minutes loading software onto one of the shuttle’s other main computers, then went back to sleep.

The astronauts, who had been asleep for about 1.5-hours when the alarm went off, got their bearings pretty quickly and floated up to the flight deck to begin trying to fix the problem.

The computers are so critical to the shuttle’s operation that NASA flies with five of them. Engineers are trying to figure out if there’s a software bug or if the shutdown was just a transient hardware glitch, triggered perhaps by radiation or electronic interference of some kind.

Read more…

Would you trust a plumber to design and decorate the interior of your new home? How about an auto mechanic to paint your car? Or a dermatologist to perform your root canal? If you answered no to any of these questions, then I would argue just as strongly that functional testers should not be handling usability testing for your software.

To be absolutely clear, there are some talented functional testers who are gifted in usability testing as well. However, this is typically the exception. The point of this blog post is not to minimize these exceptions, but rather to highlight the differences between these two testing types and point to the necessity of having one set of experts test your software for functionality, and another set of experts test for usability.

So why do we need separate experts? Here are several noteworthy observations:

• Different fields of study: In general, usability testing experts come from non-technical fields of study – psychology, cognitive science, human behavior, marketing, etc.
• Different skills required: In general, functional testers have more in common with engineers and scientists while usability testers have more in common with artists and psychologists
• Different focus: In general, functional testers focus on “how can I break this” whereas usability testers focus on “how intuitive is this for the end customer”
• Different persona: In general, functional testers do not need to fully understand how an end user would use the software whereas the usability tester needs to study and emulate end users
• Different departments in the workplace: In general, usability testing lives in the product management and marketing department while functional testing lives in the engineering department

Given these differences, in particular regarding fields of study and skill sets, I would recommend against entrusting your usability testing to the typical functional tester (the opposite is true as well – entrusting your functional testing to the typical usability tester could be disastrous). There is a particular art and science to usability testing that requires an entirely different approach and mindset. So from a company’s perspective, it’s best to separate these roles and recognize their distinct value propositions.

Read more…

Every year, the fine folks over at Common Weakness Enumeration (CWE) team up with the SANS Institute, MITRE and other experts to publish a list of the 25 most dangerous software errors. The best thing about the list (aside from the fact that it’s free) is that it offers something for everyone, including developers, project managers, customers, and yes, even software testers.

You would be wise to read the report in its entirety, but I wanted to highlight a few key points and give you a preview of the errors that made this year’s list. To start, here’s an important excerpt written for the software customer (i.e. end user):

Recognize that market pressures often drive vendors to provide software that is rich in features, and security may not be a serious consideration. As a customer, you have the power to influence vendors to provide more secure products by letting them know that security is important to you. For the software products that you use, pay close attention to publicly reported vulnerabilities in those products. See if they reflect any of the associated weaknesses on the Top 25 (or your own custom list), and if so, contact your vendor to determine what processes the vendor is undertaking to minimize the risk that these weaknesses will continue to be introduced into the code.

Of course, software testers play a key role in minimizing this risk – a point that is certainly not lost on publishers of this report, who have included dozens of technical links and tips for testers who want to find such errors.

So what are the worst software errors of 2011? Glad you asked. Here are the top ten (after the jump)

Read more…

“Every day, someone come along with a new programming language and the hype machine begins. This time, I won’t be fooled. Give me one good reason… why would I try using jQuery?”  Such  comment are not uncommon among web developers – including many who resist. particularly when it comes to adoption of new technology.

jQuery is a perfect example. There’s a lot of buzz around this language, but how does it measure up in the real world? A better question would be: When would a dev want to consider using jQuery?

For those who aren’t already familiar, jQuery is an open-source library based on one the powerful client-side scripting language, JavaScript. I used Action-Script 3.0 (scripting language used for Flash animation) at an earlier point in my career and my first crush on jQuery came from the power of doing more with less code. Powerful animation effects can be achieved with few lines of code in jQuery, whereas in some other languages the compiler will suck the last drop of your computer’s resources for doing basic animation.

I have seen lot of other cases where jQuery simply rocks, such as its ability to talk to any DOM (Document Object Model) element and its powerful event (mouse click etc.) handling engine. For rendering any HTML page, the web-browser’s rendering engine parses HTML elements (like anchor <a>, paragraph <p> etc) into DOM. Most of the web browser’s rendering engines are written into the C++ programming language. With jQuery, you can access any element within your DOM and do your client side logic on to that element which gives lot of power to developer.

We might or might not be able to see a language better than jQuery in next decade, but their philosophy and the way their followers are increasing is pleasantly surprising. Their vision of “Write Once & Use Everywhere” and “Write Less and Do More” seems impressive. The jQuery team recently launched jQuery mobile which will help in developing your mobile web-app within minutes.

Read more…

We’re just about halfway through the year but I’m calling it now: 2011 is the year of the hacker. Grim?  Maybe.  Just about every week there has been a new story about a company being hacked and it’s costing companies millions of dollars and even more for their brand reputation.

While only two of these hacks really impacted a company I use heavily, I thought I’d do a quick countdown on the top hacks of 2011 and the associated costs.

7) DropBox
The file-sharing site opened the doors for four hours this week, allowing anyone with a login to access other accounts.  It turns out that it was a self-inflicted wound and DropBox broke their own authentication system.  While the finacial impact probably won’t be released, just browse through the 600+ customer comments to see how the issue and their response impacted their brand.  It’s a bug, not a hack, but certainly something that could have been avoidable with ample testing prior to a full launch.

Responsible: Themselves.

Cost: A self reported “much less than 1%” of their more than 25 million users were impacted to an undisclosed extent.

6) MovableType / PBS.org
In a pure retaliation a group of hackers targeted PBS.org in response to an episode of Frontline’s portrayal of of WikiLeaks leaker Bradley Manning.  The hackers gained control of PBS.org and republished false information.  PBS was not able to immediately regain control and was forced to utilize their Facebook page as their primary news source.

Responsible: LulzSec.

Cost: One of their Sr. Correspondents, Judy Woodruff, wrote a post on “Calculating the Cost of an Attempt to Silence the Press”.  While they didn’t disclose any financial costs or specific user information loss, it has certainly been a struggle for them to regain control of their site and all of their content.

Read more…

Depending on who you ask this question to, you’re likely to receive various degrees (pardon the pun) of yes and no. Or you may find many others who answer in a noncommittal way: “it depends.”

Having worked closely with thousands of software testers in the uTest community, I can attest to the fact that many testers do in fact have impressive resumes with regard to higher education (master’s degrees, PhD.s, etc.). However, there is also convincing evidence that demonstrates quite the opposite. So if you let the data speak for itself, what is one to believe?

We recently had a lengthy discussion on this topic in the uTest Forums (here’s the direct link to the thread). Below are a few explanations from the various camps.

No, a college education is not needed:

• It “helps” to have a relevant college education, but it’s not necessary to have one. Other factors are equally important, including natural talent and the character of a good tester (Ray, from The Netherlands)
• As a company, you want to have people from various backgrounds testing your product. Having some testers who do not have a college education may provide a different perspective that’s useful for testing (Connor, from the US)

Yes, a college education (or higher) is needed:

• My experience through undergraduate and graduate programs have allowed me to build up my testing skills: systematic thinking, being able to focus on the details and the bigger picture, attentiveness and accuracy, root problem analysis, soft skills, etc. (Roman, from Russia)
• In my experience in working with colleagues, those who have a college education nearly always outshine those without. A college education provides both a broad and narrow-level of exposure to the outside world that is difficult to emulate through self-taught learning (Manal, from the US)

It depends:

• If you’re trying to “break” into the field as an entry-level tester, a college degree is not necessary. In general, the barrier to entry is very low, especially when compared with requirements to become a doctor, nurse, or pilot (Shane, from Australia)

I would tend to agree with Shane’s point of view, so I’ll continue quoting him:

Read more…

After the conclusion of this year’s Apple WWDC conference, Gene Munster of Piper Jaffray released the results of an informal survey he performed among conference attendees who were also iOS developers. In it, he asked them what their plans were for developing on different platforms, including Apple’s own OS X. The results were surprising.

iOS developers love iOS (of course), and as recently as 2008, 50% of them were also OS X developers. But today, that percentage has dropped to 7%, and most iOS developers are now actively developing for other platforms instead (including the iPad). This makes a lot of sense – the skillset for developing a mobile application has become more and more specialized, and the developers who can do that well may not have the skills or interest in developing for a desktop platform.

But the data holds other clues as well. For example, almost half of iOS developers also develop for Android. And even though all the developers think iOS is the best platform for monetization (they were attending WWDC after all), 40% of them thought Android was the platform with the greatest potential for future growth. By the way, that question included iOS as an option as well, meaning that 40% of iOS developers attending WWDC actually thought Android was going to grow faster than iOS.

What other platforms did these developers think would have any chance of growing in the coming years? The only other one that made the list was Windows Phone 7 with 9% of respondents. That’s small, but interesting. Microsoft could have something good on their hands.

More details from Fortune and Macrumors.

Remember the dial-up modem? I’ve been speaking with some mobile testing tool vendors recently, and I just can’t stop but think back to the 90s. Let me explain.

For those of you who remember (yes, I’m showing my age), in the mid-90s when PCs were just becoming mainstream (Windows 95!) and consumers started adopting internet at home (AOL!), a bunch of us were busy building web sites -  static brochureware, personalized portal web sites, e-commerce sites…. I spent a lot of time with my team optimizing those web sties to make the user experience better.

And it was hard. I remember having tons of late night discussions with designers and developers trying to balance rich imagery and page load times. Serving the right image file with the right resolution, reducing network traffic, figuring out the right caching strategy, and all the tricks that we thought up and shared with colleagues – ahh, those were fun days.

Fast forward to the modern day web. Super-fast PCs rivaling (or surpassing) the supercomputers back in the day… Fiber optics to the home, allowing people to watch HD movies and play real time networked games with friends without noticeable latency… Optimized Javascript engines with JIT compilers letting you run console-type games in a browser… Now I know I would be offending a lot of people if I say “web developers are getting lazy” – but all this more powerful, faster environment means that we can get away with not having to think too much about figuring out how to reduce all waste. But that’s okay, we should be focusing the more important things – user experience flow, time-to-market, the value to the user.  But this approach is taking a toll in the mobile web app world. Web developers have taken for granted a lot of assumptions that don’t apply in the mobile web world.

Read more…