Software Testing Blog

Every once and awhile, when there’s nothing topical to blog about, I decide to go back in time and focus on a software testing classic. Today is one of those days.

With that in mind, I wanted to draw your attention to Classic Testing Mistakes by Brian Marick. Whether you’re a tester or manager, experienced veteran or wide-eyed newbie, this 25-page article outlines some of the most classic testing mishaps and offers valuable tips on how to avoid them.

So what are some classic testing mistakes? One would be not reading this document in its entirety. As for the others, here are a few clips I found interesting. Note the bold titles are mine, everything in italics in Brian’s.

Mistake: Testers Not Responsible for Usability
If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn’t either.

Mistake: Misunderstanding the Role of “QA”
A first major mistake people make is thinking that the testing team is responsible for assuring quality. This role, often assigned to the first testing team in an organization, makes it the last defense, the barrier between the development team (accused of producing bad quality) and the customer (who must be protected from them). It’s characterized by a testing team (often called the “Quality Assurance Group”) that has formal authority to prevent shipment of the product. That in itself is a disheartening task: the testing team can’t improve quality, only enforce a minimal level. Worse, that authority is usually more apparent than real.

Mistake: Bad Timing on Load Testing
Putting stress and load testing off to the last minute is common, but it leaves you little time to do anything substantive when you discover your product doesn’t scale up to more than 12 users.

Read more…

If you’ve worked in the software industry, chances are you’ve encountered words and phrases that you did not understand. I certainly have. So to help make sure we’re all on the same page (that means common understanding), I’ve decided to start a running thread of some of the more common slang terms that apply to software testing. Of course, this list will be woefully inadequate without your input, so please add your slang terms in the comment section below.

Here’s a few off the top of my head, as well as some that I found in this Quora thread:

• Dogfooding: When a company tests its own software internally before releasing to beta
• Low hanging fruit: Easy tasks that can be completed in short order
• WAG: Wild ass guess
• SWAG: Scientific wild ass guess
• Staging: A development enviroment; one level before production
• Automagically: Describes something that occurs in software that is either too complicated to explain, or the person describing the process really has no clue how it works
• Quick and Dirty: A quick and simple solution to an otherwise complicated problem
• Showstopper: A bug that makes software unusable
• Brown-bagger: A very embarrassing bug found soon after release
• Whack-a-mole: The practice of repeatedly getting rid of a bug, only to have it continually reappear
• Drink the Kool-Aid: Lacking objectivity
• Heavy lifting: Difficult and challenging work
• SoLoMo: Social, location and mobile – testing components of many mobile apps
• Burndown: a chart that is a graphical representation of work left to do versus time
• FUBAR: F#@cked up beyond all recognition
• PEBKAC: Problem exists between keyboard and chair (good one John Montgomery)
• RTFM: Read the F#$cking manual
• Fast-track: To speed up a process

Like I said, I’m certain that I’ve left out 99.99% of all the great testing slang terms. So please, pick up the slack (help me out) in the comment section below.

“I’m sorry that testing is complicated, folks. No wait. I’m not sorry at all. Go away if you don’t like it.” – James Bach, Testing the Limits, 2010

James has a point here: If you’re not willing to learn testing the proper way, then you should probably find another line of work (something that could be said of almost any profession). So what is the proper way to learn software testing? Well, that’s very much up for debate.

As many of you know, uTest has gone to great lengths to ensure that testers of all experience levels can learn by doing (uTest Sandbox program and paid projects), learn by studying (Crash Courses) and learn by peer-to-peer discussion (uTest Forums). Apart from uTest, there’s also tester certifications, college courses, training seminars and a host of other methods.

So which method is most likely to help you successfully learn testing? Before you answer that, let’s consider a few points from a recent Wired.com article Everything You Thought You Knew About Learning Is Wrong:

Taking notes during class? Topic-focused study? A consistent learning environment? All are exactly opposite the best strategies for learning…

…first, think about how you attack a pile of study material. “People tend to try to learn in blocks,” says Bjork (director of the UCLA Learning and Forgetting Lab), “mastering one thing before moving on to the next.” But instead he recommends interleaving, a strategy in which, for example, instead of spending an hour working on your tennis serve, you mix in a range of skills like backhands, volleys, overhead smashes, and footwork. “This creates a sense of difficulty,” says Bjork, “and people tend not to notice the immediate effects of learning.”

Read more…

Answer: In-the-wild testing.

This according to our very own CMO Matt Johnston, who recently sat down with Rich Hand of softwaretestpro.com for an in-depth discussion on the growing importance of in-the-wild testing. This lively Q&A was then adapted into a featured article  (no word yet on its big screen debut).

Anyway, if you’re new to the concept of in-the-wild testing or want to know more about it, then I highly recommend giving it a proper read.  Here are a few clips to get you started.  Enjoy!

Does it ever seem that no matter how much time, effort and money your QA team spends improving and refining software testing processes that some sort of defect is always found in the application, website, or mobile app after launch? Surprisingly, it has little to do with your organization’s in-the-lab testing – whether in-house or outsourced, manual or automated. In fact, it’s likely due to the fact that the lab environment you’re testing in cannot adequately replicate real-world conditions (real users, real devices, diverse locations, imperfect connectivity, not to mention a range of devices, operating systems, browsers, etc.).

And thus, companies test extensively in the lab, launch their apps into the real-world conditions of users, and we’re all surprised when these products don’t perform as expected. But what kind of testing can fill this gap effectively and affordably? It was out of this persistent question that uTest and crowdsourced testing was born. And the by-product of this was a new category of testing which has become a must-have for mobile, social and local apps known as “In-The-Wild Testing.”

“In-The-Wild Testing” (ITWT) is an effort to educate tech leaders about how to help QA teams and organizations launch higher quality software, quicker, faster, and cheaper. The idea of in-the-wild testing is about providing organizations with the real-world testing data necessary to make informed decisions about releasing products to market. According to Matt Johnston, Chief Marketing Officer for uTest, “Don’t be fooled by the word ‘wild’ when it comes to testing software. When you think of the term ‘In-the-wild testing’ think of it as ‘real-world vs. laboratory conditions.’” This is not outsourcing or beta testing, and it’s definitely not suggesting you replace the QA teams or solid processes you have in place within your test lab. Rather, this is about complementing, scaling, and aligning professional testing resources with your in-house or outsourced QA team. I predict that this concept will explode in the next five years . But the first step is to understand what ITWT is (and isn’t).

Continue reading >>>

Today, we’re thrilled to announce the results of our third annual uTester of the Year Awards. Every year, we recognize uTesters who have consistently gone above and beyond their call of duty in their participation with uTest projects. This year’s winners were selected by our community and project management teams, who have had the privilege of working closely with such an extremely talented community of professional testers. From test automation to test team lead, these winners are truly experts in a variety of testing domains. The level of talent continues to impress, with each year’s accolades becoming more and more difficult to attain (and judge). So without further ado, let’s meet our 2011 winners!

Top honor for the 2011 award goes to David Honeyball from the United Kingdom!

David joined uTest in June of 2009. Since joining us, David has become a Gold rated tester in Functional, Load and Localization testing, as well as having achieved the silver rating in both Usability and Security testing. David also became our Top Test Team Lead in 2011, successfully leading nearly 200 projects alongside of uTest project managers. David had this to say about his experience with uTest during the past year:

I joined uTest back in the summer of 2009 and have to say I have never looked back. What started out as something extra in my spare time has taken up more and more of my time in a good way and has increased my confidence as a tester to levels I never thought possible. I have been a tester for nearly 15 years but can safely say that every day with uTest is a new experience and a new challenge.

I have met so many wonderful people including other testers, CMs, PMs and customers who are all committed to achieving their goals and creating a wonderful service. They have guided me and helped me in the last year and increased my communication skills as well.

TTL (Test Team Lead) Experience:

2011 has been a big success for me personally, as it has for uTest in regards to growth and development, in many ways due to my journey from tester to TTL. I started out as a TTL early in the year and felt at the time that it was a great way forward and would be of huge assistance and help to customers. Since then I have been involved in something nearing 100 cycles as TTL, but that could be more now!  I enjoy the TTL role immensely as I get to speak to the PMs and testers and help out others who are stuck. I hope if you have worked with me that you know that I take that side of things very seriously. One of the great rewards is helping someone who is stuck to complete a test case or test for example. Above all, I believe that with my experience I am a very fair TTL and have the best interests of customer and testers close at heart.

As time goes on I hope to grow more and more into this role and improve further as there is still so much to learn.

Special Projects 

Apart from testing and TTL work which does take up some time as you can imagine, I also help with test case writing for certain PMs which I enjoy and get value from. I was also heavily involved in the startup of the test case conversion to uTest which is the new system we see today. This side of things linked in with my testing and other roles forms a vital component of what I do as well and hope it adds value to the company as a whole.

So, just want to say a huge thanks to everyone involved with uTest for the opportunity and long may we continue to grow together and become invaluable to customers across the globe…

The complete list of winners is shown below:

Read more…

In the second part of our Testing the Limits with Anne-Marie Charrett, we get her thoughts on the meaning of exploratory testing, the challenge of agile adoption, how to grow as a tester and more. Enjoy!

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

AMC: I think companies that demand value from their testing are generally more receptive to new ideas and change in testing. I don’t think it’s fair to silo this into industries.

Take for example the finance industry, yes many large insurance and bank corporations are risk averse and resist change but not all. For example Barclays Bank are using coaching & Rapid Software Testing.

I’ve worked with small companies in R&D who you would associate with flexibility and being pro-active, yet they want very traditional, heavily documented testing processes. Often this is because someone did testing ‘once’ and this is what they did.

I’ve seen testing practices change within sectors too. For example, the telco sector in the mid 1990‘s were typically heavily documentation orientated. Often testing went on for years before a product was released. By the late 90’s and early 2000’s testing practices had to evolve as smaller companies with lighter and more flexible delivery approaches challenged this paradigm.

uTest: There’s a good debate right now on the true meaning of exploratory testing, with people like James Bach and Michael Bolton chiming in with their opinions. What is your definition of exploratory testing? And in your view, what is the most misunderstood term in testing today?

AMC: So many questions!! The beauty of Exploratory Testing is that it can mean different things to different people. Thats why there are so many different perspectives on it.

There are some core values to Exploratory Testing, namely that it’s an approach (not a technique), it’s simultaneous learning, design and execution and that it’s tester centric.

The latter ideal is something that I cherish and hold dear.  I think it’s essential that we take responsibility for the testing we do. This means each tester decides on their testing approach, what they test and when they’re done. Owning these decisions is what matures a tester, helping them become skilled, confident and motivated to excel in their testing.

Read more…

To kick off another amzing year of Testing the Limits we reached out to Anne-Marie Charrett, an independent tester who has worked for the likes of Mercury Interactive, IBM (twice) and Nortel – just to name a few. She also arranges for speakers to visit Ireland as part of Softtest Ireland and blogs about her testing experience and offers coaching at mavericktester.com. 

In part I of this month’s interview, we learn what motivates Anne-Marie to coach via Skype, what’s caught her interest lately, how her book with James Bach is coming and what the biggest mis-conception about testing is. Come back tomorrow for part II.

uTest: In terms of writing, speaking and researching, you are one of the most active testers in the business. So we’ll start by asking you this: What hot topics within testing have captured your interest recently?

AMC: 2012 has kicked off with a flurry of activity. Key topics appear to be, How we learn, Rapid Test Management and more recently James Bach has been looking Exploratory Test Documentation.

It goes like this. Typically we write tests and charters as artifacts for other people as evidence of work performed. But writing is a lot more powerful than that, it has the ability to assist in design (think brainstorming in mind maps). Exploratory Test Documentation is about changing the purpose of writing from an end product to a by product.

I also like the way new conferences and peer workshops are happening at a grass roots level, for example Lets Test in Stockholm. These are not necessarily big conferences, but ones that offer value to testers and that encourage participation. I hope that this will be the conference circuit of the future!

uTest: You’ve made quite a name for yourself as a testing coach; offering advice to testers free of charge via Skype. In your experience, what areas require the most coaching on your part? In other words, what does a typical tester coaching session cover?

AMC: Often testers come looking for coaching in a particular skill (e.g Test Automation), but many fail to understand basic testing concepts such as: “What is testing?” and “How do you determine bugs?”

Understanding testing is key to improving your testing skill.  After all, if you don’t understand something, how can you improve it?

Software delivery typically doesn’t allow for this type of introspection. Our jobs demand we focus on delivery, often to the detriment of how well we are doing our testing.

Coaching is the breathing space that all testers need to learn and grow.

In coaching I encourage testers to work through tasks to acquire skill. I’m there to guide and help them, but they need to work out the answers. That way, their learning experience is deeper and more meaningful and empowering.

Read more…

In the software business, it’s all about precision, as even the slightest coding mistake can lead to catastrophic failure. This lesson is clearly not lost on the folks over at the United Nations telecommunications agency, who are meeting as we speak to decide whether or not to abolish the leap second. That’s right, the leap second.

The Sidney Morning Herald explains how this relates to software testing:

Unlike the better-known leap year, which adds a day to February in a familiar four-year cycle, the leap second is tacked on once every few years to synchronise atomic clocks – the world’s scientific timekeepers – with Earth’s rotational cycle, which, sadly, does not run quite like clockwork. The next one is scheduled for June 30 (do not bother to adjust your watch).

The United States is the primary proponent for doing away with the leap second, arguing that these sporadic adjustments, if botched or overlooked, could lead to major foul-ups if electronic systems that depend on the precise time – including computer and cellphone networks, air traffic control and financial trading markets – do not agree on the time.

Abolishing the leap second “removes one potential source of catastrophic failure for the world’s computer networks,” said Geoff Chester, a spokesman for the US Naval Observatory, America’s primary timekeeper. “That one second becomes a problem if you don’t take it into account.”

By now, you’re probably wondering what the “debate” is all about. Is anyone voting in favor of catastrophic failure? On the other hand, how can a unit of time be abolished, even if it’s only a second? The story continues:

Read more…

While our usual maniacal focus is on quality over quantity, it’s not unreasonable to recognize a major milestone that occurred today, January 18, 2012: surpassing 50,000 testers in the uTest community! Just to be clear, that’s over 50,000 testers from 185 countries around the world – from experts in automation to gurus in usability testing. Here are several other facts about our community:

• Every month, there are approximately 1,000 new tester registrations
• Over 99.9% of these registrations are organic – word of mouth, tradeshows and conferences, tester referrals
• The majority of testers span rather evenly across North America, Europe, and Asia. The rest fill out in South America, Africa, and Australia
• Over 80% of uTesters have a Bachelor’s degree or higher
• uTesters bring a wealth of knowledge and diverse set of skills to the table: creating test cases, usability surveys, load and performance scripts, automation scripts, security coverage reports, usability audits and expert reviews; executing test plans, usability surveys, live load test cases, security scans, exploratory tests, and translation tasks and proofs

And…back to our maniacal attention to quality. Although there is certainly strength in numbers and meaning to this milestone, the real excitement stems from the various “homegrown” programs that shape our crowdsourcing model. Less than a year ago, we announced several new initiatives that have transformed the uTest community from an unruly crowd to one that is self-sufficient, self-teaching and self-policing. From paid leadership roles for our top testers to unpaid auditions for newbie testers, there is a role for nearly everyone and a path for the most ambitious. And now that most of us have embraced the New Year, it’s only fitting that there are new programs just around the corner – ones that leverage the foundation built in the past year and continue to benefit our community at large. More details to come shortly!

For now, please join me in raising your glass to celebrate this major milestone with us!

Think optimizing for mobile web isn’t important? Think again! This was reported by PCWorld:

Tablet computers will eventually replace laptops, according to nearly half of Americans polled earlier this month.

But don’t panic yet …

Of course, “eventually” is a very long time, and the recent rollout of Amazon’s Kindle Fire and anticipation over the Apple iPad 3 might have survey takers overreaching a bit.

While the Poll Position phone survey of 1,155 registered voters found great enthusiasm for tablet computers, with 46% saying tablets would surpass laptops eventually, 35% said tablets will not replace laptops and 19% had no opinion.

Among younger Americans (18-29 age group), 49% said tablets will not replace the PC and 37% said they will. A higher percentage of men (53%) than women (39%) foresee tablets overtaking laptops.

The tablet market was hot last year and is expected to remain so this year. IDC recently said it expected 2011 worldwide tablet shipments to total more than 63 million units, with Apple selling about 6 in 10 of those. Recent Canalys figures show a total PC market of 356 million units in 2011, minus tablets.

Read more…