Software Testing Blog

As a way to extract the collective wisdom of the uTest community, we decided to experiment with a Guest Blogger program beginning in April. To say that it’s been a success would be an understatement, but we’ll say it anyway (the number of page views don’t lie!). Having covered a wide range of topics – including mobile app testing, tester overconfidence, security testing and more – the series has become a big hit within the community — and a great way for testers to get published in front of a large audience.

Here are a some of the highlights from our 2009 guest blogger program.  Stay tuned for an even bigger 2010!

Who is the User? – by Lucia Maldonado:  In what ways is software similar to architecture? And how can this help steer testers in the right direction? In this post, Lucia Maldonado takes an in-depth look at user accessibility standards, and offers a number of essential tips for testers in this field.

Security Testing Tips (from a Bug Battle Winner) – by Bernard Shai Lelchuck:  When it comes to security testing, few can match the expertise of Bernard Shai Lelchuck – one of uTest’s first (and finest) QA professionals. In this post, Bernard covers the basics methods of security testing, including tips for  information gathering, logical attacks and injection attacks. Oh, and here’s Part II.

Respect the Defect: Advice That Will Change the Perception of  Testing – by Joseph Ours:  Testers need to reconsider they way they report bugs – this was the position taken by Joseph Ours in his first (and hopefully not last) uTest blog post. Challenging testers to demonstrate their value by writing more clearly about the bugs they uncover (among other tactics), Joesph has sparked an interesting debate among our community. Visit the comments section to see for yourself.

Step Away from the Simulator: Putting Mobile Applications Into a Tester’s Hands – by Brad Sellick:  What makes mobile testing different from conventional software testing? For one, the simulators and emulators are far less reliable. In this post, uTester Brad Sellick – a self-made expert on mobile app testing and development – explains the dangers of relying on these tools while performing mobile app testing.

What You Need to Know About Writing Effective Test Cases – by Valerie Dale:  Despite all evidence to the contrary, test case design is often seen as work with no real value – a remedial task with no significant ROI. One would think that with the added pressures to launch a quality product on schedule, test case design and planning would be a top priority. It’s not. At best, there is minimal attention paid to the practice. At worst, it’s non-existent. In this post, Valerie Dale makes a great defense of  this beleaguered practice.

Your Overconfidence is Your Weakness: Lessons from a “Crash Specialist” – by Pradeep Soundararajan:  In our most-popular guest post to date, noted blogger Pradeep Soundararajan explains why finding lots and lots of bugs isn’t necessarily a good thing. Reliving his days as a “crash specialist” Pradeep examines how a tester’s ego can get in the way of their objective. His advice is as funny as it is useful. Simply put: a must read.

Software Testers: The “Eyes of the Battlefield” – by Brian Rock:  Our testers come from all sorts of backgrounds, including the armed forces. Brian Rock – a former Sgt. for Combat Arms Forward Recon Team in the U.S Army – is a great example. In this post, Brian makes analogizes testers with cavalry scouts. That is, they are the “eyes of the battlefield.”  Advocating exploratory software testing (especially for those in the uTest community) this post will make you rethink the role of testers.

You’re a Professional Mobile Tester (you just don’t know it yet) – by Bernard Shai Lelchuck:  As the title would imply, this post makes the case that anyone with a mobile phone and an inquisitive mind can become a successful mobile tester. It worked for Bernard Shai Lelchuck! Here Beranrd explains the rise in mobile applications, how he himself broke into the field and some basic tips for those who would like to get started in this growing (and highly lucrative) field.

Question the Connection: Tips for Diagnosing User Login Failures – by Sherry Chukpa:  Forget the sweeping generalizations about software testing “best practices.” This post by uTester Sherry Chupka gets right to the point on a very specific issue: user login failures. If you’ve ever been pitted against this problem in the testing lab, Sherry feels your pains, and has some invaluable advice for you as you move forward.

It’s been a great year, with some terrific insights into the world of testing, but our Guest Blogger program is just getting started. So if you have an opinion to express, a tip to share or a bone to pick, we’re always eager to share the thoughts of our tester community. Email us your ideas at marketing@utest.com.

Last week, uTest hosted a webinar on exploratory software testing with James Whittaker.  We received a fantastic response from the 250+ attendees, and we couldn’t get to all the questions before our time was up.  Luckily, James was kind enough to sift through a stack of the remaining questions and provide answers to several that jumped out at him.

Also, remember that we’re handpicking five webinar attendees to receive a free copy of his new book on exploratory testing, signed by James.

Q: When making a tour specific to your own application domain, doesn’t that become what is usually called a test scenario? How do you see tours being different from scenarios?

A: Great question and I cover this in my book. Chapter 4 deals with “Tours” and chapter 5 deals with “Scenarios.” In a nutshell, I see scenarios as more prescriptive than tours. Tours are meant as general guidance and scenarios, at least in my mind, are more specific. A tour specifies goals and approach to coming up with test cases, a scenario actually provides an outline of the test cases. Tours leave much more of the actual test case to be constructed as you test. A scenario, in other words, has less variation.

But don’t get caught up in semantics. It’s a continuum of detail really. At one end of the spectrum are fully detailed test cases, at the other is ad hoc testing. Scripts, scenarios, tours, patterns … they all fall somewhere in between.

Q: What is the difference between exploratory analysis and exploratory execution?

A: I don’t like introducing new terms – testing has too many of them already – so I will talk about the concepts here rather than reinforce these exact names. The thought processes that go into exploratory testing are generally considered something that you do while you are executing test cases but this only works with manual testing. In Google’s case, we do substantial test automation and once the test code starts running your chances of introducing exploration is pretty much gone. The automation will execute your test case with brute force and little flexibility.

With automation, you have to do your exploratory thinking up front and this is where we came up with the idea of exploratory analysis. Simply put, the idea is to run the Tours in your head and let your thinking inspire your automation. The best example we have within Google is Rajat Dewan’s example he presented at STAR East and explained on the Google Testing Blog.

Q: Do you have some tips on how to keep testing fresh and new when there is release after release? (To avoid people getting bored and always testing the same things and missing new issues.)

A: In fact, I do. I think this very problem is what I was trying to tackle with the tours. But your question gives me the opportunity to clarify this intent. Static test cases might be fun to come up with and fun the first couple of times you run them but running them build after build and release after release not only gets dull, it also introduces the pesticide paradox. The reality of the situation is that test cases, as a specific physical entity are too low level. They specify a precise sequence of user actions. Tours are a higher level concept and specify purpose and intent and remain flexible on specific input sequences. In this manner a single tour represents any number of test cases.

Now the secret is finding the balance. Some test cases are really important as they once found a bug or they represent an important user-initiated scenario. We want to run these no matter how bored we get. But beyond that, tours allow us more flexibility to increase coverage around the specific test cases and supply the variation that will keep our heads from exploding in boredom.

Q: Can you offer any advice for a developer to be a better partner in the testing process?

A: Indeed. But I want to point out that your questions is asking for advice to devs, not about what test can do to help this partnership (which is the harder answer, so I thank you for that).

I manage a dozen or so projects from cloud to client to back end data center stuff. Some of these have great developer participation and some less so. The devs who are great partners are very involved in testing. They review and provide feedback on our test plans and designs. They become concerned fairly often about whether we are doing a good enough job in test (I mistrust anyone who trusts me and my team too much). They try to steer testers to areas of the product not covered by dev-penned unit tests. They fret more over us finding very few bugs than when we find a lot of bugs (think about that one a moment). They show great interest in what our automation is doing and like to suggest new manual test cases. When they find a bug, they take the time to show it to us instead of just checking in a fix. They invite us to give presentations during all hands and engineering reviews and they take the time to share credit with us when the team succeeds.

I like this question. Maybe I’ll keep thinking about it and make my answer into a paper.

Q: Have you found that your tours work well or help in cases where requirements are sporadic, vaguely defined or non-existent.

A: Having never worked on any other type of project, I can say with some confidence that, yes, they work quite well.

Sincere thanks to James for a great presentation, and to all the attendees for some excellent questions.  We always enjoy seeing discussions about testing be elevated to a strategic level, and so much passion and interest in the subject.  Rest assured, we’ll be scheduling more webinars in the coming year.  In the meantime, you can find a library of free resources about software testing, including eBooks, whitepapers and recorded webinars.  Have other questions for James?  Have suggested topics for future uTest webinars?  Drop us a comment and let your voice be heard!

I’m proud to share with you that uTest took home the Top Innovator Award at the New England Venture Summit by youngStartup Ventures last week. The award recognizes cutting-edge companies driving the future of innovation in tech, life sciences and clean-tech sectors, and we’re excited to be among them.

As one of the winners, Doron was invited to present at the exclusive Summit, where a select group of 450 entrepreneurs and investors gathered to be the first to meet the next wave of forward-looking companies.

After giving his presentation about uTest’s on-demand testing model and his entrepreneurial journey, Doron also walked away with the top honor for Best Presenter at the event!

Between the Bug Battle results, the Whittaker webinar and this prestigious honor, it was a busy week around the halls of  uTest.

Attention uTest Community and prospective uTesters:   If you haven’t registered for tomorrow’s free webinar (December 10th from 1pm to 2pm ET) on Exploratory Software Testing, please click here to reserve a spot.  It’s a hot ticket, with more than 300 testers from around the world already registered to attend.

Many of you have expressed interest in additional resources to help sharpen your testing skills, so this is a great opportunity to attend a free webinar with James Whittaker. He will discuss topics from his new book on Exploratory Software Testing. Additionally, we will be handing out five free copies to attendees (signed by James) – winners will be announced at the end of the webinar.

Hope to see you there!

Just noticed something new and cool when I hit our home page tonight — the uTest community is now operating in 160 countries around the globe.

What’s that mean?  How many countries are there?  Well, depending upon who you ask (the United Nations, the US State Department, the World Almanac, etc), there’s between 189 and 195 countries on planet earth.

So recruiting professional testers from 160 different countries and getting them to profile their testing experience, demographic information, hardware and software, is no small feat.  Anyway, we just wanted to point out that the world’s largest marketplace for software testing services just got a little bigger!

Thanks to our testers from every corner of the globe for making our community so vibrant and diverse.

uTest was on fire at Under the Radar Mobility this year. I think Under the Radar said it best!

For anyone looking to deploy an app across multiple mobile platforms and a gazillion different handsets, one massive problem awaits them: QA. uTest solves this problem with an army of testers across the world. Crowdsourced QA… Problem solved. (Click here to see Doron’s presentation.)

And that’s not all! Doron was able to connect with multiple partners, prospects and top media outlets, including Mashable’s Ben Parr (@benparr), editor in chief at IntoMobile.com Will Park (@willpark), ReadWriteWeb’s Dana Oshiro (@suzyperplexus), as well as participated in a couple great video interviews with bnetTV’s Michelle Sklar (@bnettv) and GoMo News’ Cian O’Sullivan (@gomonews) which are posted below!

Take a peek at the video interviews below to learn more:

Doron Reuveni-CEO of uTest speaks with bnetTV.com at the Under the Radar event.

Read more…

In this month’s installment of “Testing The Limits”, we sit down with Matt Heusser (@mheusser) — prolific blogger for STPCollaborative, thought leader and testing extraordinaire.  We’ll discuss the state of software testing, SpeedGeeking, the role of chaos in testing software, and the lack of fistfights at STPCon 2009

uTest:  We loved the SpeedGeeking session you led at STPCon, so we’re going to flip it on you – If you had just five minutes to teach, motivate or inspire the uTest audience about software testing, what would you say?
MH: Well, I’d start by asking the audience what they are doing today – what’s the greatest point or opportunity they feel – and asking what options they see to improve. Most of the time, I hear that testing is “too slow” or “the bottleneck” or something like that.

So I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting – it is very important the team stop the time tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the numbers for improvement. For example, many of the people I talk to can spend 80% of their time or more in meetings, working on documentation, working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from 80/20 to 60/40 will double the amount of time the team spends actually doing testing.

Another thing to look at is the amount of time spent trying to reproduce defects, document defects, file bug reports, “verify” fixes, and so on. We think of these activities as testing, and they can take a substantial chunk of that 20% – but they are really accidental. That’s not a testing bottleneck – it is a development bottleneck. If test can work with development to improve the quality of the software prior to code complete, that will improve the speed of the whole system. Realizing this, and having a little bit of data to “prove” it, can help the entire system improve.

So if I had five minutes, I would say start with measuring how you track your time, and ask yourself if this is the best use of your time and what can change. Sometimes, the big boss will say “no, we absolutely need you to fill out all seven pages of documentation per test run”, and you can say “ok.”  Six months from now, when someone asks why the big project is late, you can point out that the business made an explicit decision to pay the full price of defined process. You presented options and those were not accepted.

That won’t save this project — but it might save the next.  It also turns out that actually testing tends to be much more fulfilling than documentation and compliance activities. Who could have guessed?

Lots of contrasting opinions at last month’s STP Conference. While there were no fist fights (that we heard about anyway), what did you see as the most contentious issue? And where do you fall on this issue?

Read more…

Here in the U.S., the end of November is marked by Thanksgiving.  This is a time of family, friends, feasts and football (not futbol).  So while much of the uTest crew is taking a well-deserved four-day weekend, I wanted to express our sincere gratitude to the entire uTest universe — customers, testers, investors, partners and media.

Without you and your passion for contributing to a bug-free world, uTest would not be where it is today.  Ultimately, a business such as ours (discovering and eliminating defects, pay-for-performance, collaborative, reputation-driven) would not exist if it weren’t for your collective desire to make things better than they were yesterday.

And for that, I am truly thankful.  What are you thankful for this season?

Late last week, our fearless leader was out in the Valley presenting at the Under The Radar Mobility 2009 event.  We first presented at UTR in April as an emerging company.  This time, we were asked back to present as a Graduate Circle company, which is reserved for those past presenters who are making a big splash in the market — quite an honor indeed.

The theme of this event was all things mobile.  The event was heavily attended by mobile OEMs, wireless carriers and all types of mobile app makers, including innovators such as Boku, Mplayit, ShoZu and Yowza!! (those are their exclamation points, not mine).  More details (and fantastic photos) after the jump.

Read more…

The mere thought of air travel during the holidays is annoying enough to send most people running to their nearest bus or train station.  The crowds, the lines, the delays, the zip-lock bags and 3 oz bottles of shampoo… but wait, there’s more!

Late last week, a five-hour computer glitch caused flight delays across the U.S. that were still rippling through the transportation system for most of the day.  The problem was made worse by the fact that the National Airspace Data Interchange Network failed at both its locations — Atlanta and Salt Lake City.  (Ed. note:  I’ll try hard to avoid using the word “crash” in this post.)

Bloomberg.com had this to say:

The Federal Aviation Administration blamed a four-hour software failure for causing airline delays and cancellations across the U.S.  The shutdown lasted from 5 a.m. to 9 a.m. ET after “a software configuration” malfunction today in Salt Lake City.

And The New York Times chimed in with this little bit of sunny news:

Read more…