Software Testing Blog

Our featured uTest guest blogger this month is Elena Houser. Originally from Russia, Elena got her start in IT with UUNet Technologies. Eventually, she caught the testing bug and has been testing for the last 7 years.

She studied software testing under the guidance of James Bach (Satisfice) and Matt Heusser (Miagi-Do School of Testing). Shortly thereafter, she was named a winner of the 2010 uTest Bug Battle and achieved “Gold” status in the uTest community.

You can learn more about Elena by reading her blog, following her on Twitter or by checking out her uTest public profile.

In this post, Elena covers her recent fascination with mobile app testing. Enjoy!

****************

When I was growing up in Siberia I dreamed of becoming either a ballerina or a painter. I have always loved ballet and dancing and I also enjoyed painting Siberian landscapes in watercolors. For the longest time I thought that I was more of an artistic or right-brain type person. I am still not sure how I ended up working in the Information Technology field.  At some point, I found that I loved breaking websites (aka testing) and haven’t stopped since.  I catch myself testing my son’s electronic toys, neighbor’s coffee machine as well as countless smartphones…which brings me to my latest passion: mobile app testing. In fact, I’m writing this post on a break from a very intense work week, during which I tackled 7 uTest projects – with 5 of them being mobile projects!

Testing mobile apps is interesting, fascinating, challenging and frustrating. It has been an amazing and eye-opening experience for me. I am still very new to it and have more to learn. I tested my first mobile application during the uTest Bug Battle last November. Since then, I have participated in over 30 mobile test cycles through uTest and reviewed a variety of applications from games to browsers to enterprise resource planning (ERP) apps.

I want to know what kind of devices, applications and providers my friends and neighbors are using. I bombard them with questions on what they do and don’t like about their smartphones and providers, what kind of apps rock their world or make them want to throw their phones against the wall. Do they use mobile vs. native apps? How do they figure out navigation and flow, what do they think about a particular page layout, colors, sounds, portrait vs. landscape view, web-surfing and social media experience? Many of them check their bank statements and pay their bills, write memos, take snapshots, watch movies, use it as FiOS remote control, listen to the radio, play Angry Birds, Stupid Zombies and other games. I take notes. I try those apps myself. I install some of them on my Motorola Xoom even though many are still not optimized for it. It’s fun to launch an app and try to figure out how it works, what buttons to tap, and also picture the data moving behind it.

Just as with a web site the following three things matter the most for mobile apps – presentation and page layout, organization (flow and structure) and interaction (what happens in response to user actions). The difference is that a phone screen is very small and the capacity of a mobile device is not the same of a web site, which makes functional testing pretty challenging. Not to mention that some mobile applications may work on iPhones but then fail on either installation or launch on Android phones.

Read more…

A quick update from the shiny, new halls of uTest (yes, I’m referring to our fancy new digs).

Earlier this week, we announced that uTest achieved more than 3X increase in year-over-year revenue. And as you can see, from the pictures to the left, we moved into some new office space in our Boston-area headquarters. And beyond all that growth, we found time to receive some rave reviews from the media and accept an award or two.

For those of you keeping score at home, here’s a recap of the recent goings-on at your favorite software testing company:

Customer, Community & Company Growth

• Signed 70+ new customer deals with innovative startups like Groupon and AirBnB; as well as with established category leaders like AOL, Virgin Airlines and Aetna
• Added 3,000+ testers to pad our lead as the world’s largest testing community — that brings us to 33,000+ professional testers from 172 countries
• Pitted the top specialty e-tailers against one another in another epic Bug Battle. During this competition, more than 600 testers from 28 countries discovered nearly 900 bugs in the web and mobile apps of eBay, Overstock.com and Zappos; check out one of the write-ups at Internet Retailer
• Added a bunch of new members to the uTest family, increasing our employee base by more than 20%.  And as you can see throughout this post, we also moved into some killer new office space

New Product Stuff

• Public Tester Profiles & Badges: With our new public profiles, testers can display their memberships and exemplary uTest ratings (bronze, silver or gold badges) for all the world to see. Testers can also now add their uTest badges to their personal site, blog, resume or email signature.
• Social Sharing: we launched a new social sharing feature that enables uTesters to share their profiles and testing accomplishments on Facebook, Twitter, or LinkedIn directly from their uTest accounts. For example, a tester can let the Twittersphere know that he just earned a silver badge or that he published his public profile.
• Accept/Decline a Test Project: Testers can now accept or decline a test project based upon their availability. This gives uTest much better visibility into each project so we can more precisely gauge whether or not more testers should be invited to meet each customer’s schedule and requirements.

Press Praise

• Venture Capital Journal in “Should You Join The Crowd?”
• ST&QA Magazine in “The Crowdsourcing Testing Model”
• And earlier this week, we were named to the Red Herring Global 100 list of hottest private companies

It was a big quarter for us, but we’re not getting cocky… we’ve got big plans for 2011.  And as always, we know that we never could’ve done it without our customers (who care more about their users & apps than any rational person should) and our testers (who share that passion for apps that make users smile).

Finding unique bugs in a software application is never easy. It’s even more difficult when you are working with dozens of other testers from around the world. In our latest guest post, uTester Amit Kulkarni – a recent Bug Battle winner – offers testers some advice for finding unique bugs when multiple issues have already been reported.

When working on test cycles in the uTest platform, it’s important to remember the model they follow. For instance, once the test cycle is open, it is made available to invited testers from all over the world, unless the customer has chosen specific criteria like geographical locatio, specialty, experience etc. And because there are different time zones, you may find yourself in a situation when you accept the invite for the test cycle and there are already a good number of bugs logged. So common, isn’t it?

Now here is the catch: Some testers wonder how do they go about it? There are already a lot of bugs logged, what should I do? How am I going to find a bug now? Well, first I would say calm down. Understand the application and what is expected from the test cycle. Once you are done with this, take a look at the bug reports already logged by other testers. This will give you an idea as to what part of the application has been tested so far, and what has yet to be tested.

I often feel quite lonely when there are no bugs logged when I accept an invite (a rare occurrence however). I would prefer to work with other testers as a team to make sure the outcome is what is expected by the customer before creating the test cycle, but that’s not always possible. That being the case, would you like to know how stand out from the other testers?

Here’s my advice:

Read more…

To the delight of many exhausted testers, the Q4 Bug Battle has officially come to an end. This final competition of 2010 spanned across 10 full days with testers from all around the world competing to find the best bugs in three popular specialty e-tailer sites. After pre-announcing the winners on the uTest forums earlier this week, we’re finally ready to unveil the official results from the competition. You can download the complete Bug Battle report (PDF), but here are several highlights that are worth noting:

• eBay placed first in every survey category and recorded the fewest bugs overall (220 bugs)
• Trailing eBay in the number of bugs was Overstock.com (314 bugs), closely followed by Zappos (354 bugs)
• Fraud protection, product search accuracy and overall usability were the most important factors when choosing an e-tailer
• Zappos came in second in the category of product search accuracy and ease-of-use, while Overstock.com came in second for fraud protection and low prices
• 65 percent of respondents trust eBay most for their holiday shopping

You may also check out the story on BostInnovation and Internet Retailer. All said and done, nearly $4,000 in prize money was distributed amongst 22 winners, covering Top Tester(s), Best Bugs (Web and Mobile), Best Usability Feedback, and finally Honorable Mentions. The complete list of winners is found here, and the noteworthy winners are again highlighted below for their outstanding accomplishments:

• Top Testers: Anna Danchenko, Nicola Sedgwick
• Best Bugs (Web): Aline Gazeau, Anna Danchenko, Gagan Talwar
• Best Bugs (Mobile): Thiago da Silva, Ryan Arnaudin, Zarina Mohammad
• Best Usability Feedback: Amit Kulkarni, Nicola Sedgwick, Anna Danchenko

Congratulations to all of the winners of the Q4 Bug Battle! If you missed out on the opportunity to participate this time, be sure to read the following blog posts from previous winners to gain insight prior to our next competition: Santhosh Tuppad and Anna Danchenko.

And while you’re at it, if you have any suggestions for our next quarterly testing competition, send them our way at marketing@utest.com.

[WARNING: Slight self promotion today, but we couldn't help but share the exciting news with you!]

Today, we are thrilled to announce a 300 percent increase in year-over-year revenues in the recently-ended Q3. During this time, uTest closed a $13 million C Round of financing (led by Scale Venture Partners) – one of the largest investments ever made in a crowdsourcing company. We also launched cool new features and functionality, such as social sign-in, dynamic reporting and integrations with more bug tracking systems.

In Q3 2010, uTest also:

• Signed 120+ new customers, including innovative category leaders The Associated Press, The BBC, The Container Store, Urbanspoon, Box.net
• Acquired 2,500 new software testers, growing the community to 30,000+ professional testers from 168 countries
• Added Scale Venture Partners Managing Director Sharon Wienbar to the Board Of Directors
• Conducted the “Clash of the Career Sites” – more than 500 testers from 22 countries discovered nearly 700 bugs in the web and mobile apps of Monster, CareerBuilder, SimplyHired and Indeed; see articles in TheNextWeb and TechTarget for details

And there’s so much more to come! I just want to take a quick moment to thank our fantastic global community of testers who exceed expectations and go above and beyond to meet the testing needs of hundreds of companies each day. Our third quarter results are a testament to the tester community’s commitment to excellence.

Just a quick update on the latest uTest Bug Battle. We’re six days into the competition, which as you know, is comparing three of the top specialty e-tailers in the universe. That would be eBay, Overstock and Zappos.

With just under five days remaining, we’ve already had more than 400 testers participate and well over 600 bugs reported. These are obviously great numbers, but it’s a little interesting to see that eBay trails the other two sites in terms of overall participants. (Hint, hint: if you want to increase your odds of winning some prize money, start testing eBay!). You didn’t hear it from me though.

Remember, the Bug Battle ends Monday, November 8th at noon ET, and there is nearly $4,000 in prize money up for grabs. It’s not too late to win the top prize, so log into your uTest account and get cracking. No more excuses!

For a primer on how to win the Bug Battle – especially the usability part of the competition that gets started next week – we highly recommend this post by Santhosh Tuppad, as well as this post by Anna Danchenko. You can also find tips in the uTest Forums.

Good luck the rest of the way!

As you probably know, the latest uTest Bug Battle is officially underway. This means that for the next week, testers from around the world will be competing to find bugs in three prominent e-tail sites. But there’s a lot more to the competition than just submitting some random bugs and hoping for the win. In our latest guest post, uTester Anna Danchenko (a former Bug Battle winner herself) offers some advice to testers on how they can best position themselves for Bug Battle success – and of course, some prize money for good measure.

*******

The Bug Battle is a software testing competition open to anyone on the uTest website. The essence of this quarterly competition is simple: You have about a week to find and report defects in the pre-selected websites or applications. The results will be judged on several criteria such as: the “clarity of the bug report”, the “innovativeness of the bug” and the “depth of understanding of the problem.” To put it simply, the more serious the bug found and the better its description, the greater your chances are of winning the battle. As a top tester in Q310 Bug Battle, I’m going to offer some advice on how to find the worst bugs and how to write the best bug descriptions.

I will start with an overview of the available materials that newcomers can use to get ready for the competition. After this, I will move to the main topics: how to find defects and how to write bug reports. Finally, I will write some recommendations on completing the final stage in the competition: feedback survey.

Available Materials
This section is a collection of the important links to be explored by uTest newcomers. I have divided it into 3 parts: “Registration”, “Bug Battle” and “Bug reporting in uTest”. To participate in the Bug Battle you need to be registered in uTest. In addition, I highly recommend that you become a member of uTest forum. This is an invaluable resource for learning about past Bug Battles, uTest and testing. In the “Bug Battle” section there are links to pages with the competition rules and advice from other participants on how to win. The last section on bug reporting contains the most important resources with uTest recommendations. I will comment on how to get the most out of these resources because a good bug report is one of the keys to getting the Best Bug award.

Read more…

The Q4 uTest Bug Battle is officially underway. As we noted yesterday, this quarter’s competition will challenge testers to find bugs in three of world’s most prominent specialty e-tailers: eBay, Overstock and Zappos.

So which of these three sites will have the most robust feature set? Which will have the most intuitive search and product comparison features? Which one will have the most user friendly mobile application? And most important of all, which one will have the most bugs?

That’s for our community to determine. Once the testing phase of the competition is over (on Monday, November 8 at noon) we’ll be sending surveys to all participating testers, where they will compare the usability and feature set of these three applications.

At this point, I shouldn’t have to tell you about the many reasons why you should participate, but I will anyway. For one, a strong performance in the uTest Bug Battle can help you get recognized by our project managers, which means you’ll be invited to more private test cycles. It’s also a great way to hone your testing skills. Oh – and did we mention that we’re giving away $4000 in prizes across a broad range of categories?

So here’s your chance. Log into your uTest account, scour these apps for defects and report them in a clear, concise manner. And if you do it better than your peers, you could be named the Q4 Bug Battle winner and earn some big prize money for your time.

Need some pointers? This uTest Forums thread will show you how to increase your odds of winning.

Have more questions? Learn more about the Bug Battle basics.

Good luck everyone!

You know how every neighborhood has that one crazy lady who starts the holiday season a little too early (like before Halloween)? Well, this time it’s us.

With the holiday shopping season right around the corner, we simply couldn’t wait any longer to kick off our latest bug-hunting competition.

At this time last year, you may recall that the Bug Battle focused on the giants of online retail. This year, as a special twist, we’ve decided to focus on specialty e-tailers. With literally thousands to choose from, we ended up going with eBay, Overstock and Zappos - three of the most revolutionary companies in the auction, overstock and traditional e-shopping spaces.

The Bug Battle gets started this Friday, October 29 at noon (ET). Testers will be given ten days (until Monday, November 8 at noon) to search these applications for the most compelling bugs, and to report them through our online platform. We’ll be awarding nearly $4,000 in prize money for categories like Top Tester, Best Bug, Best Survey Feedback (survey runs November 8-10), Best Mobile Bug and more.

More details on this Bug Battle – as well as the rules, prizes and deadlines – can be found in the uTest Forums. Of course, we’ll be providing frequent updates via the uTest blog, as well as through Twitter and Facebook.

This competition is available ONLY to members of the uTest community. If you would like to become a member, sign up here.

Happy testing!

The Q3 Bug Battle is now one for the history books. As you might recall, our brave community of testers spent a week in August competing to find bugs in four of the major career sites. After pre-announcing the winners last week, we’re finally ready to de-classify the official results. You can download the complete Bug Battle report (PDF), but here are a few top-line findings:

• Indeed had the least reported bugs (139), followed by Monster (169), CareerBuilder (171), and then SimplyHired (189).
• While Monster placed first and Indeed placed last in every survey category, CareerBuilder and SimplyHired were deadlocked in the survey portion of the contest, each receiving favorable rankings in three out of six categories.
• More than half of the testers (55%) chose ‘job search accuracy’ (ability to browse relevant jobs by job title, keyword, location, category, etc.) as the most important criteria when evaluating career applications.
• When asked if concerned about the privacy and safety of personal information stored on job websites, an overwhelming 82 percent of respondents answered “yes.”

Check out the story on TheNextWeb! The $4,000 in prize money was spread out across a total of 24 testers. Here are few of the notable recipients:

Read more…