Software Testing Blog

Your computer just crashed taking with it that document you’ve been working on for the past hour.  Naturally, you cry out in anguish over all those terrible software bugs that conspire to crash your computer and drive you nuts. But new evidence suggests that something else could be the cause: bad memory.

Google just announced the results of a new study based on the thousands of servers they manage in their data centers, and the results are surprising.  Memory error rates are thousands of times higher than anyone previously believed.  (CNET has a good summary of the research here.)

For software testers, this represents a significant new wrinkle in identifying bugs.  Wonky behavior and computer crashes could really be a hardware problem, and these sorts of hardware issues are way more common than we previously thought.

So what does that mean for your testing?  Here are a few thoughts that could help.

Read more…

Once we announced the Search Engine Bug Battle results, the press rooms were buzzing. Here’s a quick snapshot of some of the top coverage the uTest community received this week. The headlines say it all – Google comes out on top, but Bing definitely impressed our testers.

• “Google Still Dominates in Search Engine User Testing”

• “Bug Testers: Google is Clean, Bing is Buggy”

• “Bing is the Buggiest (But Second-Best) Search Engine”

• “Bug Testers Prefer Google to Bing”

• “Google Tops Bing, Yahoo in Bug Battle”

For a complete list of bug battle coverage, check out the uTest Newsroom!

We couldn’t have done it without our talented, bug-hunting uTesters – thanks again for all your hard work and feedback!

After countless hours of review, discussion and debate (and a few late nights of pizza and beer), the official results of the Q3 Bug Battle are in!  During a week in late August, a total of 606 bugs were reported across Google, Google Caffeine, Bing and Yahoo; and nearly $4,000 in prize money was doled out to those testers who’s bugs and feedback were judged to be the best.  For complete details, see the Q3 Bug Battle report.

Inside uTest, we all agree that this was the best Bug Battle yet!  With more than 1,100 testers from 50+ countries around the world participating, we had more high-quality bugs and feedback than in any of our previous three Bug Battles.

First, let me say thanks to all the testers who participated.  Second,  it’s safe to say that this was the toughest Bug Battle we’ve had to judge yet (a good problem to have).  Third, I want to express our sincere congratulations to all the winners!

The top tester prize went to Joseph Ours of Columbus, Ohio, who reported a number of highly valuable bugs and provided some outstanding feedback. Other winners included noted testing blogger Pradeep Soundararajan of India (top bug), as well as Max Sterneck from Japan (top novice).  Other big winners included Bernard Lelchuck of Israel and Ranchhod Prajapati of India.  Check out the complete list of Q3 Bug Battle winners .

A Few of the High-Level Findings from the Bug Battle:

• 71% of respondents stated search results accuracy as the most important when choosing their search engine.
• In overall accuracy, Google led the way with a top two box score (those rated as “excellent” or “good”) of 90%. Caffeine followed with 83%. Yahoo & Bing trailed with 53 and 42%, respectively.
• Bing had the most reported bugs; however, it scored very high for usability and pics/vids search (note: yesterday, Bing unveiled a new “visual search feature)
• More than 30% noted that Bing surprised them favorably; and 10% said that after testing all three engines, they would make Bing their default engine.

Read more…

As you may have heard, we recently wrapped up our 3rd quarter Bug Battle competition.  This time, we pitted search engine heavyweights Google vs. Bing vs. Yahoo and turned our community of testers loose on them to do exploratory testing, report bugs, and then complete an extensive user survey.

The official results won’t be announced until next week, but I can say with certainty that this was our best Bug Battle yet.  We had greater quantity and quality of participation than in any of our other Bug Battles.  In all, we’re doling out nearly $4,000 in prize money to the testers whose bugs and feedback were judged to be the highest quality and most interesting.

The actual winners of our Bug Battle competitions are a closely guarded secret.  Well, unless you’re a member of the uTest forums (available only to uTesters), in which case we’ve pre-announced the winners late this afternoon.  For everyone else, the winners and results of this competition will be available mid-next week.

• Which search engine had the most bugs?
• Which has the most popular UI?
• Who was judged to have the coolest new features?
• Which offers the most accurate results?
• What do users love & hate about Bing?
• How did Google Caffeine stack up against Google (caffeine-free)?

Feel free to post your best guesses here in the comments.  The real results will be unveiled next week!

Our Bug Battle of the Search Engines is now open and running. Starting on Friday, August 7th, uTesters can test the quality, features and usability of the most popular search engines: Google, Bing, and Yahoo! We’ll be awarding more than $3,000 in prize money to the top testers, based on:

• Top overall tester (based on the quality of bugs and feedback)
• Top novice tester (same criteria)
• Top individual bugs in each of the sites (most interesting, most severe, best documented)
• Best survey responses (comparing the feature set, functionality and usability)

Visit our Bug Battle homepage for more details. Also, make sure you read our Tips to Win the Bug Battle in our uTest Forums.

So if you’re a uTester, get started today.  And if you’re not yet a uTester, join our community and take part in the fun.  Good luck!

In the past, we’ve brought you the Browser Brouhaha (Q408), a Social Networking Skirmish (Q109) and Twitter App Apocolypse (Q209).   Well it’s that time again, so we’re going to be kicking off our Q3 Bug Battle later this week.  For this quarter’s competition, we’ve chosen to turn our testing community loose on the most popular search engines:  Google, Bing and Yahoo!

The Q3 Bug Battle will begin on Friday, August 7th and end on Friday, the 14th.  We’ll be awarding more than $3,000 in prize money to the top testers, based on:

Read more…

Beta software tends to frighten Big Business. They’re skeptical of the phrase “almost done,” and for good reason. They demand a finished product. There are very few exceptions to this.

Take the recent news from Google as an example. A few days ago, the company finally removed the “beta” tag from Gmail and several other apps. Most people didn’t realize Gmail was still in beta (or what beta meant, for that matter) and most people didn’t care. So why bother with the change? This article from The New York Times explains why:

Practically speaking, the change will mean precious little to Gmail’s millions of users. But it could help Google’s efforts to get the paid version of its package of applications, which includes Gmail, Calendar, Docs and other products, adopted inside big companies. Corporate technology managers tend to shy away from beta products, and Google wants to remove any barriers to adoption that it can.

And there you have it: Although the meaning of beta has changed slightly, the perception of it hasn’t. Even though Gmail was essentially a completed product, with tons of great features and no major usability/functional issues, the beta perception was preventing them selling it to the larger companies.  It might not be fair, but it’s true.

Read more…

Once a month, we’re going to “test the limits”, interviewing a leading thinker in the world of testing and quality.  It could be a journalist, an industry analyst or an exec from a top software company.  To kick this program off, we could think of no better person than our good friend, Dr. James Whittaker.  So we recently interviewed James by bouncing emails back & forth over the course of a few days.

Several of these questions came directly from our community of testers.  The whole exchange is fairly lengthy, so we’re splitting it into two posts.  Come back and check out the 2^nd half later this week.

uTest:  So the news is out about your move to Google. What prompted you to make this move?

JW:  I didn’t so much leave Microsoft and I did join Google. I was attracted by all the Googlers I met at conferences and what I read on their blogs about the way they test. When they offered me the opportunity to be a part of it, one might even argue an important part of it, I found it impossible to decline.

uTest:  Is there something about Microsoft you’ll miss the most?

JW:  Yes, the breadth of both products and expertise. You literally have every type of software imaginable and a chance to collaborate with the people who make that software. From an intellectual standpoint, Microsoft is mind-blowing.

uTest:  What specific work at Microsoft did you enjoy the most?

Read more…

As consumers, we’re loyal to our brands.  Whether it’s Mac vs. PC, Nike vs. Adidas, or Starbucks vs. Dunkin Donuts, we like what we like.  And it’s tough to convince us that some other product/service/website is as good as our favorite.  Companies go to great lengths and expense to try and alter our preferences (who could forget the Pepsi Challenge?), but all too often, we walk away feeling marketed to (read as: manipulated).

Last week, Microsoft launched it’s new search decision engine – Bing – amid great fanfare.  And one enterprising MSFT employee named Michael Kordahi launched an app that he calls BlindSearch, which offers a “taste test” of the top three search engines.  Simply enter in a search term and BlindSearch will pull the results from Google, Bing and Yahoo into one interface… then you can vote for your favorite based only on the quality of the search results.

As Michael notes, this isn’t a perfect test… after all, we choose our search engine based on more than just the results (speed, overall user experience, number & placement of ads, other filters and features).  But when we search, we’re generally looking for the most accurately matched results.  While there are a lot of ways to perform user preference and usability testing — surveys, focus groups, A/B analytics — this is an interesting and entertaining experiment and a cool way to test your own preferences.

Take the BlindSearch Test and then come back and tell us which search engine you actually prefer!

As we mentioned previously in this space, uTest was recently chosen to present at the Under the Radar event in the heart of The Valley.  With only 24 start-ups being handpicked from among hundreds to present at UTR, just being selected is an honor.  Their track record for finding future stars before they pop is beyond impressive.

So last Friday afternoon, in front of a crowd of 200+ start-up execs, VCs and members of the media, Doron took the stage to tell the uTest story — and to do it in six minutes or less.  At an event that was focused almost entirely on cloud computing (infrastructure, management, apps, etc.), we knew our vision of crowdsourced software testing would be the exception and we wondered how it would be received amidst all the cloud talk.

And while I’m unabashedly biased, my initial reaction was that our story was popular within the room.  Apparently I was right because, later that day, we learned that uTest had won the judges’ choice award in the “Applications” category.  A high honor indeed, as our judges were such notables as John Foley from InformationWeek, Matthew Glotzbach from Google and Robin Vasan from Mayfield Fund.

The entire experience was positive — some great meetings with prospective customers and partners, good meetings with members of the media, and another feather in our cap as we rewrite the rules of what’s possible in the world of software testing.

Check out the six minute presentation for yourself and drop us a comment with your thoughts.