Software Testing Blog

This is the second half of our recent interview with testing guru, James Whittaker.  Today, we’ll cover his new book, his new gig and what he sees over the horizon in the world of software testing.  If you haven’t read it already, check out the first half of the interview.

uTest:  And when all is said and done what will be the professional accomplishment you’ll look back on with the most pride?

JW:  Creating an actual discipline around software quality. Note I said quality and not testing. I want software projects as a whole to run more smoothly and more predictably. I really think that’s what software testing is all about — reducing the uncertainty of software development and finding ways to muscle errors out of the process. A process in which mistakes are harder than doing the right thing is the ultimate goal. We can’t eliminate them, but we can make doing the right thing to be the easiest thing to do.

uTest:  What’s your first assignment at Google?

JW:  To raise the level of testing precision and diligence. Google has a lot of smart testers, my job is to help mold them into a serious fighting force and let our bugs beware. But this isn’t so much an individual commitment. Google has a culture of collaboration that I am fascinated by as a Noogler.

We share offices (which might explain their interview strategy), inhabit common areas, collaborate constantly and work as a community. If I am successful, there will be many people who can take credit and if I fail, I won’t go down alone! I think the whole free food thing is at the heart of this as food is often the centerpiece for bringing people together. Lots of work gets done while your mouth is full. I hope to succeed before I have to buy bigger clothes.

uTest:  Rumor has it that you have a new book coming out.  What’s it about and when will it hit Amazon’s shelves?

Read more…

Once a month, we’re going to “test the limits”, interviewing a leading thinker in the world of testing and quality.  It could be a journalist, an industry analyst or an exec from a top software company.  To kick this program off, we could think of no better person than our good friend, Dr. James Whittaker.  So we recently interviewed James by bouncing emails back & forth over the course of a few days.

Several of these questions came directly from our community of testers.  The whole exchange is fairly lengthy, so we’re splitting it into two posts.  Come back and check out the 2^nd half later this week.

uTest:  So the news is out about your move to Google. What prompted you to make this move?

JW:  I didn’t so much leave Microsoft and I did join Google. I was attracted by all the Googlers I met at conferences and what I read on their blogs about the way they test. When they offered me the opportunity to be a part of it, one might even argue an important part of it, I found it impossible to decline.

uTest:  Is there something about Microsoft you’ll miss the most?

JW:  Yes, the breadth of both products and expertise. You literally have every type of software imaginable and a chance to collaborate with the people who make that software. From an intellectual standpoint, Microsoft is mind-blowing.

uTest:  What specific work at Microsoft did you enjoy the most?

Read more…

In my kitchen I have a mandoline.  No, it’s not a musical instrument (that’s a mandolin), but instead a fancy food slicer that can thinly cut or dice just about anything.  Anything, that is, including my fingers.

You see, I am extremely clumsy with my mandoline.  Every time I’ve used it, I have somehow sliced open one of my fingers.  My mandoline is literally covered in knives - there are knives for slicing, knives for dicing, and even more knives stored underneath and inside the thing for torturing vegetables in additional ruthless and unimaginable ways.  The whole thing is one big vegetable and finger slicing machine.

The crazy thing about my mandoline is that despite the fact that it cuts me every time I even look at it, it’s not actually broken.  In fact, it’s in great shape.  All of its knives are sharp, its frame is sturdy, and it slices and dices just like one would expect.  I’m sure there is some kind of spec somewhere that says “verify mandoline can smoothly cut onions exactly .125 inches (3.17 mm) thick.”  I’m sure mine passes with flying colors.  What the spec probably doesn’t say is “verify mandoline won’t cut fingers.”

Specs can’t capture these sorts of negative tests very well.  You can write them, but a spec like that is a lot like a spec that says “make sure your software doesn’t crash.”  It’s very nice to include, but also kind of worthless.  This is where good testers can really show their worth.  A tester can think beyond the spec and relay feedback that can be just as valuable as any bug or error.

What are some of the evil mandolines in your projects?  If you relayed them to your developers or product managers, do you think you could make your software better?

Our guest blogger this month is Joseph Ours, a recent Bug Battle winner with more than 12 years of IT experience, including software testing and quality assurance. In this post, Joseph advises testers to re-examine the way they report defects in software applications.

Testers and testing are viewed as a cost center in many organizations. If you look at the roles of other “main” players, you quickly see that testers face what I call an issue of intangibles. Here’s what I mean:

• Project managers - They are task masters driving a product to completion. Businesses absolutely want products created on time and under budget - which is why they are (correctly) viewed as an absolute necessity.

• Analysts - These guys get the great job of descriptively conceptualizing the idea. This is akin to a paper prototype, and gives the business the first real glimpse of how it might look and work.

• Developers - They are the cream of the crop. They get to create an actual product that businesses can see and feel.

• Testers - Well, we say if it works or not.

Read more…

And by trophies, we mean nearly $4000 in cash prizes.

Yes, the results of the Battle of the Desktop Twitter Apps are final. The week-long QA competition - in which more than 600 testers  searched for bugs in five of the most popular Twitter apps - was a  success. As expected, we received hundreds (320 to be exact) of interesting GUI, technical and functional bugs, so selecting a handful of winners was obviously no easy task.

In fact, it was so difficult that we decided to expand our winners list. Also, be sure to download our free special report (PDF), detailing the results of the post-battle usability survey.

As you’ll see from the results page, the Battle of the Desktop Twitter Apps saw several repeat winners, along with some fresh, new uTest talent. Here’s a top-line list of the winners:

• Top Tester: Bernard Lelchuck (Israel)
• Top Novice: Tyler Ritchie (USA)
• Top US Tester: Joseph Ours

Tyler and Bernard also finished 1st and 2nd in the “Best Bug” category. Other winners for “Best Bug” included:

• Claire Pelayo (Philippines)
• Bryan Fisk (New Zealand)
• Aymen Chehaider (India)

As far as the apps were concerned, the uTest community ranked Tweet Deck #1 in terms of usability, feature set and overall quality. Coming in second (also in every category) was Seesmic Desktop, followed by Twirl, Tweetr and Twitteroo.

The results have already been featured in a Mashable story yesterday, ase well as a bunch of other news outlets, and of course, hundreds of individual tweets.

Once again, congratulations to all of the Bug Battle winners, and thanks to all those from our community who participated. If you’re aleady looking forward to our next competition - and we know you are - be sure to send us your ideas. We can be reached at marketing@utest.com, and of course, on Twitter.

Last night, at a reception in Times Square in the heart of Manhattan, uTest won the greatest honor in our brief history:  the American Business Awards presented us with the Stevie award for the Top New Company of 2009.

We’re sincerely honored (and yes, a tad bit surprised) to win this award when we launched our software testing marketplace just 10 short months ago. The “top new company” category was full of worthy and well-established finalists, including Aster Data Systems, Carpio, Evolve IP, HubSpot, Jivox, Mazooma and Zuora.

The 2009 Stevies saw more than 2,600 entries submitted from companies of all sizes and industries, and more than 200 executives across the country participated in the judging process. The ceremonies were hosted by Cheryl Casone of Fox Business Network and broadcast nationwide on radio by the Business TalkRadio Network. We’ll try to track down some pics and vids from the festivities and post them later.

As Doron said when he accepted the award, this is gratefully accepted on behalf of 20 uTest employees, nearly 18,000 testers in our community and more than 120 software companies that turn to us to help get their apps tested. We couldn’t do it without you all!

Tonight we released the version 2.3 of the uTest platform.  There are a lot of fixes and features in this release, but we’re most excited about two new features:

Discussion Threads

Starting today, testing managers have a new tool for working with the uTest community.  With discussion threads, it’s now possible to quickly and easily create a virtual meeting room so they can communicate with their testers and the uTest project manager.

Testers who are working on a project can “meet” in real-time, ask & answer questions and share information with each other about that particular release.  And customers have a great new way to share critical information or requirements while a release is underway.

Discussion threads will enable testing managers to better manage and interact with their virtual testing teams and ultimately create even higher quality apps.

(Click to see a larger screenshot.)

Application Profiling

Another great feature we’re introducing is a tool to help testing managers better match their applications to testers expertise through refined application profiling.  When creating a new release, testing managers will have the ability to classify their application and testing needs.  Our system will use then use this information to automatically identify testers who are more likely to match the release’s exact testing needs.

The drop-down menus will allow testing managers to select:

• Application Type - Web, Desktop, Mobile
• Testing Type - Functional, Usability, or Load & Performance
• Industry Type - Social Networking, Retail, Entertainment/Gaming, Financial Services, etc.
• Audience - Consumer, Business, or Both

This matching process is just the beginning, and coming soon we will release several additional features for both testers and testing managers.  In the coming month, we will be requiring uTesters to update their profiles for even stronger project matching.

In addition to these new features, this release applied fixes to more than 40 bugs (all found by our uTester community).

Think crowdsourcing is a fad?  Think it won’t last?  Then don’t read the latest edition of BusinessWeek, which has a couple of insightful pieces on the topic from Jon Winsor.  While it’s widely acknowledged that the current global economic conditions have enabled B2B crowdsourcing to grow more quickly than they would have otherwise, Winsor believes that this community-driven movement will outlive any economic cycle:

Some might think once the global financial crisis has come to an end these marketplaces will dwindle as people go back to work.  Instead, I believe that they will evolve further, supplying a more efficient and creative way for companies to engage with and harness the crowd for help.

uTest is included in BusinessWeek’s article on crowdsourcing leaders and the related slideshow uses a great quote from our CEO.

You can probably guess what we think about this topic:  open marketplaces and crowdsourcing are the most efficient, frictionless, collaborative way to deliver professional services like software testing!  But we’d love to hear from you.  Is crowdsourcing the natural evolution of outsourcing (as James Whitaker has said) and thus, here to stay?

A quick heads up: The Twitter Universe might - just might - collapse on itself tomorrow around noon. We apologize for the short notice.

According to these guys:

The Twitpocalypse is similar to the Y2K bug. Very soon the unique identifier associated to each tweet will exceed 2,147,483,647¹

For some of your favorite third-party Twitter services not designed to handle such a case, the sequence will suddenly turn into negative numbers. At this point, they are very likely to malfunction or crash.

Like the over-hyped Y2k bug, we’re a little skeptical of this scenario actually occurring. And because none of our expert testers found anything like this in our recent Twitter App Bug Battle (not that it’s your typical bug), we’re fairly confident the Twitterverse will continue its infinite expansion.

Of course, we could be wrong. In which case, popular apps like Twirl, Tweet Deck and Twitteroo will all cease to function, and one can only assume what kind of chaos might then ensue. To quote Dr. Peter Venkman in Ghostbusters: “Human sacrifice, dogs and cats living together… mass hysteria!”

Maybe not, but it’s something to keep an eye on either way.

On Monday, Apple announced a whole bunch of exciting and innovative new products.  Two of the more anticipated were the iPhone 3GS and version 3.0 of the iPhone software.  Both of these introduce interesting new features for iPhone users everywhere.  You can watch the entire keynote here.

Apple’s keynote presentation went very well, but there were a couple of snags while demoing some of the new iPhone apps.  Two different apps had problems with their demos.  One app had an issue with some on-stage props, but the other simply stopped working in the middle of its presentation.

The presenters for both apps recovered and continued to highlight other features of their products.  However, I can’t exactly fault them for having trouble.  In fact, what actually surprised me was that so many of the apps worked given how difficult Apple makes testing iPhone apps.

Read more…