Tag Archives | testing

Name That Plague (software testing plague, that is)

Frequent readers of the uTest blog are by now aware that we’re big fans of James Whittaker – software testing expert,james_whittaker author and now one of Google’s top QA guys. Over the past few weeks, James has been writing a provocative series called the “Seven Plagues of Software Testing”. You can find it on his new blog.

As you’ll notice, only six of the plagues have been published thus far. The seventh and final software testing plague was intentionally omitted, as he is accepting submissions from his readers.

Before you send him your suggestions (we included an email address at the bottom of this post), here’s a few excerpts from some of the plagues he’s discussed so far:

The Plague of Blindness: “Software testing is much like game playing while blindfolded. We can’t see bugs; we can’t see coverage; we can’t see code changes. This information, so valuable to us as testers, is hidden in useless static reports. If someone outfitted us with an actual blindfold, we might not even notice.”

Continue Reading →

Continue Reading

Bug Battle Part III: Twitter Apps!

After web browsers and social media sites, you were probably wondering how we’d top ourselves for the next uTest Bug Battle. Well, after months of debate and deliberation, uTesters will compete to find bugs in six of the top Twitter desktop apps!

Unless you’ve been asleep, in a coma, or camping in the wilderness for the past two years, you’re likely aware that the Twitterverse is expanding at a mind boggling rate (thanks, Ashton and Oprah). And since hundreds of apps have been created around the popular micro-blogging site, we figured they’d make a perfect subject for our now famous Bug Battle. Here are the Twitter apps we’ll be testing, in no particular order:

1.    Tweet Deck 0.25
2.    Seesmic DESKTOP
3.    Twhirl 0.9.2
4.    Tweetr 3.4
5.    Twitterific 3.2 (mac only)
6.    Twitteroo 1.5 (pc only)

The contest kicks off RIGHT NOW (12:01am ET on Thursday) and will run through 11:59pm ET next Wednesday, June 3rd.

During that time, members of our QA community will be searching for the most serious, compelling bugs they can find, including technical, functional and GUI bugs. uTesters can focus their efforts on as many or as few of these as they choose.  We will be awarding more than $3,000 in prize money for:

* Top overall tester (based on quality of bugs and feedback)
* Top novice tester (same criteria)
* Top US-based tester (same)
* Top five individual bugs (severity and complexity)
* Best feedback (post-contest survey about the feature set, functionality and usability)

You can get more info on the Twitter Bug Battle. To find out how you – yes you! – can win the money finding bugs in these Twitter applications, be sure to check out the Bug Battle thread in our testers-only forums.

Have questions or ideas for future Bug Battles? Drop us a comment. Want to keep up with the Bug Battle action while it’s going on?  We’ll be sharing real-time updates throughout the weeklong Twitter contest on (where else?) Twitter.

Continue Reading

uTest Relaunch in the News!

The word is out!  With the launch of our revamped website, a new “Meet the Testers” application and Tester Forums, last week uTest made significant strides in turning our crowd of 16,000+ QA professionals into an interactive community.

Here’s what the media had to say:

  • crowdsourcingformatted1Jeff Howe, author of best-selling book Crowdsourcing & contributing editor at Wired Magazine, twittered the news to his 2,000 followers calling us out as a Company to Watch.

Other notable posts at  StickyMinds.com, the Cloud Architect, and MSNBC.com.

Let us know what you think about the new site!

Continue Reading

uTest Hitting The Road This Week

This evening, Doron Reuveni will be joining a new meetup here in Boston called the Ultra Light Startups. The meetup is a resource for entrepreneurs to share best practices for launching tech startups. The topic is near andultralight dear to our hearts: Crowdsourcing. If you’re in the neighborhood, come join us at Boston University at 595 Comm. Ave. Doors open at 6pm! Questions the panelists (Local Motors, Acquia, GeniusRocket and uTest) will tackle:

  • What is crowdsourcing?
  • What are the benefits?
  • Why is it a disruptive model?
  • What are the most effective ways to build communities?

Doron will also be flying down to Orlando to present at STAREAST 2009 on Thursday morning. His presentation will introduce this new era of community-based software testing and delve into how companies can launch higher quality apps while staying within budget through crowdsourcing. Challenges like shorter release cycles, increased customer expectations, smaller budgets and fewer testing resources are forcing us to rethink our stareastQA methods.

Doron will discuss how crowdsourced testing helps to meet these challenges head on. Here’s your chance to take the kids to Disney and be a part of the largest and most advanced testing forum to keep you up on the latest trends, technologies, and strategies in the industry today. Shoot us a note if you’ll be down there!

Continue Reading

Testing Tip: Screenshots on Firefox

Without a doubt, one of the most important things you can submit with a bug is a screenshot.  For those of you using Firefox, check out the excellent Screengrab extension.  It’s completely free and allows you to take a screenshot of the entire page at once rather than just what’s visible in your window.  It also saves the files in PNG or JPG formats – making them smaller and more manageable than TIFF or BMP files.

Take a look at this screenshot of the homepage for the Software Testing Club:

software-testing-club-online-qa-software-testing-community_1239721283757

Only half of that would be visible in an ordinary browser window.  The plugin also lets you capture selections of a page, so you can also use it to highlight important errors.

Here’s a bonus link.  Check out this excellent post on Mashable with 7 Useful Tools for Web Development Testing.

Continue Reading

Bug Battle Builds Big Buzz

Apologies for the alliteration, but things have been ultra-busy around the halls of uTest this week.  The reason our phones have been ringing off the hook is because we announced the results of our 1st Quarter Bug Battle on Tuesday.

In case you missed it, we’ve received some fantastic media coverage in the past few days.  Who knew the world would be so interested in the results of 1,000+ QA professionals from 64 countries simultaneously testing the world’s three most popular networking sites?  Ok, well we had a pretty good idea they’d care, but we’re glad it captured their attention.  Here are a few of the noteworthy articles:

– TechCrunch: uTest Bug Battle: Which Social Network Is The Buggiest?
– Dr. Dobbs: Bugs In Social Networking Software? You’re Kidding, Right?
– eWeekFacebook Triumphs LinkedIn, MySpace in uTest’s Bug Battle

Our community’s ability to mobilize quickly and provide real-world testing coverage is making a strong impression on a growing number of companies.  These firms are looking for ways to do more with less — and the uTest community is primed to help them meet their QA needs.  Keep up the good work!

Continue Reading

How Many Bugs Do You See?

Like many Americans, I’ve been watching housing prices very intently.  While looking at data on the website for the Massachusetts Association of Realtors, I came across a very common and ordinary web bug: a broken link.

Screenshot

(Click the thumbnail for a full sized version.)

Now broken links are incredibly common, and anyone with a website has had one happen to them at some point.  But I want to dig a little deeper.  Take a look at that screen shot and think about how many bugs do you see?

By my count, I see three very real bugs:

  • There’s the broken link.  Looking at this error, I think this is a convoluted way of saying that the file I wanted wasn’t found.  Whoever uploaded the file didn’t set a folder correctly.
  • The Folder is not set exception didn’t get handled.  Ideally, some kind of exception handler should have caught this exception and provided the user with a more coherent error message.
  • The user was shown an incoherent error message.  An end user should never see this kind of error message in the first place.  It reveals information about the underlying system configuration that could be dangerous.  At the same time the user doesn’t receive any kind of friendly help about why the error occurred or how to resolve it.

Each of these is a very real bug, but each one is also more difficult to fix than the previous.  For example, fixing a broken link may be as simple as changing a file location in the content management system.  The person responsible for posting the content could take care of that.

Correcting the next bug requires help from the developers to write a proper exception handler to catch this exception. While fixing a broken link is something that could be handled in the field, writing a new exception handler usually has to be done during the development process.  This, of course, requires a solid QA process to be in place during development.

For most projects, the last bug is the most challenging to fix.  It usually requires careful planning from the earliest stages by product management, development, and QA to agree that reducing the complexity of bugs is both important and feasible.  Software testers will have to be involved early in the process so that they can understand how to discover and interpret bugs that are designed to be customer friendly or even invisible.

Even simple bugs, like broken links, can lead to deeper questions for testing and development teams.  Not every bug has to be fixed, but make sure to keep your customers and users in mind when planning your project.  Good planning could turn a boring broken link into something that’s both apologetic and helpful.

Continue Reading

uTest Hits The Big Apple For FutureTest and TechAviv

This week, Doron, Roy and I hopped a train to NYC.  And while most people go for a Broadway show, a Yankees game or the latest exhibit at The Guggenheim, for the three of us, it was a doubleheader day at FutureTest2009, and TechAviv NYC.

uTest was asked to present at both events on the same day.  Needless to say, it was an eventful Wednesday, but we had a great time meeting a wide range of people — from leaders across the QA industry at FutureTest to start-up mavens at TechAviv.

At FutureTest, Doron shared the story of launching uTest, and the steps we took to design and build a community-driven site.  He talked about how the most successful new ventures plan their business, their audience and their offering, before they design or develop their site.  He also discussed the role that testing should play before, during and after the launch of a successful web app.  You can view Doron’s full presentation here.

At TechAviv, Doron performed a live demo of the uTest platform.  He showed how quick and easy it is for companies to launch a test cycle, to see bugs being reported in real-time, and to manage the entire test cycle.  We followed that up with a vigorous Q&A with the entrepreneurs in the crowd.

What was fascinating to me was how different these two crowds were — one group of QA experts from some of the largest enterprises and government agencies; the other, a crowd of bright-eyed, can-do entrepreneurs.  And yet, both were extremely engaged, asked a lot of great questions, and showed significant and genuine interest in the uTest concept.

Like I said, busy day.  But I’d take 50 more Wednesdays just like it over the next year.

If you’re a member of any testing or entrepreneurial groups and you’d like Doron or another uTest exec to speak at an upcoming event, drop us a comment or shoot me an email at mattj [at] utest.com.

Continue Reading

A Real-World Primer For Building iPhone Apps

In the past few months, we’ve been providing QA services on a bunch of mobile apps.  As a result, we’ve had the good fortune of meeting some really cool, creative entrepreneurs and developers who are building top-shelf apps for the iPhone, Blackberry, G1 and others.

So I’ve been keeping my eyes peeled for articles from leading mobile app entrepreneurs.  And while perusing Alley Insider, I came across an article from Dan Frommer.  Frommer highlights a fantastic first-hand account from mobile app veteran Craig Hockenberry, maker of iPhone hits such as Twitterrific and Frenzic.

Hockenberry’s 2,000 word piece is a must-read for aspiring iPhone app developers.  Frommer was kind enough to summarize it as follows:

– Learn how to develop Web pages for Mobile Safari so you know how to think about designing apps for a mobile device.
– Buy a Mac. You can’t make iPhone apps from a PC.
– Sign up for Apple’s developer program.
– Watch the “getting started” videos in the iPhone dev center.
– Goof around with some of the sample code on Apple’s site.
– Check out a few of the better iPhone coding books.
– Read up on a few of Craig’s old posts.
– Go for it!

In particular, Hockenberry mentions a few books iPhone app dev books:

If you’re just starting out, I’d highly recommend Beginning iPhone Development: Exploring the iPhone SDK by Dave Mark and Jeff LaMarche. The best thing about this book is the step-by-step approach it takes to working with Xcode, Objective-C and the iPhone APIs. They’ll lead you through the basics and you’ll be building your own apps in no time at all.

As you get more comfortable with the tools and AppKit/UIKit frameworks, I’d recommend you take a look at Erica Sadun’s iPhone Developer’s Cookbook: Building Applications with the iPhone SDK. This book presumes a bit more knowledge about the SDK, but is a very handy reference both to the official and unofficial APIs.

Since you’re going to be working with Cocoa Touch on the iPhone, you’ll also want to start thinking like a Cocoa programmer. Every great iPhone and Mac developer has nothing but wonderful things to say about Cocoa Programming for Mac OS X by Aaron Hillegass.

If you have previous development experience with C, C++ or Java, you’ll want to read this mailing list post by Erik Buck that enumerates some of the difficulties that you’ll have coming up to speed with Objective-C and Cocoa.

As we come across more of these practical, hands-on resources, we’ll share them here.  If you know of other helpful articles or posts, please send them our way.  For you future mobile mavens, I hope some of these resources are helpful.

Continue Reading

How Long Does An iPhone App Stay Fresh?

For all of you mobile app developers who dream of creating the next hot iPhone, Blackberry or G1 application, a key question to ponder is this:  once you’ve conceived, developed, tested and launched your killer app, how long will it remain killer?

Well, TechCrunch highlights the answer from a recent Pinch Media presentation, and it’s not the cheeriest news for mobile app developers.  It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it.  And by day 30, less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.  Grim.

The moral of the TechCrunch story is this:

It answers the eternal question that all iPhone developers have: Should my app be free or should I charge for it? For all but the most successful apps, the free route does not make much sense because there is not enough time to recoup the costs of developing the app from advertising.

Free apps tend to be run 6.6 times more often than paid apps, but even with that increased usage, it is not enough to make more money.

Yardley estimates that less than 5 percent of all apps would make more money right now with advertising than charging for paid downloads. His advice: “Unless there is something inherent about the app that screams free, sell it.”

I completely agree with Yardley’s math and logic, but I think there’s another important lesson to be learned here.  He addresses the issue of maximizing revenue, but ignores cost containment as a means to profitability.  Said differently, another powerful driver of profitablity is the cost and speed in bringing your mobile application to market.

By finding ways to develop, test and launch mobile apps more quickly and for less money, developers extend the money-making window, enable themselves to launch more apps per year, and decrease their break-even levels.

What do you think — what’s the secret to profitability for the creators of iPhone, Android and Blackberry apps?  Drop a comment and drop some knowledge on us.

Continue Reading

It’s Bug Battle Time: Facebook vs. LinkedIn vs. MySpace

In Q4, we held our first-ever Bug Battle — the “Battle of the Browsers”.  This QA contest yielded 672 bugs within IE, Firefox, and Chrome.  1,300+ testers participated and uTest awarded nearly $3,000 in prize money.  Now it’s time for our next Bug Battle challenge.

After a great deal of discussion with bloggers, analysts and members of our testing community, we’ve selected a fun and high-profile “battleground”:  Social networking sites.

Join other uTesters from around the world in testing the application quality and usability of the top three social networks:  Facebook, LinkedIn and MySpace.

The contest begins Wednesday, Feb 11th and runs through Tuesday, Feb 17th.  uTest will once again be awarding $3,000 in prize money for:

  • Top overall tester (based on quantity and quality of bugs and feedback)
  • Top novice tester (same criteria)
  • Top individual bugs in each of the sites (highest profile, most interesting, most severe)
  • Best feedback (about the feature set, functionality and usability)

See complete details for the uTest Social Networking Bug Battle.  Get ready to hunt some bugs, make some serious money, and be recognized as one of the top testers in the community!

Have questions or ideas for future Bug Battles?  Drop us a comment.

Continue Reading

Buggy Software – A Strategic Choice

Buggy products can be a real customer turn-off.  Witness the recent release of the BlackBerry Storm, Research In Motion’s response to Apple’s iPhone.  The Storm had tremendous promise as a great new touchscreen phone, but customer response has been limited because of early bugs in the device’s software.  Here’s a quote from one Storm customer in an article from the Wall Street Journal:

“I found myself wanting to throw it in the ocean due to my frustration with its overall usability,” said Steven Golub, a longtime Verizon customer from Morristown, N.J., who bought the Storm the day it was released, but returned it a few weeks later.

That’s pretty damning, but let’s stop and give RIM some benefit of the doubt.  Buggy software is a customer turn-off, and undoubtedly bad reviews will dampen customer enthusiasm.  But here’s a quote from the same Wall Street Journal article:

Verizon and RIM, determined to release the Storm in time for the holidays, rushed the device to market despite glitches in the stability of the phone’s operating system, according to people close to the launch.

RIM had a choice to make – release a buggy product in time for the holidays, taking bad reviews on the chin, or wait until the Storm worked better and try to gain market share during the traditionally slow Q1 made even slower by a bad economy.  We really can’t know for sure if RIM made the right decision, but there are some lessons other companies can learn here:

1.) Know Your Limitations – We all have limitations in our planning: not enough time, not enough people, or not enough testing.  Maybe your budget is limited or maybe you need to hit a holiday launch date.  Either way, it’s very important to be aware of your limitations early in the process.  If you have a tight schedule and a firm launch date, then you should make sure you have enough resources for last-minute development and testing.  If you don’t have enough people, then you should evaluate different contingencies for alternate launch dates.

2.) Work Smarter – You may feel like you have to cut corners to overcome your limitations, but the Internet can make it easy to expand your efficiency on-demand.  There are many companies that can help you add capacity to overcome bottlenecks and release a better product on time.  The uTest software testing service is perfect for helping companies improve their testing, but solutions exist for everything from coding to graphic design.

3.) Have Good Customer Data – If you’ve already passed the point of no return, then you will have to make a tough decision.  In this case, nothing helps more than really good customer data.  In RIM’s case, they had to decide between weaker customer demand because of timing or because of bugs.  Between the two, they chose to cast their lot with a buggier product over a late product.

RIM still sold 500,000 Storms over the holidays, and that’s not bad.  This is also their first touchscreen device and it holds tremendous design potential.  While Storm Version 1 had a rocky launch, Storm Version 2 may take the world by, well, storm.

Continue Reading