Software Testing Blog

Nick Jones over at Gartner wrote a great piece about the current state of mobile apps.  It’s a must-read for mobile app developers and marketers.

His post is written about the iPhone, but it applies equally to Android, Blackberry or Symbian apps.  It also helps to explain why mobile apps have been the fastest growing segment of the uTest biz (phenomenal growth + still evolving market = a lot of testing to do).

Here are a few realizations from his recent trip to his local Apple store that seemed especially relevant for our readers (bullet points are his and the running commentary is mine) :

Read more…

With the summer sun finally paying us a visit in Boston, so too is the mobile testing space heating up out here! Tomorrow (7/16), Doron Reuveni will be speaking at the 2009 Software Industry Conference (SIC) at the Boston Marriott Quincy on how crowdsourcing is helping companies test their mobile apps.

With the new iPhone 3G S, Blackberry Storm 2, and the new Palm Pre, the smartphone apps competition is fiercer than ever! Mobile app developers are battling for media attention, consumer dollars, VC investments and more.

But, status quo testing doesn’t quite work for mobile apps. Multiple testing criteria — such as wireless carriers, handset makers and models, locations, languages, platforms — all must be taken into account. With crowdsourcing, any mobile app company can have world-class QA coverage.

In his session, Doron will examine the growing trend of crowdsourced software testing for mobile apps – highlighting its benefits, challenges and uses as compared with conventional QA options.

If you’re in town and want to meet up, shoot us a note!

Our guest blogger this month is Brad Sellick, a recent Bug Battle winner with expertise in mobile testing (see his full bio below). In this post, Brad examines the shortfalls of simulation tools in testing mobile applications.

There’s a challenge in developing and testing applications that is as old as the personal computer itself. The challenge being that the design, development, and testing of applications often takes place in an environment much different from “the real world” where users run and interact with those applications.

With the rise of mobile platform devices like the Blackberry and iPhone, I believe that challenge is greater now than at any time in the past. We now find ourselves producing applications in an environment (a desktop or laptop computer) that is completely different from the device the application will run on.

This challenge is compounded by the fact that development platforms have a very convenient method of testing a mobile application on the desktop via a simulator tool.

My own experience with iPhone development was a major eye-opener. I spent a lot of time building and testing applications on my Mac desktop and the simulator application. However, when I finally loaded an application onto an iPhone for the first time, it was a completely different experience.

Read more…

We hear it over and over from countless customers, QA managers and testers:  getting mobile apps tested is tough.

Compared to mobile, the worlds of web and desktop apps are mature and stable:  standards have emerged, the OS and browser universe is more mature, testing tools and best practices have been established (even though they’re always evolving).  Mobile, on the other hand, is still early in its overall lifecycle, so development and testing aren’t yet fully baked.

With that in mind, I came across an article from CNET yesterday titled, Testing the iPhone 3Gs (or why the phone gods hate product testers.  This piece underscores the difficulties that confront companies (even the well-equipped experts at the CNET labs) when trying to test mobile devices and apps.

Eric Franklin, a senior tech editor at CNET, was trying to test the talk time on the new 3Gs… sounds simple enough,  right?  Well, in Eric’s own words, he encountered a number of “roadblocks” and was “slapped back to reality” on more than one occasion.

Read more…

Yesterday’s post — announcing that we need more mobile app testers to keep up with rapidly growing demand — has drawn a great deal of interest from testers around the world (which, I suppose was kinda’ the point).  One of the questions that I’ve heard from a bunch of tester in the past 24 hours is “where do you have the most mobile testing demand?”.

The answer is that, while our mobile growth has been across a variety of handset makers, wireless carriers and geographies, our greatest areas of need are:

• iPhone app testers in the US and Canada
• Blackberry testers in Spain, France, Italy and Germany

So, for the dozens of mobile testers who signed up for the uTest marketplace in the last 24 hours, thank you and welcome aboard!  For anyone else who’s interested, just send us an email and tell us your location, as well as your handset maker(s) wireless carrier(s).

I know… these two words aren’t seen very often in today’s world.

As a marketplace, uTest builds supply (testers) and demand (customers) for software testing services.  And since we have 15,000+ testing pros from 152 countries around the world, we usually have more than enough supply.  But as uTest grows, we do run into scenarios where demand grows faster than supply.

This is exactly what has happened in mobile during 2009.  In the past three months, dozens of companies have come to uTest to get their mobile apps tested.  And, while we have a lot of mobile testers around the globe, we need more.  So consider this a very public announcement that uTest has mobile app testing work available and we are actively looking for more testers to keep up with demand.

If you are a testing pro (or if you know someone who is) who wants to make some extra money for testing iPhone, Blackberry and Android apps, sign up as a tester today.  For our current crop of QA pros, update your profile to tell us which mobile devices you own.

In many cases, mobile testing offers bigger payout than web or desktop app testing.  For example, a recent Blackberry app release is paying a guaranteed $100 for a completed test script, plus $20 for each approved bug.  While payment varies by release, the point is a tester can make good money in just a few hours of heads-down testing.

Currently, our most urgent need for mobile testers is in Europe, but with the way demand has been growing, we are soliciting mobile testers in the US, Canada, South America and Australia as well.

If you know of QA pros or developers who are willing and able to test mobile apps, send them a link to this post.  Likewise, if you know of websites, message boards, forums or e-newsletters where uTest can advertise to find more mobile QA professionals, drop me a comment.

Last week we announced that, during the first few months of ‘09, we’ve observed tremendous growth in mobile application testing within the uTest marketplace.  Since then, we’ve received a surprising number of requests for more info from mobile developers and members of the media.  It seems that testing and quality is a bit of a hot topic among those who spend their days & nights working on (or writing about) mobile.

One of the most common questions is whether our QA community has coverage for a range of mobile platforms like iPhone, Android and Blackberry.  The answer to all of the above is ‘yes’ (the answer to the multi-location, multi-language question is ‘yes’ as well).

This announcement has already resulted in several great articles from top-shelf tech sites, including eWeek and TechTarget, and got reported by mainstream media outlets such as MSNBC, MarketWatch, TMC.net and StreetInsider.

Over the weekend, as I pondered why this simple announcement resonated with so many mobile app companies and journalists, two thoughts came to mind:

1.  Shut up.  Don’t overthink this — just ride this wave!

2.  Testing for mobile apps is a widespread and serious problem with few viable solutions.  In fact, getting mobile apps properly tested is even tougher than web apps because the space is more complex and less mature.

And I guess that’s where a global community of professional testers plays nicely into the mobile app equation — because the hardest thing to build (global coverage across locations, languages, handsets, carriers, operating systems, etc.) is baked right in.  And the fact that our community can test mobile apps quickly, efficiently and on-demand doesn’t hurt either.

I’d love to hear from mobile app developers about how they test today, or from testers about the challenges that are unique to mobile. It’s a topic that’s going to be top-of-mind for the next year or more (just wait ’til mobile really clashes with the enterprise), so let’s start the discussion now.

In the past few months, we’ve been providing QA services on a bunch of mobile apps.  As a result, we’ve had the good fortune of meeting some really cool, creative entrepreneurs and developers who are building top-shelf apps for the iPhone, Blackberry, G1 and others.

So I’ve been keeping my eyes peeled for articles from leading mobile app entrepreneurs.  And while perusing Alley Insider, I came across an article from Dan Frommer.  Frommer highlights a fantastic first-hand account from mobile app veteran Craig Hockenberry, maker of iPhone hits such as Twitterrific and Frenzic.

Hockenberry’s 2,000 word piece is a must-read for aspiring iPhone app developers.  Frommer was kind enough to summarize it as follows:

- Learn how to develop Web pages for Mobile Safari so you know how to think about designing apps for a mobile device.
- Buy a Mac. You can’t make iPhone apps from a PC.
- Sign up for Apple’s developer program.
- Watch the “getting started” videos in the iPhone dev center.
- Goof around with some of the sample code on Apple’s site.
- Check out a few of the better iPhone coding books.
- Read up on a few of Craig’s old posts.
- Go for it!

In particular, Hockenberry mentions a few books iPhone app dev books:

If you’re just starting out, I’d highly recommend Beginning iPhone Development: Exploring the iPhone SDK by Dave Mark and Jeff LaMarche. The best thing about this book is the step-by-step approach it takes to working with Xcode, Objective-C and the iPhone APIs. They’ll lead you through the basics and you’ll be building your own apps in no time at all.

As you get more comfortable with the tools and AppKit/UIKit frameworks, I’d recommend you take a look at Erica Sadun’s iPhone Developer’s Cookbook: Building Applications with the iPhone SDK. This book presumes a bit more knowledge about the SDK, but is a very handy reference both to the official and unofficial APIs.

Since you’re going to be working with Cocoa Touch on the iPhone, you’ll also want to start thinking like a Cocoa programmer. Every great iPhone and Mac developer has nothing but wonderful things to say about Cocoa Programming for Mac OS X by Aaron Hillegass.

If you have previous development experience with C, C++ or Java, you’ll want to read this mailing list post by Erik Buck that enumerates some of the difficulties that you’ll have coming up to speed with Objective-C and Cocoa.

As we come across more of these practical, hands-on resources, we’ll share them here.  If you know of other helpful articles or posts, please send them our way.  For you future mobile mavens, I hope some of these resources are helpful.

Interested in building an iPhone App?  If so, there’s a must-read article over at Mashable.  In their words:

Ten23 Software has created an entire 37 page guide to the development process (below), decisions they made and what they learned during the creation of their PhotoKast app. Their hope is that the document might provide insights for other developers when they start out on iPhone App development projects.

Have experience building mobile apps for the iPhone, Blackberry, Android or other mobile platforms?  If so, drop us a note and share your experience and tips.

For all of you mobile app developers who dream of creating the next hot iPhone, Blackberry or G1 application, a key question to ponder is this:  once you’ve conceived, developed, tested and launched your killer app, how long will it remain killer?

Well, TechCrunch highlights the answer from a recent Pinch Media presentation, and it’s not the cheeriest news for mobile app developers.  It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it.  And by day 30, less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.  Grim.

The moral of the TechCrunch story is this:

It answers the eternal question that all iPhone developers have: Should my app be free or should I charge for it? For all but the most successful apps, the free route does not make much sense because there is not enough time to recoup the costs of developing the app from advertising.

Free apps tend to be run 6.6 times more often than paid apps, but even with that increased usage, it is not enough to make more money.

Yardley estimates that less than 5 percent of all apps would make more money right now with advertising than charging for paid downloads. His advice: “Unless there is something inherent about the app that screams free, sell it.”

I completely agree with Yardley’s math and logic, but I think there’s another important lesson to be learned here.  He addresses the issue of maximizing revenue, but ignores cost containment as a means to profitability.  Said differently, another powerful driver of profitablity is the cost and speed in bringing your mobile application to market.

By finding ways to develop, test and launch mobile apps more quickly and for less money, developers extend the money-making window, enable themselves to launch more apps per year, and decrease their break-even levels.

What do you think — what’s the secret to profitability for the creators of iPhone, Android and Blackberry apps?  Drop a comment and drop some knowledge on us.