Software Testing Blog

We all know a mobile app released with a poor user experience pretty much guarantees its failure in the app stores. NHS Direct, an extension of the UK’s National Health Service, could not afford to take that risk (after all, there were patients on the line, literally), and turned to uTest for usability testing before their app hit the Android Market and Apple App Store.

Originally designed to be used via landline phones, NHS Direct – a service giving patients the ability to assess symptoms, get self-care and advice from nurses on call – eventually migrated to the web, and most recently, to smartphones.

Charlie Young was responsible for overseeing testing of the the app across iOS and Android OS devices. Lacking the in-house resources needed for this type of project, Charlie turned his attention to uTest.

“The safety of the application wasn’t an issue at that point,” said Charlie, who mentioned that the clinical requirements had already been met. “What we really needed to know was how humans would use this app. People use smartphones in many different ways, so we needed to make sure the user experience lived up to expectations – and that’s where uTest provided such value.”

Read more…

Apps! Apps! And more apps! As the summer starts winding down here at uTest, we’ve been able to take a step back and a closer look at the big trends emerging all around us. What has been most apparent is the tremendous spike in mobile app testing needs. From top marketing agencies to retail giants to social gaming startups, our customers are developing more mobile apps to grow (or define) their businesses than ever before.

According to Game Developer Research, 25% of game developers are now making mobile games – that’s up from a mere 12% in 2009!

In addition, a survey conducted by iGR found that more than half (53%) of US mobile developers are building apps for Apple’s iPhone OS. BlackBerry was the next most popular, followed by Android and Windows Mobile.

In response to this incredible momentum, this year marks the launch of Mobile App World 2010, where global leaders in mobile tech and app development and entrepreneurs will gather to network and learn about the latest developments and innovations.

uTest will be among the outstanding line-up of more than 40 speakers, which includes Google, Microsoft, Ericsson, Orange Global and the BBC, who will be discussing the future of mobile apps. Shoot us a note if you’ll be around!

Note: If you’re looking for some cool, new mobile apps, check out Mobile App World’s August Apps Of The Month. You may spot a uTester’s favorite app!

So maybe it wasn’t AT&T’s fault after all.

Apple recently revealed that there is a fundamental flaw in their method for calculating how many signal bars to display.  And we have the iPhone 4 (and its “learn to hold your phone the right way” fiasco) to thank for bringing this software snafu to light.

CNN Money shares the following details from Apple:

“Upon investigation, we were stunned to find that the formula we use to calculate how many bars of signal strength to display is totally wrong,” Apple wrote in a statement posted on its website. “Our formula, in many instances, mistakenly displays 2 more bars than it should for a given signal strength.”

That means, for example, that iPhones sometimes display four bars when they should be displaying two. Apple said users reporting a significant drop in bars when they hold their iPhone 4 are probably in an area of “very weak signal strength” but were unaware of that because the phone displayed four to five bars.

“Their big drop in bars is because their high bars were never real in the first place,” the company said.

Perhaps most surprising, Apple disclosed that the problem is not confined to the iPhone 4.  The faulty formula has been present in every iPhone model since the 2007 original.  Questions remain about whether the issue is strictly software-related, or if it also involved hardware problems.  However, Apple has said it will release a free software update in the next several weeks to fix the glitch. It will use a new formula recommended by AT&T.

The major networks have been engaged in a decades-long struggle to win the hearts and minds (and eyes) of viewers. Whether it’s news, sports or sitcoms, these battles are now being fought on multiple fronts – including the distribution of their content through their web and mobile applications.

Which brings me to our latest bug-hunting competition. This quarter’s uTest Bug Battle will challenge testers to find bugs on the web AND mobile apps of the four major TV networks: CBS, NBC, ABC and FOX.

The competition gets started this Friday, February 5 at Noon (ET). At that time, testers from our community will be given one week to search these sites for the most compelling bugs, and to report them through our online platform. We’ll be dishing out nearly $4,000 in prize money for categories like Top Tester, Best Bug and Best Survey Feedback. As an added twist, we’re also including prizes for the Best Mobile Bugs.

More details on this Bug Battle – as well the rules, prizes and deadlines – after this commercial break. Just kidding, you can find them in the “Bug Battle” thread in the uTest Forums and we’ll be sharing more details here in the next few days.

So testers, you’ve got a few days to prepare for our most challenging competition to date. Until then, stay tuned.

Nick Jones over at Gartner wrote a great piece about the current state of mobile apps.  It’s a must-read for mobile app developers and marketers.

His post is written about the iPhone, but it applies equally to Android, Blackberry or Symbian apps.  It also helps to explain why mobile apps have been the fastest growing segment of the uTest biz (phenomenal growth + still evolving market = a lot of testing to do).

Here are a few realizations from his recent trip to his local Apple store that seemed especially relevant for our readers (bullet points are his and the running commentary is mine) :

Read more…

Several months ago, we outlined how to build an iPhone app and presented a primer for building iPhone apps.

But as demand continues to grow for top-quality iPhone apps — 100,000 different apps have been downloaded more than 2 billion times — many companies just can’t keep up.  So in the spirit of offering a build vs. buy alternative, now we’re back to share a great article from BusinessInsider about how to hire a great iPhone app developer.

Among the tips for companies looking to find a rockstar iPhone app developer include knowing what problem you want your app to solve, what it’s going to cost you and to avoid the temptation to rush your app to market (said differently, quality matters!).  It’s a good read, and definitely worthwhile if you participate (or intend to) in the rapidly growing market for iPhone apps.  Got your mobile app(s) developed, but want to know about the best ways to test your mobile apps?

Have other advice for entrepreneurs or companies who are looking to hire mobile app developers?  Drop us a comment and drop some knowledge.

Pundits are predicting a breakout year for Android in 2010.  The upstart mobile OS (if you can call anything Google does an ‘upstart’) is earning rave reviews from mobile app developers, and is growing rapidly.  And while the Android Market and it’s 20,000 apps have a ways to go before catch up with iPhone and it’s 100,000 approved apps, the growth has been impressive.  As we’ve seen time and time again, he who wins the hearts & minds of the development community, wins the battle.

Also helping Android’s case are some great new smartphones from manufacturers like HTC, Motorola and Samsung.

Robin Waulters from TechCrunch wrote a great piece today detailing the milestone and making some predictions for 2010:

2010 is going to be a big year for the Android operating system, with many new handsets finding their way to stores around the world (including Google’s own phone) and an increasing number of developers building tools, games and the likes for the fast-growing platform.

One way of noticing that the OS is poised for a big breakthrough at the expense of Windows Mobile, Symbian and other operating systems designed to run on various mobile devices, is the number of applications already available for download in the platform’s own application store, Android Market.

Waulters cited recently published data from AndroidLib, which showed an wildly impressive growth line for the Android Market (see the chart after the jump):

Read more…

Mobile is the next great frontier for games, music, media and more.  In fact, Gartner says that  more than 139 million smartphones were sold last year.  And the phone makers aren’t slowing down:  Apple is planning to launch the iPhone into the Chinese market and a more affordable Android handset expected to hit the market by the end of ’09.  But haven’t we all learned that big markets make big targets — for VC dollars, for advertisers, for media coverage… and for hackers.

Doug Gross wrote a very interesting article over at CNN Tech about the one trend that could threaten to derail the otherwise unstoppable mobile movement.

Security analysts say they’ve already seen all of the major online threats — Trojan horses, viruses, worms — spreading on smartphones, often through e-mail attachments sent to the phones.

And as mobile apps have evolved from games and tip to calculators to company email, financial transactions and other mission-critical activities, the risk of security exploits grows considerably for mobile users and their employers.

The good news, experts say, is that phones present problems for hackers and other bad guys that traditional computers don’t.

Most viruses and other malware are designed for Windows, because that’s the most popular operating system. Since smartphones use a variety of different platforms, someone writing malicious software needs to pick and choose.

Wow, getting code to work across mobile platforms is so hard that even the hackers are having trouble!

Read more…

Last week we announced that, during the first few months of ’09, we’ve observed tremendous growth in mobile application testing within the uTest marketplace.  Since then, we’ve received a surprising number of requests for more info from mobile developers and members of the media.  It seems that testing and quality is a bit of a hot topic among those who spend their days & nights working on (or writing about) mobile.

One of the most common questions is whether our QA community has coverage for a range of mobile platforms like iPhone, Android and Blackberry.  The answer to all of the above is ‘yes’ (the answer to the multi-location, multi-language question is ‘yes’ as well).

This announcement has already resulted in several great articles from top-shelf tech sites, including eWeek and TechTarget, and got reported by mainstream media outlets such as MSNBC, MarketWatch, TMC.net and StreetInsider.

Over the weekend, as I pondered why this simple announcement resonated with so many mobile app companies and journalists, two thoughts came to mind:

1.  Shut up.  Don’t overthink this — just ride this wave!

2.  Testing for mobile apps is a widespread and serious problem with few viable solutions.  In fact, getting mobile apps properly tested is even tougher than web apps because the space is more complex and less mature.

And I guess that’s where a global community of professional testers plays nicely into the mobile app equation — because the hardest thing to build (global coverage across locations, languages, handsets, carriers, operating systems, etc.) is baked right in.  And the fact that our community can test mobile apps quickly, efficiently and on-demand doesn’t hurt either.

I’d love to hear from mobile app developers about how they test today, or from testers about the challenges that are unique to mobile. It’s a topic that’s going to be top-of-mind for the next year or more (just wait ’til mobile really clashes with the enterprise), so let’s start the discussion now.

Buggy products can be a real customer turn-off.  Witness the recent release of the BlackBerry Storm, Research In Motion’s response to Apple’s iPhone.  The Storm had tremendous promise as a great new touchscreen phone, but customer response has been limited because of early bugs in the device’s software.  Here’s a quote from one Storm customer in an article from the Wall Street Journal:

“I found myself wanting to throw it in the ocean due to my frustration with its overall usability,” said Steven Golub, a longtime Verizon customer from Morristown, N.J., who bought the Storm the day it was released, but returned it a few weeks later.

That’s pretty damning, but let’s stop and give RIM some benefit of the doubt.  Buggy software is a customer turn-off, and undoubtedly bad reviews will dampen customer enthusiasm.  But here’s a quote from the same Wall Street Journal article:

Verizon and RIM, determined to release the Storm in time for the holidays, rushed the device to market despite glitches in the stability of the phone’s operating system, according to people close to the launch.

RIM had a choice to make – release a buggy product in time for the holidays, taking bad reviews on the chin, or wait until the Storm worked better and try to gain market share during the traditionally slow Q1 made even slower by a bad economy.  We really can’t know for sure if RIM made the right decision, but there are some lessons other companies can learn here:

1.) Know Your Limitations - We all have limitations in our planning: not enough time, not enough people, or not enough testing.  Maybe your budget is limited or maybe you need to hit a holiday launch date.  Either way, it’s very important to be aware of your limitations early in the process.  If you have a tight schedule and a firm launch date, then you should make sure you have enough resources for last-minute development and testing.  If you don’t have enough people, then you should evaluate different contingencies for alternate launch dates.

2.) Work Smarter – You may feel like you have to cut corners to overcome your limitations, but the Internet can make it easy to expand your efficiency on-demand.  There are many companies that can help you add capacity to overcome bottlenecks and release a better product on time.  The uTest software testing service is perfect for helping companies improve their testing, but solutions exist for everything from coding to graphic design.

3.) Have Good Customer Data – If you’ve already passed the point of no return, then you will have to make a tough decision.  In this case, nothing helps more than really good customer data.  In RIM’s case, they had to decide between weaker customer demand because of timing or because of bugs.  Between the two, they chose to cast their lot with a buggier product over a late product.

RIM still sold 500,000 Storms over the holidays, and that’s not bad.  This is also their first touchscreen device and it holds tremendous design potential.  While Storm Version 1 had a rocky launch, Storm Version 2 may take the world by, well, storm.