Testing the #SXSW Mobile Apps (iOS vs. Android vs. Windows Phone vs. RIM)

For the second year in a row, uTest will be making an appearance at SXSW, the world-famous music/film/interactive conference in Austin, Texas. Unlike last year – where we spent most of our time eating, drinking and schmoozing with storm troopers – we have  some new, big plans in store.

The obvious difference is that we’ll be cruising around Austin in the RVIP Lounge, hitting up hotspots, giving rides, singing karaoke (poorly) and playing host to SXSW attendees throughout the week. More to come on that, but you can follow @InTheWildTest for deets on our adventures, and real-time locations if you’re at SXSW..

The other difference is that, instead of just talking about the merits of in-the-wild testing, we decided to show a real-world demonstration. So, over the last 36 hours, we assembled a select group of US-based testers to put the official SXSW mobile apps through their paces. In-the-wild testing means live testers, real devices, imperfect connectivity… basically, true real-world conditions. So we went to work testing SXSW’s official apps for iOS, Android, Windows Phone and BlackBerry. For iOS and Android, we also included tablet testing, to bring the comparison total to six.

Below are some top-level results (note that each category ranged from 1-5):


% of Total Bugs 17.7% 18.3% 18% 6.6% 23% 16.4%
Overall Score 4.1 4.0 4.6 4.7 3.8 4.2
Usability & Design 4.2 4.1 4.6 4.8 4.1 4.2
Features & Functionality 4.1 4.1 4.7 4.7 3.2 3.9
Application & Performance 3.3 3.2 3.9 4.0 3.5 3.7


Of course, these figures only tell part of the story. As the apps were tested in terms of functionality, performance, design, connectivity and other factors, several issues popped up on more than one occasion. Here were a few areas where some notable bugs were uncovered:

  • Incorrect time displays
  • Sync issues with registration and deleted items
  • Crashes on various tablet OS versions
  • Issues with installation
  • Social media integration
  • Issues with rating and uploading photos

It should be noted that despite these issues, the overall reaction from our community was positive for each of these applications. In fact, the overall ratings you see above are substantially higher than the industry norms, so kudos to the respective dev teams.

Anyway, if you’re at SXSW and want to learn more about In-The-Wild Testing, be sure to stop by the RVIP Lounge. If you’re not able to attend, then head on over to inthewildtesting.com.

Essential Guide to Mobile App Testing

Announcing uTest Express: Mobile App Testing for Early-Stage Startups

With the vast majority of mobile apps being developed by startups, wouldn’t it be great if there was an easier, more affordable option for young companies to get professional testing under real-world conditions? We thought so, which is why we had the testing gnomes in our research lab cook up uTest Express.

Designed for startups, Express helps young companies validate the quality & performance of their mobile apps, as well as speed up their app store approval process. Think of Express as a time machine for mobile app developers: they can transport a month or two into the future to see what type of ratings their apps will likely receive and if there are any major defects — before they launch.

Whether you’ve built an app for iPhone, iPad, Android, Blackberry, WinMo or Symbian, Express makes it easy to get real-world testing and expert feedback that meets your needs and budget. Oh, did we forget to mention that plans start at only $499? That’s half of what you probably spent on Red Bull and Starbucks while you were building your location-based, freemium, socially-linked, caffeine-free, voice-powered, 3D, virtual reality, highly-addictive, semantic-searching, gaming-layered, hybrid-powered, native app, right?.

For each project, uTest handpicks members of its tester community from North America who best match the testing requirements and have the right mobile devices and operating systems. The customer’s mobile application is then tested professionally on real devices across real carriers, providing real-world testing results and expert feedback that aren’t possible with emulators, simulators or remote access.

At the conclusion of each project, customers receive a list of well-documented bugs, including screenshots and videos with steps to reproduce them. They also receive expert feedback from the testers about the application – including app ratings and feedback for interface design, usability, app performance and more. To learn more about how it works,  watch this brief Product Tour.

Here are a few key features of the service:

  • Cost-Effective: With three testing packages available, uTest Express offers mobile app testing to fit any startup budget;
  • Easy-to-Use: A simple interface walks customers through a series of questions to identify their testing needs and facilitate the creation of the testing project;
  • Professional Testers: Testers are selected from our community of 35,000+ professional testers;
  • Real Devices and Carriers: Testing on real devices and real carriers – no emulators, simulators or remote access
  • All Major Mobile Platforms Available: uTest Express is available for all mobile operating systems including iOS, Android, Blackberry, Windows Mobile and Symbian;
  • Accessible 24/7: Customers can access their testing projects on-the-go, using uTest’s downloadable app for the iPhone and iPad, or through the traditional browser-based interface.

The launch coincides nicely with our trip to CTIA Wireless – one of the largest mobile technology events in the world – so we’ll be spending the next few days talking to some of the most promising names in the business about this unique new service.

In the meantime, if you’re early-stage startup looking for an easy, affordable way of testing your latest mobile app (or your latest version), look no further than uTest Express.

Essential Guide to Mobile App Testing

BlackBerry Ups The Ante To Woo Developers

All you mobile app developers:  You may not get roses sent to your office, or chocolates on Valentine’s Day, but let there be no mistake.  As you sit behind your dual monitors designing slick interfaces and cracking out code, you’re being courted by the App Stores.  Every type of company with a storefront– the platforms, device manufacturers, carriers, independent app ‘superstores’— knows that you have the power to make or break their commercial success based on which ones you decide to develop for.

But you know this.  And you also know that some app stores are a whole lot better at courting talent than others.  The good ones know that you don’t want roses or chocolate (well, occasionally wouldn’t hurt, right?).  What you really want is a faster, more streamlined approval process and a pricing structure that makes it worth your while.  You’re under a ton of pressure to speed your product to market, and probably feel the heat most when you’re doing the testing necessary to make sure you get those 5-star reviews out the gate…versus the dreaded 2-stars.  [cue the scary music]

That’s why RIM gets kudos this week for listening to the developer community and making a series of announcements on their dev blog that will simplify the Blackberry app store certification process.  Among them, they removed the requirement that developers must submit notarized proof of identification to distribute their apps on their storefront. They also posted some educational information to make it easier for developers to build apps for the BlackBerry Playbook.

Considering recent research, including a survey released in late 2010 by Appcelerator, that reports interest is waning for building BlackBerry apps, RIM is stepping up its game to make sure Android and iPhone don’t hog developers’ dance cards.  Time will tell…

What do you think?  Feeling confident that BlackBerry’s dev relations are headed in the right direction?

Essential Guide to Mobile App Testing

Symbian And BlackBerry Hangin’ Tough as Android Takes Over

According to Mashable and research firm Canalys, Android overtook Symbian to become the world’s most popular smartphone platform in Q4 2010.

Out of the 101.2 million Q4 worldwide shipments of smartphones, Android claimed 33.3 million compared to Symbian’s 31 million. Apple’s iOS took the bronze with 16.2 million smartphone shipments, followed by RIM with 14.6 million, and Microsoft rounds out the list with 3.1 million devices shipped.

There is no denying that, just as Apple’s iOS revolutionized the smartphone category, Android has come on the scene and made major waves as well. Take a look at the graphs and the article here, both courtesy of Nielsen, to see this impact in vivid color.

How will the recently launched Verizon iPhone impact smartphone market shares? It’s not clear right now, but if pre-order reports are any indication, it could be quite significant. And what about the impact of Windows Phone 7 OS? Still too early to tell.

What is clear, is that some of the one-time industry leaders are being challenged and even surpassed. Unless they respond with new and better versions, they’ll be left on the sidelines to watch as the new kids on the block (no, not the real NKOTB) take over. All we can say for those who are looking up at the leaders is to keep “hangin’ tough.”

Essential Guide to Mobile App Testing

Is 2011 The Year Of The Mobile Enterprise App?

We write a lot about the latest and greatest apps on smartphones, tablets and other devices. But what about enterprise apps? We haven’t taken a look inside the enterprise for signs of a mobile app uptick (support or implementation) in quite sometime. That’s why this study conducted by Kelton Research really struck a chord.

Yesterday, Kelton Research announced the results of their study on mobile enterprise apps, and I think they may shock you. According to Kelton, a whopping 90% of U.S. and UK IT managers surveyed will implement new mobile apps in 2011! 250 IT managers in the U.S. and UK at companies with revenue of $100MM+ were polled.

At first, I thought these stats seemed slightly inflated, but the more research I did, the more surveys I found that indicated that 2011 is indeed the year of the mobile enterprise app. In fact, ABI Research expects worldwide enterprise mobile data revenues to reach $133 billion by 2014.

With new apps for BlackBerry, iPhone, iPad, and Android making deep in-roads in healthcare, finance, education, media, and retail, employees are using mobile apps to access everything from CRM systems, to financial results, to marketing campaigns, to tracking orders, etc.

Another very interesting finding in the Kelton study… Saving money is the most popular reason (63%) for deploying mobile apps (chart above). I agree with Eric Lai of ZDNet who said, “It’s heartening, as it shows that mobilizing appears to be a fiscally sound strategy, not something with a pie-in-the-sky ROI.”

Some other interesting stats from the study include:
Continue Reading

Essential Guide to Mobile App Testing