Software Testing Blog

Between our office March Madness bracketology and the Bug Battle of the TV Networks rocking the headlines, it’s been an exciting week here at uTest! The Q1 Bug Battle peaked the interest of several leading pundits and bloggers, including TechCrunch’s Robin Wauters (@robinwauters), VentureBeat’s Robert Mullins (@SiliconMoon) and BostInnovation’s Kyle Psaty (@bostinnovation):

• uTest Finds 908 Bugs In Web Apps Of Major U.S. TV Networks

• uTest Rates Best TV Network Web Sites, And Catches Bugs

• uTest Finds 1,000 Bugs in Web Properties of TV Networks

• “Battle of the TV Networks” Uncovers Nearly 1,000 Bugs

Additional press coverage includes Radio Business Report, NewTeeVee, Broadcast Newsroom and VentureFizz.

Thanks again to all of our cunning uTesters for exploring these web and mobile apps voraciously and hunting down some very crafty bugs. And congratulations to all the winners of the Bug Battle of the TV Networks!

Until the next Bug Battle, happy bug hunting!

Over in Barcelona at the Mobile World Congress, 24 of the world’s leading wireless carriers and mobile OEMs announced their plans to create the Wholesale Applications Community (WAC) — a unified platform which developers can use to build a mobile app once and have it run seamlessly on any handset, OS or carrier.  Among the impressive roster of backers are mobile heavyweights like AT&T, Verizon, Orange, LG and Sony.  Sounds like a utopia for mobile developers, right?  It could be… if it works.

There are more than a few skeptics, including Jason Kincaid (@jasonkincaid) over at TechCrunch.  As Kincaid states (with a bit of help from Google’s Andy Rubin):

If it sounds too good to be true, that’s because it probably is. Andy Rubin, Google VP of Engineering (and the man in charge of Android) has already shared his skepticism, saying, “There is always a dream that you could write [a program] once and [have it] run anywhere and history has proven that that dream has not been fully realised and I am sceptical that it ever will be“. To put it another way, this is a pipe dream from carriers looking to loosen Apple’s stranglehold over mobile applications and there’s very little chance that it’s going to work.

The reasons Kincaid thinks the WAC won’t work out include:

Read more…

Mobiles’ 2009’s impressive growth storyline continues in the early days of 2010. By now, you’re probably familiar with the impressive growth in the number of mobile app:  100,000+ approved apps in Apple’s store, nearly 20,000 Android apps, and yesterday, InformationWeek reported that Palm’s app catalog has surpassed the 1,000 mark.

And while it’s widely accepted more apps = more users = more impressions = more revenues, how about some updated financial figures that are closer to the top line?  Well, Jason Kincaid over at TechCrunch outlines a recent report from AdMob that shows that the number of mobile ad impressions doubled in just TWO months:

Some more stats from AdMob’s latest post:

• Increased device diversity: In December, 7 devices generated more than 3% of requests each: the Motorola Droid, HTC Dream, HTC Magic, HTC Hero, Motorola CLIQ, HTC Droid Eris, and the Samsung Moment. This is up from only 3 devices in October (HTC Dream, HTC Magic, and HTC Hero).
• Droid Invasion: The Motorola Droid is already the leading Android handset in the AdMob network and generated 30% of requests in December.
• US leads Android adoption: 90% of Android traffic was in the US in December, up from 84% in October. Top countries by requests are the US, UK, Germany, France, and Canada, respectively.

It’s an understatement to say that mobile was hot in 2009 — from our vantage point, mobile app testing was the fastest growing segment of uTest’s business.  But a growing number of trends are pointing to 2010 growing at an even faster rate.

Is it possible that this year will make ‘09 look like the quaint, simple, slow times in the world of mobile apps?  Based upon what our customers (hundreds of app developers of all shapes and sizes) are telling us, the answer is a definitive ‘yes’. What say you?

UPDATE 1/5/10: And if all the above stats didn’t convince you, Apple just hit the 3 billion download mark! In just over 3 months, Apple went from a record 2 to 3 billion downloads by iPhone and iPod Touch users. Simply incredible.

Pundits are predicting a breakout year for Android in 2010.  The upstart mobile OS (if you can call anything Google does an ‘upstart’) is earning rave reviews from mobile app developers, and is growing rapidly.  And while the Android Market and it’s 20,000 apps have a ways to go before catch up with iPhone and it’s 100,000 approved apps, the growth has been impressive.  As we’ve seen time and time again, he who wins the hearts & minds of the development community, wins the battle.

Also helping Android’s case are some great new smartphones from manufacturers like HTC, Motorola and Samsung.

Robin Waulters from TechCrunch wrote a great piece today detailing the milestone and making some predictions for 2010:

2010 is going to be a big year for the Android operating system, with many new handsets finding their way to stores around the world (including Google’s own phone) and an increasing number of developers building tools, games and the likes for the fast-growing platform.

One way of noticing that the OS is poised for a big breakthrough at the expense of Windows Mobile, Symbian and other operating systems designed to run on various mobile devices, is the number of applications already available for download in the platform’s own application store, Android Market.

Waulters cited recently published data from AndroidLib, which showed an wildly impressive growth line for the Android Market (see the chart after the jump):

Read more…

Catching up on my reading and came across an interesting article on the App Store from Jason Kincaid over at TechCrunch.  Ostensibly, this piece is about Apple’s overly strict and seemingly random App Store approval process and a high-pressure NSFW rant from a high-profile developer, Joe Stump, whose popular app has a new version that is on the outside of the app store, looking in.

Stump outlines a problem that he had with Chess Wars, the Facebook Connect-enabled chess game that came out in July. After catching a show-stopping bug soon after the initial release, his company Blunder Move promptly issued an update. Soon thereafter they noticed another bug, which they quickly released a fix for. Unfortunately, this second update has sat in App Store purgatory for many weeks now, and Apple has gone silent on when it will be approved.

What struck me — other than the string of bad press that Apple has earned via the lousy treatment of its developers — is that THIS is a new cost of bugs.

Not to lay blame at Stemp’s feet.  We, of all people, know that bugs happen.  Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many.  But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version.

Chalk it up as another real-world reason to achieve maximum professional testing coverage to eradicate all quality, security and privacy defects before your app goes live.  Have other stories of software bugs causing havoc in interesting, scary or funny ways?  Drop us a comment and tell us about it.

Every startup team has great and spirited debates about its products (or services).  We debate what works, what doesn’t, what makes it unique, and most importantly, what users want. We draw inspiration when new products launch and change the world.  Think Salesforce.com back in the day, the family of iPhones or, more recently, Facebook and Twitter.

And similarly, we witness product missteps that make us wince, rant or just shake our heads.   One such case emerged today with DiggBar (Digg’s URL shortening service) under the bright lights of a TechCrunch article titled, DiggBar Commits Career Suicide, Starts Redirecting To Digg Homepage.  In the words of  TC’s Jason Kincaid:

Read more…

Big buzz this week around the halls of uTest (ok, we don’t actually have halls at uTest HQ, but it does make for a pretty word picture).

Don’t get me wrong, we’ve gotten more media coverage in the last few months than most start-ups receive in their entire lifetime.  In the last month alone, uTest has been written up by such tech media stalwarts as TechCrunch, eWeek, InfoWorld, StickyMinds and TechTarget.  It’s safe to say the uTest secret is out.

And yesterday, we made our Forbes.com debut in an insightful article about starting a business during a recession.  Our fearless leader, CEO Doron Reuveni, was interviewed by author, Melanie Lindner for her piece about 10 start-ups success stories. It’s a great write-up that offers some sources of hope and practical tips for you current and future entrepreneurs.

As is the case with all of our wins — whether they’re with media, investors, awards or new customers — we know that it’s a direct reflection on the testers and customers who have helped uTest become what it is today.  And we thank you sincerely.

We’re always on the lookout for ways we can improve.  Have big ideas, small suggestions or general feedback?  Drop us a comment or shoot me an email.  We want to hear from you.

Apologies for the alliteration, but things have been ultra-busy around the halls of uTest this week.  The reason our phones have been ringing off the hook is because we announced the results of our 1st Quarter Bug Battle on Tuesday.

In case you missed it, we’ve received some fantastic media coverage in the past few days.  Who knew the world would be so interested in the results of 1,000+ QA professionals from 64 countries simultaneously testing the world’s three most popular networking sites?  Ok, well we had a pretty good idea they’d care, but we’re glad it captured their attention.  Here are a few of the noteworthy articles:

- TechCrunch: uTest Bug Battle: Which Social Network Is The Buggiest?
- Dr. Dobbs: Bugs In Social Networking Software? You’re Kidding, Right?
- eWeek:  Facebook Triumphs LinkedIn, MySpace in uTest’s Bug Battle

Our community’s ability to mobilize quickly and provide real-world testing coverage is making a strong impression on a growing number of companies.  These firms are looking for ways to do more with less — and the uTest community is primed to help them meet their QA needs.  Keep up the good work!

If you or someone you know is struggling to develop iPhone apps, this recent item from TechCrunch may provide relief:

At the recent Future of Web Apps conference in Miami, Y-Combinator-backed 280 North announced Atlas, a drag-and-drop visual editor for building desktop web applications with Cappuccino, 280 North’s Javascript-based framework…Atlas can wrap up iPhone Web Apps like native applications, granting them access to a significant portion of the iPhone API and allowing them to be sold through the App Store.

This lowers the barrier of entry for iPhone development substantially, allowing those with Javascript knowledge to create fully functional applications on the platform without requiring them to learn a whole new language. The same limitations that apply to Javascript apply here, presumably – in other words, don’t expect to be throwing down ultra-rich OpenGL-based 3D games, but mid-range apps (such as Twitter clients, RSS readers, etc.) should be completely doable.

I’ve gathered the significance of this from a developer’s perspective, but I’m curious as to what the testing community thinks, and three questions immediately come to mind:

1. Will this complicate the QA process for these apps, make it easier, or have no affect at all?

2. Besides Twitter clients and RSS readers, what other types of mid-range apps should I expect to see?

3. Will the lower barrier to entry result in lower quality apps, or just the opposite?

Leave a comment and let us know what you think.

For all of you mobile app developers who dream of creating the next hot iPhone, Blackberry or G1 application, a key question to ponder is this:  once you’ve conceived, developed, tested and launched your killer app, how long will it remain killer?

Well, TechCrunch highlights the answer from a recent Pinch Media presentation, and it’s not the cheeriest news for mobile app developers.  It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it.  And by day 30, less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.  Grim.

The moral of the TechCrunch story is this:

It answers the eternal question that all iPhone developers have: Should my app be free or should I charge for it? For all but the most successful apps, the free route does not make much sense because there is not enough time to recoup the costs of developing the app from advertising.

Free apps tend to be run 6.6 times more often than paid apps, but even with that increased usage, it is not enough to make more money.

Yardley estimates that less than 5 percent of all apps would make more money right now with advertising than charging for paid downloads. His advice: “Unless there is something inherent about the app that screams free, sell it.”

I completely agree with Yardley’s math and logic, but I think there’s another important lesson to be learned here.  He addresses the issue of maximizing revenue, but ignores cost containment as a means to profitability.  Said differently, another powerful driver of profitablity is the cost and speed in bringing your mobile application to market.

By finding ways to develop, test and launch mobile apps more quickly and for less money, developers extend the money-making window, enable themselves to launch more apps per year, and decrease their break-even levels.

What do you think — what’s the secret to profitability for the creators of iPhone, Android and Blackberry apps?  Drop a comment and drop some knowledge on us.