Software Testing Blog

Catching up on my reading and came across an interesting article on the App Store from Jason Kincaid over at TechCrunch.  Ostensibly, this piece is about Apple’s overly strict and seemingly random App Store approval process and a high-pressure NSFW rant from a high-profile developer, Joe Stump, whose popular app has a new version that is on the outside of the app store, looking in.

Stump outlines a problem that he had with Chess Wars, the Facebook Connect-enabled chess game that came out in July. After catching a show-stopping bug soon after the initial release, his company Blunder Move promptly issued an update. Soon thereafter they noticed another bug, which they quickly released a fix for. Unfortunately, this second update has sat in App Store purgatory for many weeks now, and Apple has gone silent on when it will be approved.

What struck me — other than the string of bad press that Apple has earned via the lousy treatment of its developers — is that THIS is a new cost of bugs.

Not to lay blame at Stemp’s feet.  We, of all people, know that bugs happen.  Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many.  But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version.

Chalk it up as another real-world reason to achieve maximum professional testing coverage to eradicate all quality, security and privacy defects before your app goes live.  Have other stories of software bugs causing havoc in interesting, scary or funny ways?  Drop us a comment and tell us about it.

Every startup team has great and spirited debates about its products (or services).  We debate what works, what doesn’t, what makes it unique, and most importantly, what users want. We draw inspiration when new products launch and change the world.  Think Salesforce.com back in the day, the family of iPhones or, more recently, Facebook and Twitter.

And similarly, we witness product missteps that make us wince, rant or just shake our heads.   One such case emerged today with DiggBar (Digg’s URL shortening service) under the bright lights of a TechCrunch article titled, DiggBar Commits Career Suicide, Starts Redirecting To Digg Homepage.  In the words of  TC’s Jason Kincaid:

Read more…

Beta software tends to frighten Big Business. They’re skeptical of the phrase “almost done,” and for good reason. They demand a finished product. There are very few exceptions to this.

Take the recent news from Google as an example. A few days ago, the company finally removed the “beta” tag from Gmail and several other apps. Most people didn’t realize Gmail was still in beta (or what beta meant, for that matter) and most people didn’t care. So why bother with the change? This article from The New York Times explains why:

Practically speaking, the change will mean precious little to Gmail’s millions of users. But it could help Google’s efforts to get the paid version of its package of applications, which includes Gmail, Calendar, Docs and other products, adopted inside big companies. Corporate technology managers tend to shy away from beta products, and Google wants to remove any barriers to adoption that it can.

And there you have it: Although the meaning of beta has changed slightly, the perception of it hasn’t. Even though Gmail was essentially a completed product, with tons of great features and no major usability/functional issues, the beta perception was preventing them selling it to the larger companies.  It might not be fair, but it’s true.

Read more…

As consumers, we’re loyal to our brands.  Whether it’s Mac vs. PC, Nike vs. Adidas, or Starbucks vs. Dunkin Donuts, we like what we like.  And it’s tough to convince us that some other product/service/website is as good as our favorite.  Companies go to great lengths and expense to try and alter our preferences (who could forget the Pepsi Challenge?), but all too often, we walk away feeling marketed to (read as: manipulated).

Last week, Microsoft launched it’s new search decision engine – Bing – amid great fanfare.  And one enterprising MSFT employee named Michael Kordahi launched an app that he calls BlindSearch, which offers a “taste test” of the top three search engines.  Simply enter in a search term and BlindSearch will pull the results from Google, Bing and Yahoo into one interface… then you can vote for your favorite based only on the quality of the search results.

As Michael notes, this isn’t a perfect test… after all, we choose our search engine based on more than just the results (speed, overall user experience, number & placement of ads, other filters and features).  But when we search, we’re generally looking for the most accurately matched results.  While there are a lot of ways to perform user preference and usability testing — surveys, focus groups, A/B analytics — this is an interesting and entertaining experiment and a cool way to test your own preferences.

Take the BlindSearch Test and then come back and tell us which search engine you actually prefer!

What do Flickr, LinkedIn, Pandora and uTest all have in common?  Before they hit it big, they were invited to share their ideas at Dealmaker Media’s Under The Radar conference – the premier event for up-and-coming startups.

As part of the “Applications Track” segment, Doron Reuveni, our fearless leader, will be discussing the future of crowdsourced software testing (uTest of course) this Friday in Mountain View, California. Out of hundreds of applicants, uTest will be one of twenty-four companies presenting their ideas for a select audience of entrepreneurs, investors and technology journalists.

Now in its eleventh year, Under The Radar has catapulted many of its past attendees (well, not literally) to much greater successes.  In fact, over the past three years, companies selected to present at the conference have gone on to raise more than $1.36 billion in funding, or an average of $14 million per company.

And so although we view our community as the future of software testing, we’re more than happy to be under the radar (for now anyway).

From my fortune cookie last night:

“From error to error, one discovers the entire truth (in software).”

I added in the last part, of course, but you get the idea.

In the past few months, we’ve been providing QA services on a bunch of mobile apps.  As a result, we’ve had the good fortune of meeting some really cool, creative entrepreneurs and developers who are building top-shelf apps for the iPhone, Blackberry, G1 and others.

So I’ve been keeping my eyes peeled for articles from leading mobile app entrepreneurs.  And while perusing Alley Insider, I came across an article from Dan Frommer.  Frommer highlights a fantastic first-hand account from mobile app veteran Craig Hockenberry, maker of iPhone hits such as Twitterrific and Frenzic.

Hockenberry’s 2,000 word piece is a must-read for aspiring iPhone app developers.  Frommer was kind enough to summarize it as follows:

- Learn how to develop Web pages for Mobile Safari so you know how to think about designing apps for a mobile device.
- Buy a Mac. You can’t make iPhone apps from a PC.
- Sign up for Apple’s developer program.
- Watch the “getting started” videos in the iPhone dev center.
- Goof around with some of the sample code on Apple’s site.
- Check out a few of the better iPhone coding books.
- Read up on a few of Craig’s old posts.
- Go for it!

In particular, Hockenberry mentions a few books iPhone app dev books:

If you’re just starting out, I’d highly recommend Beginning iPhone Development: Exploring the iPhone SDK by Dave Mark and Jeff LaMarche. The best thing about this book is the step-by-step approach it takes to working with Xcode, Objective-C and the iPhone APIs. They’ll lead you through the basics and you’ll be building your own apps in no time at all.

As you get more comfortable with the tools and AppKit/UIKit frameworks, I’d recommend you take a look at Erica Sadun’s iPhone Developer’s Cookbook: Building Applications with the iPhone SDK. This book presumes a bit more knowledge about the SDK, but is a very handy reference both to the official and unofficial APIs.

Since you’re going to be working with Cocoa Touch on the iPhone, you’ll also want to start thinking like a Cocoa programmer. Every great iPhone and Mac developer has nothing but wonderful things to say about Cocoa Programming for Mac OS X by Aaron Hillegass.

If you have previous development experience with C, C++ or Java, you’ll want to read this mailing list post by Erik Buck that enumerates some of the difficulties that you’ll have coming up to speed with Objective-C and Cocoa.

As we come across more of these practical, hands-on resources, we’ll share them here.  If you know of other helpful articles or posts, please send them our way.  For you future mobile mavens, I hope some of these resources are helpful.

Interested in building an iPhone App?  If so, there’s a must-read article over at Mashable.  In their words:

Ten23 Software has created an entire 37 page guide to the development process (below), decisions they made and what they learned during the creation of their PhotoKast app. Their hope is that the document might provide insights for other developers when they start out on iPhone App development projects.

Have experience building mobile apps for the iPhone, Blackberry, Android or other mobile platforms?  If so, drop us a note and share your experience and tips.

For all of you mobile app developers who dream of creating the next hot iPhone, Blackberry or G1 application, a key question to ponder is this:  once you’ve conceived, developed, tested and launched your killer app, how long will it remain killer?

Well, TechCrunch highlights the answer from a recent Pinch Media presentation, and it’s not the cheeriest news for mobile app developers.  It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it.  And by day 30, less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.  Grim.

The moral of the TechCrunch story is this:

It answers the eternal question that all iPhone developers have: Should my app be free or should I charge for it? For all but the most successful apps, the free route does not make much sense because there is not enough time to recoup the costs of developing the app from advertising.

Free apps tend to be run 6.6 times more often than paid apps, but even with that increased usage, it is not enough to make more money.

Yardley estimates that less than 5 percent of all apps would make more money right now with advertising than charging for paid downloads. His advice: “Unless there is something inherent about the app that screams free, sell it.”

I completely agree with Yardley’s math and logic, but I think there’s another important lesson to be learned here.  He addresses the issue of maximizing revenue, but ignores cost containment as a means to profitability.  Said differently, another powerful driver of profitablity is the cost and speed in bringing your mobile application to market.

By finding ways to develop, test and launch mobile apps more quickly and for less money, developers extend the money-making window, enable themselves to launch more apps per year, and decrease their break-even levels.

What do you think — what’s the secret to profitability for the creators of iPhone, Android and Blackberry apps?  Drop a comment and drop some knowledge on us.

Start-ups and entrepreneurs have all seen the news… heard the stories… felt the pinch.  As the financial markets have shuddered and lurched, VCs have tightened their purse strings.

This climate was first given voice when Sequoia’s “R.I.P. Good Times” presentation spread through office parks in the Bay Area, Silicon Alley, Boston and everywhere in between.  The presentation (worth a read, if you haven’t yet seen it) paints a bleak short- and long-term economic picture.

More recently, TechCrunch shares that the National Venture Capitalist Assocation and PriceWaterhouseCoopers reported just how much the VC market tightened in Q4.  The total amount of VC financing in the 4th quarter was down 33 percent year-over-year, and down 26 percent quarter-over-quarter.  For the entire 2008 year, the total amount VCs invested into startups was $28.3 billion, down 8 percent.

VentureBeat recently highlighted a survey of 400+ VCs that reaffirmed just how slow 2009 will be for VCs — and how tough it may be for start-ups to raise fresh capital.  The good news?  These same VCs believe that markets will loosen in 2010, and anticipate they will actually fund the same number of deals in ’09 as they did in ’08 (albeit at a smaller average deal size).

A majority of the VCs said they expect to fund the same number of companies or more, so the bigger effect may not be on the number of fundings, but on the amount each startup can raise.

This means that some start-ups will find funding, although the average deal size and valuation will almost certainly be lower.  In these difficult times, it’s more critical for entrepreneurs to control their costs — particulary their fixed cost structure — and still find ways to innovate.  Getting new products and versions to market more quickly, and outrunning the competition (who have their own concerns and challenges, by the way) will be more important than ever.

How is your firm striking the balance between cost containment and getting groundbreaking apps to market quickly?  Drop us a comment and share your thoughts.