You may be familiar with our Testing the Limits series where we interview thought leaders in software development and testing. These experts answer questions, give advice and share nuggets of wisdom that will help readers succeed in their careers.

How do you know they’re not just full of hot air? Well, for one, they probably wouldn’t be well respected members of the QA community if they were just lying all the time. But also, real life backs them up.

In this series, we’ll take excerpts from Testing the Limits interviews and pair them up with real life situations that prove these experts know what they’re talking about. There’s our first expert excerpt:

“Real testing, to me, should be based on investigating how the software allows people to deal with what we call ‘exceptions’ or ‘corner cases.’ That’s what we call them, but if we bothered to look, we’d find out that they were a lot more common than we realize; routine, even.” – Michael Bolton

This is what in-the-wild testing is all about. Ultimately, real life people will be using your software in real life situations. Sometimes, those real life situations are scenarios you couldn’t test for in the lab, didn’t occur to you or that you didn’t even dream of. But odds are your users will find them. While a crazy fringe use case will pop up from time to time no matter what you do, it’s important to take a step back, think through your testing and make sure you’ve come up with as many use cases as possible.

One of the best, most obvious and repetitive examples of “corner cases not really being that uncommon” can be found in video game development. Gamers are dedicated people and will try to make your software do things you wouldn’t believe. The biggest problems occur when they’re trying to do something reasonably within scope – uncommon maybe, but not unusual or unheard of. For instance, when Mass Effect 3 came out early last year the game encountered a bug that wouldn’t let players import customized avatars from earlier editions. If players set the appearance of their Commander Shepard avatar in the original Mass Effect and didn’t tweak it at all in Mass Effect 2, the character would revert to factory appearance when imported into game 3. From CNet:

Read more …

If you had to point to one weak spot in the Android ecosystem, it would have to be the pace in which new versions of the operating system are deployed.

Last week, Google updated their Platform Versions page – and it shows the challenge for Android developers.  While 29% of the installed based are now on Ice Cream Sandwich (ICS), the majority of users, at approximately 45%, are stuck on Gingerbread.  Jelly Bean is even further down the list with only 13.6% of the installed based. The Next Web puts it into perspective:

Last month, Android hit a bunch of new milestones: Jelly Bean finally passed 10 percent share of the droid’s pie, Gingerbread fell below the 50 percent mark, and Froyo dipped below 10 percent. Compared to then, Android 4.2 has budged a minor 0.2 percentage points (from 1.2 percent to 1.4 percent) and Android 4.1 has gained a solid 3.2 percentage points (from 5.9 percent to 9.0 percent).

Android 4.0 has stagnated, dropping 0.1 percentage points to 29.0 percent, suggesting it may not ever cross the 30 percent mark. Android 2.3 meanwhile is down 2.0 percentage points (from 47.6 percent to 45.6 percent) and Android 2.2 fell 0.9 percentage points (from 9.0 percent to 8.1 percent). The bigger picture is still the same though: Gingerbread (released December 2010) is first, ICS (October 2011) is second, the latest and greatest Jelly Beans (June 2012 and November 2012) are third, and Froyo (May 2010) is fourth.

If ICS has indeed peaked, and we should know this for sure in the next month or two, Jelly Bean will rather quickly move up into second place. We would estimate that this will happen before the end of 2013, though Gingerbread will probably remain in first throughout.

Contrast that with adoption of iOS6 – an upgrade that actually had a slower adoption curve due to the issues with Apple Maps that made people hold off on upgrading (heck, I still haven’t upgraded my two iPads). Even with the Apple Maps issue, iOS6 is now on close to 60% of all devices.

Of course, it isn’t the consumer’s fault that they are stuck on Gingerbread – and it’s not Google’s fault either. There are multiple reasons why they haven’t got the latest upgrade, including phone capability and carrier policies/approach. But none of that makes it any easier for the Android developer who wants to maximize adoption of their app, and it makes the development and testing of that app that much harder, but far from impossible.

If you pay attention to what users want and focus on making simple functions work well you’ll win yourself some fans. While that doesn’t sound like rocket science, Sarah Lacy over at PandoDaily pointed out that it’s not an approach many companies (especially non-tech companies) take. Sarah wrote last month about being a big fan of the Chase mobile banking app. Once she realized that she had stopped complaining about a previously atrocious online and mobile experience, Sarah did what any journalist would do and dug into the issue (journalists are curious people). She found that Chase’s success can be traced back to that simple concept of doing things right from the start. Here’s a bit of what she found (from her PandoDaily article):

I … found an interesting interview with Chase’s senior vice president in charge of these efforts, Ravi Acharya. Yeah, not exactly a household name in the tech world.

But his sentiments had some wisdom that all older or non-tech companies looking to build their first mobile apps should consider. He talks about how functionality really lead the approach in mobile banking. Back in 2006, it debated whether it should do an app or a mobile Web site, and it decided to scale things back further to simple yet functional two-way, text-banking. You could get a note if you were in danger of over drafting and message back if you wanted to move funds. That’s nothing that will win awards, but starts a mobile relationship with customers that’s actually useful. The bank grew its functionality as mobile audience and utility grew and devices became more sophisticated.

The approach was in Acharya’s words, “Let’s do things in a simple fashion so that lots of our customers can use it.” It was one of the first to offer remote deposits, saying, simply it worked because it “solved basic problems.”

While I have no idea if Chase tested its mobile offerings with real users in-the-wild, Acharya’s approach to creating mobile apps and services embodies the spirit of in-the-wild testing. In that interview Sarah came across, Acharya spells it out perfectly:

We continue to look at how we can leverage various aspects of the mobile device, while at the same time trying to keep it a really simple experience for customers. You don’t want to complicate things too much. Sometimes people rush off and implement technology for technology’s sake. Those things usually don’t end up working too well, and we try to stay away from that.

Listen to your users. It’s that simple. Your app (or product or software or website) won’t succeed if you’re not giving your target audience what they want and need. To prove yet again that Acharya and Chase get this concept they changed their mobile apps when optimzing for tablets – tailoring what the app offers and does based on how customers use tablets.

Read more …

Over the past several years, the web development community has been enthralled with Ruby on Rails. The combination of the Ruby language with the Rails framework has proven extremely powerful, and many of the web’s top sites are built using the two technologies. For example, sites like Twitter, 500px, Groupon and more were all built with Ruby on Rails as their framework. Both new and veteran developers have adopted the platform because of its ease of use, rich library of components, and outstanding tools.

Late last month, the gleam of Ruby on Rails dulled considerably as a new class of security attacks emerged targeting the framework. Like many security vulnerabilities, the attacks started out as academic exercises which were quickly spun into automated attack bots designed to knock over Rails servers en masse.

Today, anyone who runs a Ruby on Rails server who hasn’t applied an update is probably already compromised. Think that’s overstating things a bit? Patrick McKenzie sounds the alarm loudly in his blog post titled What The Rails Security Issue Means For Your Startup:

It is imperative that you understand that all Rails applications will eventually be targeted by this and similar attacks, and any vulnerable applications will be owned, regardless of absence of these risk factors.

Still think that’s overstating things? Read more …

Nope, you read that headline correctly. Earlier this week, Forbes named uTest the 8th Most Promising Company in America. No sense burying the lede on a story this big.

Over the years, uTest has been fortunate to pick up several impressive honors; some for our innovation, some for our products and even some for our workplace culture. But this award – which puts uTest in a class with the likes of Box, Evernote, SecondMarket, ShareThis, uSamp, Marketo and HubSpot- is truly remarkable.

What makes this award even more special for us was how non-subjective the selection process was. For those who are unfamiliar with how this list is created, Forbes ranks the top 100 most promising companies based on several key factors, including:

• Company growth (both in sales and hiring)
• Quality of management team and investors
• Margins
• Market size
• Key partnerships

The top 50 spots on the list appear in the February 11 issue of Forbes. In the meantime, you can check out the complete list on Forbes.com.

As startup junkies, we’ve put our lives into building uTest. And much of that time, it’s been about keeping our heads down to focus on execution and optimization. But sometimes, you need to pause, lift your head up, and take stock of how you’re doing compared to other hard-charging startups, including those in industries other than your own. This is one of those times.

Yeah, that little startup in the rough-and-tumble, utterly unsexy world of software testing…that company you entrusted with your web & mobile app testing…that test community that you joined out of curiosity…that crazy crew that used to run quarterly Bug Battles just for fun…yep, those guys have grown into Forbes’ 8^th most promising company in the US.

The best is yet to come. Stay tuned.

Good news for the customization-loving jailbreakers out there; the long, nearly five month wait is over. A jailbreak for iOS 6 is now available, and works on the iPhone 5, iPad 3, iPad mini and most current iOS devices.  The jailbreak allows users to personalize their device quickly, eliminating Apple’s restrictions.

The jailbreak, known as “evasi0n”, was developed by the underground team of hackers called evad3rs. According to Andy Greenberg of Forbes, evasi0n hit peak traffic almost immediately:

“In its first six hours online, the crack had already been used at least 800,000 times, according to Jay Freeman, administrator of the Cydia appstore for jailbreakers, and he says that’s a conservative estimate–his count was fouled up when the tsunami of traffic knocked his server offline several times over the course of the jailbreak’s first day online. By Tuesday his count was up to 1.7 million.”

Today Craig Llyod, of Slash Gear, said that Cydia saw an unfathomable peak of 14,000 hits per second. This is a perfect example of why all developers, even the underground teams, cannot overlook load testing. You can’t predict if the application, app store or nifty jailbreak you launch is going to have a slow and steady adoption, or flood with traffic immediately. Development teams need to know; at what point does the application begin to degrade, how many concurrent users can it support, are there bottlenecks and how does all of this effect the user experience? The benefits of load testing and attaining peak performance cannot be outweighed by common cost or time arguments. Teams need to plan for the worst case (or rather best case peak traffic) scenarios.

In the case of evasi0n, the rush of traffic perhaps caught evad3rs and the Cydia appstore developers by surprise. Jailbreaking, while having major benefits, puts users’ device stability at risk and can void an Apple warranty. However, yesterday’s incident perhaps indicates that the adoption of jailbreaking is growing. Whether your an enterprise, startup or underground developer – if you want your software to work remember to load test.

Quantifying success is a rewarding if not always exact or easy process. As you know, last week we made a bit of a splash with the announcement of Applause, an new type of app analytics tool that uses complex math to scour over 55 million reviews in the App Store and Google Play in order to produce a 0-100 Applause Score for your app.

But did you know we’ve also been busy working with the analysts at IDC to quantify the return on investment (ROI) for companies that leverage our in-the-wild testing services? Companies big and small – from startups to enterprises – have had tremendous success working with uTest and our community of more than 70,000 professional testers in 190 countries. They’ve decreased their time to market, increased their app quality and reduced their testing costs. And now, thanks to our just-released ExpertROI Spotlight report by IDC’s Randy Perry, you’ll learn how one client saved money and expanded its mobile app footprint with their customers. The paper takes an in-depth look at the success real estate company Trulia found with uTest and how the investment paid off in only a matter of months.

Highlights of the paper include:

• Trulia experienced a quantifiable ROI of 280%
• Trulia reduced testing costs by 95%
• Trulia increased its customer base by 10%-15%

For more detail on how Trulia benefited from working with uTest, download the IDC report today. For more uTest news, stay tuned for a major announcement in the coming days.

For those familiar with our Testing the Limits series, you may have noticed that I like to throw in an open-ended, fill-in-the-blank question every once and awhile. So instead of keeping these questions exclusively for our guests, I thought I would open them up for you, our readers.

So if you’re up for it, go ahead and answer these 10 questions for software testers in the comments field below:

1. My favorite part about being a QA professional is ____.
2. If I had to quit my job as a tester, I would become a _____ because _____.
3. The testing subject that I’m interested in learning more about is ____.
4. The worst bug I ever found would have resulted in _____ if it hadn’t been discovered.
5. If there was one thing I wish people would understand about testing, it’s that _____.
6. I’ve never understood why developers _____.
7. I think that tester certifications are _____.
8. My experience with the Agile methodology has generally been  _____.
9. If I had to suggest one testing book to read, it would be ____.
10. When I’m not testing software, you can find me ____.

Thanks for participating. Have a great weekend everyone!

As you may have heard it was a big week here at uTest Nation with the announcement of a first-of-its-kind, free mobile app analytics tool – Applause. And if you hadn’t heard yet, then you can stop clapping… that’s the name of the tool.

Applause analyzes more than 50 million reviews and ratings across 1 million iOS and Android apps. The tool was launched by the forward-looking research arm of uTest, uTest Labs. Until yesterday, there had been no way for business and technical owners to gather insight into what users like and don’t like about their app – other than manually sorting through mounds of data and thousands of reviews. Even then, the data was likely to be subjective at best… and inaccurate at worst. This is why Applause is, well, a big deal.  

Yesterday’s announcement was reported by numerous business and tech publications. So how does the press feel about Applause? If you didn’t catch the news, here are a few of the more prominent stories right now:

• uTest’s Mobile App Genome: Forbes (Peter Cohan) “You might think that a start-up should focus on one product and try to make it a success before branching out. But Southborough, Mass.-based uTest – a 120-person start-up that bills itself as the world’s leader when it comes to ”in-the-wild app testing services” that would have found Apple (AAPL) Maps’ six most epic fails before launch — is betting about 10% of its peoples’ time on a two new business initiatives”… Read More.

• New Analytics Service Applause Ranks App Quality: ReadWrite (Dan Rowinski)
  “Businesses with several popular apps on Google Play and Apple’s App Store need to know what people think of their apps without having to slave over the notoriously skewed star rankings and comments. They want to be able to parse that data to know where they need to get better. Mobile testing company uTest set out to do the extremely difficult: quantify app quality. Wednesday morning it released a new product called Applause designed to do just that”… Read More.

• uTest Launches Applause, An iOS And Android Mobile App Analytics Dashboard: TechCrunch (Romain Dillet)
  “Boston-based uTest just announced a new product, Applause. It’s a mobile app analytics service that has two key advantages. It works across Apple’s App Store and Google Play and it gives an easy-to-understand score, reminiscent of other scoring services such as social influence startup Klout.”…Read More.

• uTest Unveils Applause, a ‘Klout for Mobile Apps’ Analytics Tool for Businesses: The Next Web (Paul Sawers)
  “uTest serves as a marketplace for ‘in-the-wild’ software testing services. With its latest Applause product, it analyzes more than 50 million reviews in the App Store and Google Play, constituting somewhere in the region of 1 million apps.”…Read More.

• Want to Know How Your App is Doing? Just Listen to uTest’s Applause: Boston.com (Michael Morisy)
  “Should you invest your limited developer resources into a design overhaul or streamlining the purchase process? uTest’s new Applause service lets you quickly analyze customer reactions and helps steer you toward a better product.”… Read More.

• uTest Launches Applause, Trying to Crack the Code on What Makes a Great Mobile App: BostInno (Walter Frick)
  “Applause crawls both Apple and Google’s app stores to see what users are saying in the reviews that get posted. While the app stores give a simple view of how much users like an app, based on a star rating system, Applause goes farther.”…Read More.

• Read more …

For months we’ve hinted- ever so subtly – that something BIG was on the horizon. Well, the wait is over, and today, we’re pleased to finally make that BIG announcement: uTest Labs has launched a new mobile app analytics tool called Applause.

I know that’s a lot to take in all at once, so let’s break it down piece by piece. First up, Applause.

Applause is a new type of app analytics tool – one that analyzes more than 50 million reviews & ratings across 1 million iOS and Android apps – enabling companies to monitor and measure mobile app quality and user satisfaction.

Applause provides an intuitive 0-100 Applause Score, which synthesizes user satisfaction, app quality and the overall amount of applause that users give an app. This enables brands to measure and improve their app experience, compared with competitors’ apps. It also grades each app across ten Applause Attributes, which enables companies to measure how their app stacks up from version to version, and against the competition.

Essentially, Applause fills a massive blind spot for companies in more accurately measuring their mobile app efforts. Until now, companies simply knew how many stars an app had in the respective app stores or counted downloads. Now, with Applause, brands can quickly and easily see results of deep data analysis and make informed decisions about where to best direct resources to provide the best customer experience possible.

Companies can utilize Applause to look shallow & wide – to compare how their app is performing among all apps in their category (social media, gaming, retail, etc.) or deep & narrow – to see how a particular app is performing across a series of deep, data-driven attributes over time and versions.

More information can be found on the shiny new Applause website. Go check it out.

And now for the second part of today’s new: The formation of uTest Labs.

uTest Labs is our recently formed research & development group. Its charter is to develop solutions to forward-looking problems or challenges that the market hasn’t clearly articulated yet. The overarching goal is to advance the cause of developing and testing web or mobile applications that simply delight users.

Co-located in our Massachusetts headquarters and Seattle offices, uTest Labs is led by some of the company’s top technologists and marketers, who have been given the resources, breathing room and directive to take big leaps and tackle big issues.

This “labs” approach played a major role in the earlier launches of AppGrader and uTest Extensions – and we intend to launch more big things from this group in the future.

Anyway, sorry for the wait. There’s much more to come, so stay tuned.

Until then, happy testing.