Tag Archives | in-the-wild testing

Four Reasons Software Testing Will Move Even Further Into the Wild by 2017

apple0132Ever since our inception, uTest and our colleagues within Applause have always been a huge proponent of what we like to call ‘In-the-Wild’ Testing.

Our community is made up of 150,000+ testers in 200 countries around the world, the largest of its kind, and our testers have already stretched the definition of what testing ‘in the wild’ can be, by testing countless customers’ apps on their own devices where they live, work and play.

That ‘play’ part of In-the-Wild testing is primed to take up a much larger slice of testers’ time. While we have already seen a taste of it with emerging technologies gradually being introduced into the mobile app mix, there are four major players primed to go mainstream in just a couple of short years. That means you can expect testers to be spending less time pushing buttons testing on mobile apps in their homes and offices…and more time ‘testing’ by jogging and buying socks. Here’s why.

Continue Reading →

Continue Reading

Mobile App Testing – An Interview with uTest’s Matt Johnston

Greetings, uTest Nation and its followers, I’m back to serve in my role as the guy who does that thing that we don’t like to do too much of around here at blog central – talk about ourselves.

Well, not really. I’m actually here to share an excellent interview that STP‘s Rich Hand conducted with uTest’s Chief Marketing Officer Matt Johnston. In the interview, Hand and Johnston discuss ways in which the current burgeoning app economy has elevated the game for the way enterprise businesses conduct mobile app testing. Johnston also describes the importance of testing mobile apps in the wild and what it means for QA professionals.

The interview comes just a few weeks ahead of Johnston’s Mobile-themed keynote at the annual STP conference in Miami.

Listen to the interview today and look for more updates on uTest’s busy fall conference schedule in the coming weeks!


Continue Reading

Obama: “Make Federal Sites More Mobile-Friendly”

It’s been a politically-themed week here on the uTest Blog. First there was the hacking mayor. Then there was Mitt Romney’s embarrassing mobile app. But why stop there? Why not blog about President Obama’s order to make federal sites more mobile friendly? You know what, that’s not a bad idea!

That’s right, President Obama has ordered federal agencies to optimize their sites and services for mobile devices. Before I share my thoughts, here’s ComputerWorld with the details:

U.S. President Barack Obama has ordered all major government agencies to make two key services available on mobile phones within a year, in an effort to embrace a growing trend toward Web surfing on mobile devices.

Obama, in a directive issued Wednesday, also ordered federal agencies to create websites to report on their mobile progress. The websites are due within 90 days.

Innovators in the private sector and the government have used the Internet and powerful computers to improve customer service, but “it is time for the federal government to do more,” Obama said in the memo. “For far too long, the American people have been forced to navigate a labyrinth of information across different government programs in order to find the services they need.”

Many government services are not optimized for smartphones or tablets, and other services aren’t available at all on those devices, Obama wrote.

Americans deserve a government that works for them anytime, anywhere, and on any device. By making important services accessible from your phone and sharing government data with entrepreneurs, we are giving hard-working families and businesses tools that will help them succeed.”

And now, a few thoughts, questions and reactions:

Continue Reading →

Continue Reading

The Best of In-The-Wild Testing (so far)

A few weeks ago we launched our In-The-Wild Testing blog, dedicated to the lighter side of testing (amoung other topics). If you haven’t visited yet, you’re really missing out. Here are a few of the top stories so far:

Sharks With Frickin’ Laser Beams Attached To Them
What better way to test your latest laser product than by attaching it to a live shark? Dr. Evil would be proud.

Test Requirement: Proofread
When testing, it’s important to read the content carefully to see if you any words out.

The World’s Worst Water Slide
It looks like it was designed by Dr. Suess, but this water slide is definitely not for kids. Or adults. Or even crash test dummies.

Coding Error Disturbs Coding Contest
Making mistakes is embarrassing enough, but making a coding mistake in a system designed to sell tickets for a coding event to coders is a whole new level of red-faced “oh-uh.”

Five Products That Will Make You Angry
The Pringles can, the over-sized power adapter and other products that will make you want to kill indiscriminately (or just get mildly annoyed).

Continue Reading →

Continue Reading

uTest Launches AppGrader for Android

There’s only a few things that can happen when a user downloads your mobile app. Unfortunately, most of them are bad. Here are a few common outcomes:

  • The app crashes
  • The app hangs
  • The app stalls
  • The app works exactly as expected

The point is this: Without proper testing, you’ll never really know how users are experiencing your mobile app. This is particularly true of the Android operating system, with its seemingly countless permutations of devices.

So to help make the mobile app testing process a little less complex, uTest is pleased to launch a new tool that we hope will help you catch some of these problems before your users do. We call it AppGrader, and today we’re launching it as a way to quickly test your Android apps.

What is AppGrader?
AppGrader is a free online tool that can be used to quickly test your mobile application on a variety of common devices. With AppGrader, you can load your application on several devices and get basic reporting about bugs associated with installing, loading and running your application.

Why Does It Matter?
As a proponent of in-the-wild testing, we believe that an application is only properly tested once it has been evaluated by real users, with real devices, in a wide number of locations. AppGrader is a way to provide a taste of in-the-wild testing, by accessing the real devices component (albiet in an automated fashion).

Those who are interested in what AppGrader has to offer will likely appreciate uTest’s full suite of testing services.

What Do You Get?
With AppGrader, you can get a custom score for your application. You’ll also receive a comparison of how well your app works compared to dozens of other popular apps on the same devices. If your app should crash on the device, you can quickly download a crash log to get a deeper look at the problem.

What Devices Does it Cover?
At this point in time, AppGrader is just for Android devices (and yes, we do have plans to launch this service for other mobile operating systems). Devices tested include:

  • Samsung Galaxy Nexus
  • Samsung Galaxy S II
  • Google Nexus S
  • LG Nitro HD
  • Samsung Galaxy Tab
  • HTC Thunderbolt
  • Sony Ericson Xperia
  • Motorola Droid X2
  • T-Mobile My Touch

AppGrader will also evaluate your application on devices across several popular carriers, including AT&T, Verizon Wireless and Sprint.

What Does It Cost?
Only your soul. Just kidding. It’s free.

How Do I Get Started?
To get started, simply fill out a brief form and upload your Android APK. Once your results are ready, you’ll be notified by email within minutes.

So what are you waiting for?

Try AppGrader Today >>>

Continue Reading

Why Testing Your Business App is Important

Make sure your app makes senseAs the age of mobile tightens its grip on the world, companies are working double-time to figure out exactly how the use of smartphones and tablets fit into their working world. While some companies have navigated this new terrain fairly easily, many companies are struggling to find the right balance when it comes to mobile programs. Their biggest downfalls are trying to completely recreate a desktop program into a clunky, over-stuffed mobile app and not understanding the tenets mobile design.

In fact, this is such an issue that it’s poised to cost US and UK based companies a pretty penny in the next year and a half. Here’s some research from Antenna Software that highlights the problem (from Computer Business Review):

U.K. and U.S business are planning to spend an estimated £285k on mobile software tools for their employees in the next year and a half, but much of that money will be wasted.

According to research released by Antenna Software, only 25% of IT and business decision makers said their employees had embraced their mobile initiatives. …

According to the Mobile Business Forecast 2012 report, many companies are failing to engage their employees on mobile projects because of poorly designed applications that lack business logic and usability.

“More businesses than ever are now building mobile apps to help employees work more effectively, but it’s clear that a good deal of time and money is going to waste through poor design,” said Ken Parmelee, Senior Director of Product Management at Antenna. “Companies need to pay more attention to the end user and how and when they are going to use the app.”

The important lesson here is that just because your app is free and has a built-in market doesn’t mean you can lower your standards or ignore what end users like. It isn’t enough to take a program employees use on a computer and make it “more accessible” by translating it into an app. You need to fully understand how the program can be useful on-the-go and focus solely on the features that would be handy and increase productivity in a mobile, untethered setting.

Continue Reading →

Continue Reading

How Will Google Test Project Glass?

Perhaps you’ve read about Google’s Project Glass – a set of augmented reality glasses that will provide users with real-time information right before their eyes. Literally.

After waiting awhile for an appending “April Fool’s” announcement that never came, we can now safely call your attention to this project’s unique testing challenges. Before we do that, however, here’s a good description of the project from TechCrunch:

To call these things glasses may be a bit of a stretch — early rumors noted that glasses bore a striking resemblance to a pair of Oakley Thumps, but the demo images on Project Glass’s Google+ page don’t look a thing like them. Rather, they appear to be constructed of a solid metal band that runs across the brow line, with a small heads-up display mounted on the right side.

The New York Times‘ Nick Bilton, who broke the Project Glass story today, went on to say that the prototype model seen in the images is just one of the potential designs currently in testing. Among others, one of the potential designs for Project Glass is (thankfully) meant to be attached to a person’s existing pair of glasses.

It’s also worth noting that as downright magical as these things could be, there’s still very little insight into how they would actually work. Bilton’s early write-up notes that the glasses will be capable of establishing a 3G or 4G wireless connection, but how exactly Google will shoehorn those components (just to name a few) into a comfortable headset is still up the air.

After watching the demo/concept video below, it’s clear that in-the-wild testing will have to play a major part in the quality of this product (should it ever come to fruition). And judging from the comments on their Google+ page, finding real-world beta testers will not be an issue. But that’s the easy part. Here’s a look at some heavy testing challenges they are likely to encounter:

Continue Reading →

Continue Reading

Lessons From the TweetDeck Security-Induced Outage

TweetDeck Taken Offline Because of Security BreachThe recent Tweekdeck issue can teach us two very important lessons: 1. You can’t find all the bugs in-the-lab because there are some issue that will only effect a small, small portion of your user base and those bugs will more often than not only be found in the hands of users, in-the-wild. 2. Security Testing doesn’t just including making sure hackers can’t get into your app. It should also test your app for proper handling of confidentiality, integrity, availability, non-repudiation, authorization and authentication. It’s this last one in particular that caught Tweetdeck off guard. In case you missed the story, here’s a recap from TechCrunch:

Twitter has taken its Tweetdeck app offline after an apparent bug has possibly given some Tweetdeck users access to others’ accounts.

A Sydney, Australia-based Tweetdeck user named Geoff Evason says he discovered today he was somehow able to access hundreds of Twitter and Facebook accounts through Tweetdeck. In an email to TechCrunch, he explained the situation like this:

“I’m a tweetdeck user. A bug has given me access to hundreds of twitter and facebooks account through tweetdeck. I didn’t do anything special to make this happen. I just logged in one day, the account was was slower than normal, and I could post from many more accounts.”

Get more details at TechCrunch >>>

Luckily the person who found this bug, Geoff Evason, wasn’t malicious or a hacker. He could have done cruel things with these twitter accounts and potentially caused even more damage with access to the private information people tend to share on Facebook. Instead, he did a little poking around to prove the bug was valid (like any good tester does) and went about getting the issue addressed and corrected.

The incident was so severe that Twitter (who owns Tweetdeck) took the entire application offline until the issue could be pin pointed and resolved. Kudos to Twitter for treating this as seriously as it did, but the question is, now what? What caused this bug? Was there an update that effected the app? Will Twitter do some comprehensive regression testing to make sure there are no other unnoticed issues? It should probably do some digging to make sure those other facets of security are up to par.

Continue Reading →

Continue Reading

Introducing: In-The-Wild Testing, the Blog (aka: The Lighter Side of Testing)

In-The-Wild TestingIf you’re a fan of in-the-wild testing (you know who you are), you’ll be pleased to hear that uTest recently launched a brand new blog dedicated to chronicling products that were tested (or those that should have been tested) under real-world conditions. From unmanned underwater vehicles, to web & mobile apps, to hardware designed to function under extreme conditions, to regular everyday products like cars or door knobs that somehow  fail miserably – and comically – we’ll be covering it all!

The In-The-Wild Testing Blog crew will attempt to put a smile on your face and remind everyone that testing in-the-wild really does pay off.  And to keep things fun and fresh, we’re going to need your help. So if you ever encounter something (online or offline products) that makes you wonder, “How in the world did this make it through testing and into the market?” write it up and send it on over to us at blog@utest.com. We’ll publish it and cite you as an in-the-wild testing rockstar!

And just to get this out of the way so our lawyers can go back to lawyering —  the views expressed in the In-The-Wild Testing Blog are not necessarily those of uTest, but we’ll have some fun along the way and explore the lighter, wilder side of testing.

So check out the newest member of our blog family and be sure to send in those guest posts if you come across something that really could have used some in-the-wild testing!

Continue Reading

7 Tips for Stellar Test Management

Tips to excel at test managementWe all know what happens when you rush through testing and push a new product out to market too early (hint: crashing and burning comes to mind). In the spirit of not releasing software with major security flaws, functional defects or usability missteps Traq Software has highlighted “7 Important Principals for Test Management.” Full disclosure: Traq Software sells QA management software, but the tips are good too keep in mind non-the-less. (Numbers 4 and 5 are my favorites!)

1. Make sure you have a repeatable process. A good process helps you see where you are and where you are going.

2. Don’t cut corners prior to a release. When the delivery date is getting closer there is, naturally, a tendency to want to skip some low priority test management tasks. In doing so you hope to get the product out on time. Resist this temptation.

3. Know the metrics. Defect find rates, cases executed and lines of code changed. All these metrics help you argue the case for why the product may, or may not, be ready for release.

4. Listen to the testers. Software testers are your projects headlights. They light the road at night and help you read the map. They are trying to help you get to your destination in the shortest amount of time. Ignore their advice and you can be sure you’ll end up taking the longest route to your destination.

5. Employ a good QA manager. The QA manager is like the pilot of the plane guiding the product to touch down. … good QA manager has an eye for balancing the demands of time, quality and features. He or she is worth listening too carefully.

6. Get the customer involved. The sooner your customer starts giving you feedback the sooner you can correct issues.

Continue Reading →

Continue Reading

Testing the #SXSW Mobile Apps (iOS vs. Android vs. Windows Phone vs. RIM)

For the second year in a row, uTest will be making an appearance at SXSW, the world-famous music/film/interactive conference in Austin, Texas. Unlike last year – where we spent most of our time eating, drinking and schmoozing with storm troopers – we have  some new, big plans in store.

The obvious difference is that we’ll be cruising around Austin in the RVIP Lounge, hitting up hotspots, giving rides, singing karaoke (poorly) and playing host to SXSW attendees throughout the week. More to come on that, but you can follow @InTheWildTest for deets on our adventures, and real-time locations if you’re at SXSW..

The other difference is that, instead of just talking about the merits of in-the-wild testing, we decided to show a real-world demonstration. So, over the last 36 hours, we assembled a select group of US-based testers to put the official SXSW mobile apps through their paces. In-the-wild testing means live testers, real devices, imperfect connectivity… basically, true real-world conditions. So we went to work testing SXSW’s official apps for iOS, Android, Windows Phone and BlackBerry. For iOS and Android, we also included tablet testing, to bring the comparison total to six.

Below are some top-level results (note that each category ranged from 1-5):


% of Total Bugs 17.7% 18.3% 18% 6.6% 23% 16.4%
Overall Score 4.1 4.0 4.6 4.7 3.8 4.2
Usability & Design 4.2 4.1 4.6 4.8 4.1 4.2
Features & Functionality 4.1 4.1 4.7 4.7 3.2 3.9
Application & Performance 3.3 3.2 3.9 4.0 3.5 3.7


Of course, these figures only tell part of the story. As the apps were tested in terms of functionality, performance, design, connectivity and other factors, several issues popped up on more than one occasion. Here were a few areas where some notable bugs were uncovered:

  • Incorrect time displays
  • Sync issues with registration and deleted items
  • Crashes on various tablet OS versions
  • Issues with installation
  • Social media integration
  • Issues with rating and uploading photos

It should be noted that despite these issues, the overall reaction from our community was positive for each of these applications. In fact, the overall ratings you see above are substantially higher than the industry norms, so kudos to the respective dev teams.

Anyway, if you’re at SXSW and want to learn more about In-The-Wild Testing, be sure to stop by the RVIP Lounge. If you’re not able to attend, then head on over to inthewildtesting.com.

Continue Reading

60% of Users Expect Mobile Sites To Load in 3 Seconds or Less

That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it. And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your site an extra two seconds (bring the grand total up to five) to load before aborting the mission. (According to data pulled from a Compuware survey). And if your site is slow just once it doesn’t matter, the damage has already been done. Need some more convincing? Check out this infographic to find out why load testing is so important.

Mobile Web Infographic

And remember, your mobile site doesn’t need to load well only in the lab. It has to work out in-the-wild … because apparently 67% of people use their smartphones on a date (and they’re probably in the middle of a crowded restaurant). From Tatango:

 Smartphones Usage By Event

Continue Reading