Software Testing Blog

Ho, ho, ho!  Whoa there, Blitzen– wasn’t it just Halloween?  It sure feels that way. After all, I still have two pounds of trick-or-treat candy to pretend I’m not eating.

Unfortunately, my four-year-old has already implored me to take down the skeleton and spiders hanging in the doorway because they’re going to scare away Santa.  So, rather than arguing the salient fact that Santa shimmies down the chimney versus ringing the doorbell, I’ve officially started morphing decor from the marvelous macabre to merry old Saint Nick.  Kids: 1. Mom: 0.

Nonetheless, the fact hasn’t escaped me that we’re two weeks away from Cyber Monday (November 28th), an occasion that online retailers have been planning for months.  Since summer, global brands and independent e-tailers have been testing and re-testing their mobile apps and web sites for functionality, usability, localization glitches and possible bottlenecks in site performance that could jeopardize their revenue potential.

Moreover, the ante has been upped now that the iPad and other tablets have entered the scene.  Online retailers that spent the last few years optimizing their mobile apps and porting them to additional platforms like Android, are now going through the process from scratch with tablets.  Not only are the specs non- standardized, varying significantly by manufacturer, device and network performance like smartphones.

Read more…

Our next video from our STPCon 2011 video series is with Karen Johnson. Karen is the owner of Software Test Management, Inc. and has been an active contributor to software testing conferences and quality assurance groups for many years. We caught up with her and asked about the mobile testing workshop she ran earlier this week:

Want to see more interviews from STPCon? Check out the full list here.

Over the past week, there’s been some hub bub over comments made by Motorola’s CEO Sanjay Jha.  According to IDG News Service, Jha “blamed the open Android app store for performance issues on some phones,” based on his statement: “Of all the Motorola Android devices that are returned, 70 percent come back because applications affect performance.”

Even though Motorola formally stated today (see MoCoNews article) that Jha’s comments were essentially misconstrued and didn’t accurately reflect his intentions, the issue has remained a lightning rod for debate.

But for those of us in the software testing community, there’s a truly, positive message embedded in this issue:  Motorola was validating the critical importance of QA testing in the app development process.  

After all, consider Jha’s statement that, “one of the good and problematic things about Android is that it’s very very open. So anyone can put applications, third-party apps, on the market without any testing process….For power consumption, CPU utilization, some of those things, those applications are not tested. We’re beginning to understand the impact that has.”

For professional software testers, that confirms how important our work is, and actually suggests that the scope of mobile testing should be expanded.

Essentially, Jha wasn’t really referring to functional testing.  Or testing exclusively in the “clean and ideal” conditions of a lab environment.  Instead, he was describing the need for usability testing in the real-world to subjectively examine how apps and devices perform in live conditions and affect the user experience.  For instance, did the app run sluggishly?   Did it seriously tax the battery life?  These are vital questions, particularly for apps heavy on audio and video. 

At the end of the day, consumers are unlikely to differentiate whether their frustration over poor performance is caused by the smartphone or the app…or the interaction of both.  They just want to have a great experience with their new mobile “toy” or get their work done. 

Because if there isn’t enough testing on every device that the app is developed for, then (as Jha said) the smartphone gets returned and everyone– including the app publisher–loses out.

We all know a mobile app released with a poor user experience pretty much guarantees its failure in the app stores. NHS Direct, an extension of the UK’s National Health Service, could not afford to take that risk (after all, there were patients on the line, literally), and turned to uTest for usability testing before their app hit the Android Market and Apple App Store.

Originally designed to be used via landline phones, NHS Direct – a service giving patients the ability to assess symptoms, get self-care and advice from nurses on call – eventually migrated to the web, and most recently, to smartphones.

Charlie Young was responsible for overseeing testing of the the app across iOS and Android OS devices. Lacking the in-house resources needed for this type of project, Charlie turned his attention to uTest.

“The safety of the application wasn’t an issue at that point,” said Charlie, who mentioned that the clinical requirements had already been met. “What we really needed to know was how humans would use this app. People use smartphones in many different ways, so we needed to make sure the user experience lived up to expectations – and that’s where uTest provided such value.”

Read more…

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.

*******

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

Read more…

Apps! Apps! And more apps! As the summer starts winding down here at uTest, we’ve been able to take a step back and a closer look at the big trends emerging all around us. What has been most apparent is the tremendous spike in mobile app testing needs. From top marketing agencies to retail giants to social gaming startups, our customers are developing more mobile apps to grow (or define) their businesses than ever before.

According to Game Developer Research, 25% of game developers are now making mobile games – that’s up from a mere 12% in 2009!

In addition, a survey conducted by iGR found that more than half (53%) of US mobile developers are building apps for Apple’s iPhone OS. BlackBerry was the next most popular, followed by Android and Windows Mobile.

In response to this incredible momentum, this year marks the launch of Mobile App World 2010, where global leaders in mobile tech and app development and entrepreneurs will gather to network and learn about the latest developments and innovations.

uTest will be among the outstanding line-up of more than 40 speakers, which includes Google, Microsoft, Ericsson, Orange Global and the BBC, who will be discussing the future of mobile apps. Shoot us a note if you’ll be around!

Note: If you’re looking for some cool, new mobile apps, check out Mobile App World’s August Apps Of The Month. You may spot a uTester’s favorite app!

Well, today he’s sticking close to home in Boston. Tomorrow he’ll land in London… and before the week is out, he’ll hit Tel Aviv.

Doron starts Wednesday morning off (after his usual 10-mile run, of course!) in London with some tea and networking with friend and colleague, James Whittaker and UK partner, TCL.

Then he’s off to QCon London, an excellent conference for the enterprise software community. On Friday, 3/12 @ 2pm, he’ll be presenting at QCon re: The Mobile App Quality Challenge & How Crowdsourcing Can Help.

Doron is one of five software testing leaders chosen to present in the “How Do You Test That?” track. This track explores unique solutions created to address situations in which automated testing does not suffice.

And on the last leg of his marathon journey, Doron will present at Garage Geeks in Israel on Monday, 3/15 @ 8pm. There, Doron will be taking a deep dive into the topic of Crowdsourcing, and how smart recruiting, training and incentives can turn an unstructured, loosely assembled mob into a unified, professional community.

So, where in the world is Doron this week?  Catch him if you can!

QUEST, one of the top software testing conferences, will be held in Dallas this year (April 19-23).  And uTest is getting geared up and is thrilled to be a part of this conference.

In addition to inviting Doron to be a keynote presenter, QUEST features a week-long agenda packed with more than 100 opportunities for attendees to build new skills and prepare for the testing professions of the future.

From exploratory testing to test automation to security audits to crowdsourced testing,  QUEST will cover a wide range of testing topics that give attendees insight into the latest best practices and innovative approaches to testing today. To learn more, here’s a sneak peek at the QUEST Magazine.

Special Note: Members of the uTest community interested in registering for QUEST are eligible for

Read more…

Today Apple announced their much anticipated tablet computer – the iPad.  With a 9.7 inch screen, the iPad is a supersized iPhone; and it’s already inspiring both love and hate from Apple fans worldwide.  Comments so far have ranged from “I want it now” to “I was expecting a lot more than an XL version of the iPhone with no phone capabilities.”

So what does the iPad mean for developers, testers, and Apple users?  Here are a few thoughts:

Layout now matters for iPhone developers.
Up until now, an iPhone app was one size fits all.  Every iPhone has the same resolution, meaning apps didn’t have to worry about scaling up or down.  Not anymore!  The iPad is a larger device, but it runs the same iPhone apps.  While it can automatically scale up an app designed for the iPhone, the results are kind of ugly.  The bigger screen real estate opens a lot of interesting possibilities, but for iPhone developers now is the time you need to start worrying about how your app will look on a larger screen that isn’t 480×320.

New interface means new challenges.
If it’s not enough that the iPad comes in a different size, now developers will also have a slew of new interface widgets to work with.  Whether or not those widgets will be available on the iPhone remains to be seen, but whatever the case developers will have their hands full making sure their apps look correct on each platform.

Testers needed!
Read more…

Nick Jones over at Gartner wrote a great piece about the current state of mobile apps.  It’s a must-read for mobile app developers and marketers.

His post is written about the iPhone, but it applies equally to Android, Blackberry or Symbian apps.  It also helps to explain why mobile apps have been the fastest growing segment of the uTest biz (phenomenal growth + still evolving market = a lot of testing to do).

Here are a few realizations from his recent trip to his local Apple store that seemed especially relevant for our readers (bullet points are his and the running commentary is mine) :

Read more…