The Verizon iPhone: Testing Challenges

A wise man once said that “if it’s on the internet, it must be true.” News that Verizon will finally carry the iPhone is all over the net, ipso facto….

But while the pundits discuss how this new alliance with affect user stats, downloads and stock prices – and while AT&T and Verizon exchange jabs – I wanted to take a minute and discuss the implications it has for mobile app testing. The practice, that is, not the website.

Until now, Apple had made things relatively easy for mobile app developers. I know that may sound absurd to developers who have lost hair and sleep over problems with UDIDs and App Store acceptance, but it’s true. You see, unlike Android, Windows Mobile, BlackBerry and the others, Apple simplified the testing process by having only one device manufacturer and carrier. Sure, you still needed to make sure your iPhone app worked across iOS 3G, 3GS and 4, but the overall testing matrix was much less complicated. Ask any Android developer.

That’s about to change in a big way.

But don’t take my word for it. We recently asked Matt Evans, the former QA Director of the Palm Pre smartphone, for his thoughts on mobile testing challenges in the coming years ahead. Granted this was before the Verizon news came out, but see if you can put 2 and 2 together based on his insight:

Continue Reading

Testing the Limits With Matt Evans from @Mozilla – Part I

What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.

********

uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.
Continue Reading

The Time Traveler’s Phone

This story was blowing up on the web last week: An Irish filmmaker claims to have spotted a time traveler – that’s right, a time traveler – in an old Charlie Chaplin movie circa 1928. What would prompt someone to make such an outrageous claim you ask? Apparently, it was the cell phone that gave her away (pictured left). Amateur time travelers…

Now, if I were to employ Occam’s Razor, I would say the most plausible explanation is that the device in question is actually a hearing aid, as claimed by numerous internet commentators. But since the person in the picture is clearly a time traveler, we must ask ourselves: What in the world is she doing with a cell phone in 1928!?! And who is she talking to?

My theory is that our time traveling friend is a software tester – possibly from our very own tester community. Hear me out.

Continue Reading

Mobile Barcode Scanning Up 700%, Android Leads The Pack

As 2010 starts winding down, the mobile app revolution continues to wholly define this year in tech. Every day more mobile innovations are being updated and perfected to match our – the mobile consumers – needs. One such emerging trend is mobile barcode scanning.

According to ReadWriteMobile, a new study by barcode tech company ScanBuy claims that barcode scanning is up 700% in 2010!

Android was the most popular smartphone platform by far with 45% of barcode users, followed by Blackberry (27%), iPhone (15%), Symbian (9%), Java (3%) and Windows Mobile (1%).
Other Key Findings Include:

  • Health and beauty products were the most popular items among 1D (UPC) scans with 21% of users, followed by groceries (14.4%), books (12.6%) and kitchen items (9.2%).
  • Over 45 countries have scanned barcodes.
  • Linking to a website is the most popular action delivered by a 2D barcode scan with 85% of scans.
  • 1D (UPC) and 2D (QR) codes are being scanned equally.

In response to this huge news, I thought I’d have some fun with URL shortening service bit.ly‘s new QR-generation tool that launched a few days ago (FYI: goo.gl launched a few weeks ago too). Go ahead! Scan away and see where it takes you! (Hint: I am the PR Maven ;).)

While barcode scanning isn’t new technology by any stretch, the 2010 mobile boom is driving its increasing popularity. Even Calvin Klein recently replaced its massive billboards in New York and LA with QR codes (pictured above) – not to mention the giant QR codes in Times Square! I wonder what new and exciting mobile app testing doors this will open…

Mobile App World, London: October 19-20, 2010

Apps! Apps! And more apps! As the summer starts winding down here at uTest, we’ve been able to take a step back and a closer look at the big trends emerging all around us. What has been most apparent is the tremendous spike in mobile app testing needs. From top marketing agencies to retail giants to social gaming startups, our customers are developing more mobile apps to grow (or define) their businesses than ever before.

According to Game Developer Research, 25% of game developers are now making mobile games – that’s up from a mere 12% in 2009!

In addition, a survey conducted by iGR found that more than half (53%) of US mobile developers are building apps for Apple’s iPhone OS. BlackBerry was the next most popular, followed by Android and Windows Mobile.

In response to this incredible momentum, this year marks the launch of Mobile App World 2010, where global leaders in mobile tech and app development and entrepreneurs will gather to network and learn about the latest developments and innovations.

uTest will be among the outstanding line-up of more than 40 speakers, which includes Google, Microsoft, Ericsson, Orange Global and the BBC, who will be discussing the future of mobile apps. Shoot us a note if you’ll be around!

Note: If you’re looking for some cool, new mobile apps, check out Mobile App World’s August Apps Of The Month. You may spot a uTester’s favorite app! :)

In-The-Lab Testing vs. In-The-Wild Testing: Lessons from “Antenna-Gate”

Not to beat a dead horse or anything, but I wanted to briefly revisit Apple’s  “Antenna-Gate” fiasco to drive home a very important lesson for companies of all shapes and sizes: Rely too heavily on “lab-testing” and you are virtually guaranteed to get burned.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

Continue Reading

Mobile Developers: Addicted to Beta Testing?

Safe to say that mobile app development has greatly outpaced mobile app testing over the last few years. In other words, while the applications and platforms have seen tremendous technological advances (iPhone 4 bugs notwithstanding) the same cannot be said of mobile testing methodologies.

Case in point: The majority of mobile app developers remain overwhelmingly reliant on internal beta testing.

Here with proof is VisionMobile, who recently published a fascinating report on the growing mobile app ecosystem – a must-read for anyone involved in the space (developers, marketers, users, etc). From a QA point of view, the report further establishes that although testing innovations will ALWAYS trail those of development, the gap need not be so wide.

Here’s an excerpt that sums the whole thing up:

Internal beta testing is the most popular technique used by the vast majority (nearly 70 percent) of respondents, with beta testing with users and peer reviewing the next most popular techniques. Only 20 percent of respondents use focus groups or research of their own. Overall, North American developers are somewhat more sophisticated in their application planning, with 97 percent using beta testing as a standard part of application development and with broader use of a portfolio of planning techniques as well.

Yet, small development firms have limited means today to beta test and peer review their applications with a crosssection of representative users. Given the hundreds of thousands of mobile apps, we believe that efficient (crowd-sourced) testing of apps in a global market of users is considerably under-utilized. This presents an opportunity for the few solution providers in this segment – Mob4Hire and uTest.com, for example – but also for network operators, who can generate a channel for testing applications with end users, and provide an open feedback support system back to developers.

Other notable findings included:

Continue Reading

How Many Bars Do You *Really* Have?

So maybe it wasn’t AT&T’s fault after all.

Apple recently revealed that there is a fundamental flaw in their method for calculating how many signal bars to display.  And we have the iPhone 4 (and its “learn to hold your phone the right way” fiasco) to thank for bringing this software snafu to light.

CNN Money shares the following details from Apple:

“Upon investigation, we were stunned to find that the formula we use to calculate how many bars of signal strength to display is totally wrong,” Apple wrote in a statement posted on its website. “Our formula, in many instances, mistakenly displays 2 more bars than it should for a given signal strength.”

That means, for example, that iPhones sometimes display four bars when they should be displaying two. Apple said users reporting a significant drop in bars when they hold their iPhone 4 are probably in an area of “very weak signal strength” but were unaware of that because the phone displayed four to five bars.

“Their big drop in bars is because their high bars were never real in the first place,” the company said.

Perhaps most surprising, Apple disclosed that the problem is not confined to the iPhone 4.  The faulty formula has been present in every iPhone model since the 2007 original.  Questions remain about whether the issue is strictly software-related, or if it also involved hardware problems.  However, Apple has said it will release a free software update in the next several weeks to fix the glitch. It will use a new formula recommended by AT&T.

MobileAppTesting.com Debuts — Promises To Tell You What’s What In Mobile

Like a rocket ship breaking the bounds of Earth’s gravity… like a bird soaring majestically over the open sunlit plains… like a spit wad hurled from the back of the classroom… today, uTest announced the launch of MobileAppTesting.com.  You can also follow our wit and wisdom on Twitter @mobile_app_test.

One thing that you won’t find on this site are ads, subscriptions, hooks or any kind of commercial agenda (don’t tell our investors!).  In fact, we created this site simply because mobile is the next frontier of app development and testing – and the fastest-growing segment of uTest’s business. So we wanted to give something back to mobile app developers, testers and entrepreneurs — and have a little fun at the same time.

We’ll will work with partners, pundits and pioneers (actively seeking co-conspirators) to create original, thought-provoking content about the entire mobile app ecosystem — from app developers to device makers to wireless carriers.  Whether it’s the apps arms race, the constant carrier battles, or the next must-have device, MobileAppTesting.com will be there with equal parts education and entertainment.

This site features user-generated content, contests, product reviews and guest interviews with mobile execs… stuff you can’t find anywhere else, including:

  • Hard-earned lessons for entrepreneurs, techies and investors who want to create world-class mobile apps
  • Overcoming obstacles unique to developing and testing apps for iPhone, Android, BlackBerry and Symbian
  • Breaking news and product reviews (for apps, devices, networks and more) from top bloggers and journalists on the front lines of the mobile app explosion
  • Interviews from people who live it, offering their insights from the worlds of mobile app marketing, design, development and testing
  • Following uTest to mobile industry conferences, networking events, speaking opps and meet-ups

Want to be published on MobileAppTesting?  Have a topic you want us to tackle?  Feel the need to ask what the heck we think we’re doing?  Shoot us a note or drop a comment.

Where In The World Is Doron Reuveni?

Well, today he’s sticking close to home in Boston. Tomorrow he’ll land in London… and before the week is out, he’ll hit Tel Aviv.

Doron starts Wednesday morning off (after his usual 10-mile run, of course!) in London with some tea and networking with friend and colleague, James Whittaker and UK partner, TCL.

Then he’s off to QCon London, an excellent conference for the enterprise software community. On Friday, 3/12 @ 2pm, he’ll be presenting at QCon re: The Mobile App Quality Challenge & How Crowdsourcing Can Help.

Doron is one of five software testing leaders chosen to present in the “How Do You Test That?” track. This track explores unique solutions created to address situations in which automated testing does not suffice.

And on the last leg of his marathon journey, Doron will present at Garage Geeks in Israel on Monday, 3/15 @ 8pm. There, Doron will be taking a deep dive into the topic of Crowdsourcing, and how smart recruiting, training and incentives can turn an unstructured, loosely assembled mob into a unified, professional community.

So, where in the world is Doron this week?  Catch him if you can!