Software Testing Blog

Not to beat a dead horse or anything, but I wanted to briefly revisit Apple’s  “Antenna-Gate” fiasco to drive home a very important lesson for companies of all shapes and sizes: Rely too heavily on “lab-testing” and you are virtually guaranteed to get burned.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

1. An insanely expensive and elaborate lab is still a lab
Notice how the lab contained rooms where there is “absolutely no interference when it comes to wireless testing.” Does this sound like an environment that exists in the real world? Yes, it’s absolutely helpful for testing some aspects of such a solution. But at some point, you actually want the wireless interference (and all the other stuff that real users must deal with). No matter how much a company spends to replicate reality, it will never become reality. A lab is still a lab.

2. You ALWAYS need a fresh set of eyes (plus ears and hands)
Apparently, testing was completed by those who were extremely close to the actual product – those who knew the device inside and out, top to bottom, front to back. As a result, many usability issues (like the death grip, for one) were never factored into the testing equation. Only through ‘in-the-wild’ testing (i.e. real devices in the hands of real users) can a company expect to uncover these type of unique issues.

3. Non-standard use cases are vital to acing the user test
Apple’s testing team should be applauded for driving around in a van, testing the wireless connection. But why stop there? Why not think of 50 more unusual scenarios – aka “non-standard use cases”? Of course, relying entirely on lab testing to accomplish these tasks would be extremely expensive, even for Apple. But by using elements of crowdsourcing, they would be able to extend their testing coverage relatively cheap.

4.Global users? Global testing!
As the iPhone 4 makes its way into other markets, it will be interesting to see whether users encounter issues specific to their geo-location. Judging from the centralized conditions of the Apple lab, it should not surprise any readers of this blog if such complications appear. To sum it all up: your product needs to work where your users are. For 99.997% of companies and apps, this is NOT in a lab.

Are there other lessons to be learned from Apple’s testing methods? Have you seen other companies pay the price for relying on lab testing, while discounting “in-the-wild” testing? Drop us a comment and tell us all about it.

Here’s a quick video “unveiling” of the Apple’s impressive test lab: