Testing the Limits With Google’s James Whittaker – Part II

In the second and final installment of our Testing the Limits interview with James Whittaker, we get his thoughts on some recent changes to Google’s test philosophy; why certain principles cannot span all types of testing teams; mobile testing challenges; the value of software testing; subject matter for his upcoming book, How Google Tests Software; and much more.

If you missed part I of the interview, you can find it here. Also, be sure to scroll down to the end of the interview for links to more material from James Whittaker. Enjoy!


uTest: You’ve been at Google now for over two years. Looking back, what’s been the biggest hurdle you’ve had to overcome during this time? And how much have the company’s testing procedures changed over this period?

James: My boss Pat Copeland took me aside a few weeks into my starting at Google and said something like “I know you’ve accomplished a bunch of things outside Google, but that’s all in the past. You’ve got to accomplish something inside Google. If you don’t no one will listen to you.” It was good advice. The message was that my past got me into Google but would get me no further. I took leadership and responsibility for testing of Chrome and Chrome OS, hard problems, important problems that required things I am good at and things I am not good at. That was the hardest part, being pushed outside my comfort zone. I never liked the execution part before, schedules and plans and meetings and disasters. It’s a lot easier as a consultant where you can happily imagine those things without experiencing them firsthand. I can’t believe that I used to advise people on these sorts of things before. I was surprised at how much I was learning and how much I was able to contribute. Now I take every opportunity to work outside my comfort zone. That’s where growth occurs.

As to the second part of this question, Google’s testing procedures have changed a lot. I think the Test Engineer role has been completely reinvented in the past two years.

uTest: We would imagine that a lot of testers and managers at smaller companies that will view your book as interesting, but not necessarily relevant to their daily testing lives. Explain why this is not the case and talk a little about the challenges of writing for an audience that includes teams of all sizes.

James: But it is the case! You can say the same things about my other books too. My books are meant to make you think differently about testing. It’s up to the reader to make it relevant by putting it into practice. There is no way I can write a book relevant to any specific style of testing or the practices of any specific company (except Google of course). All I can do is offer information and ideas and deliver them in, hopefully, an entertaining way. The problem is that too many people work for companies who just want them to keep doing the same old stuff. Change is too hard for them. Even if they wanted to test the way my books suggest, they can’t. I feel sorry for those people. In a better economy I would tell them to get a new employer. In this economy, well, it’s tougher. But there are also a lot of people who do own their destiny and can make changes in the way their company does testing and treats testers.

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Google’s James Whittaker – Part I

Two years ago, we got this crazy idea to start interviewing the giants of software testing, for a series now known as Testing the Limits. To kick things off, we shot some questions back and forth with the distinguished testing author/teacher/speaker/consultant…(deep breath)…. James Whittaker!

A frequent guest of the uTest blog – as well as the author and host of several eBooks and webinars – James is known throughout the industry as the Test Director for a little company called Google. If you’ve stopped by the Google Testing Blog at all in the last year or two, chances are you’ve read his posts.

Anyway, we’re extremely excited to have James back for his third exclusive interview – and we’ve got a lot of ground to cover. In the first part of our Q&A, we discuss some recent changes at Google; a possible book deal; the future of cloud computing; testing Chrome OS; the problem with test automation; the upcoming GTAC event and why testing is (believe it or not) getting easier. For the second half of the interview, be sure to check back tomorrow.


uTest: Good to have you back, James. Tell us what you’ve been up to? Anything different in your life at Google?

James: Funny you should lead with that! Yes, my role at Google is changing. I’ve passed the leadership of Chrome and Chrome OS testing to a colleague and taken over Cloud Computing. Lots of amazingly complicated back end/data center testing issues which are new and exciting. Google keeps pushing me outside my comfort zone in the most endearing way. Retirement in place hasn’t really been an option. But hey, one can dream.

uTest: We’ve noticed you’re blogging a lot more of late and you’ve started a Twitter account! How do you manage to find time for these things with the new challenges you’re faced with?

James: Believe it or not, the Cloud stuff is actually smaller than the Chrome/Chrome OS work in terms of the amount of time out of my day. The level of automation is very high and there is a lot we’ve been able to do put some very hard sub-problems on autopilot. My team here is seriously Jedi quality folk. Definitely good fodder for some future talk. But the Google-wide testing efforts in terms of security testing, testing tools, development tools and general evangelism is also officially part of my day job. I own the Google Testing blog and am also running GTAC this year. And yes, I finally succumbed to Twitter. It’s actually a lot of fun so far, 140 characters is so do-able.

uTest: Speaking of the blog, I noticed Pat Copeland mentioned the popularity of the current “How Google Tests Software” series. Any plans on turning that into a book?

James: More than plans, I am under contract with Addison Wesley and am working diligently on it. I am finding that as I dig deeper into Google internal testing processes I have to be more careful about what I publish. I have to avoid talking about confidential technology and innovations created by other Googlers who aren’t ready to discuss their work externally. So there is a backlog of stuff that requires review and once approval occurs, it makes sense to publish it all at once. The book format is ideal for that as opposed to trickling it out in blog form. Besides I love books, they are so cuddly and blogs are so not cuddly. And I also have two co-authors Jeff Carollo and Jason Arbon who I am mentoring through the writing process and that is bringing me a lot of satisfaction.

Continue Reading

Essential Guide to Mobile App Testing

uTest Wins A Spot On BBJ’s ‘Best Places To Work’ List (2nd Year In A Row!)

We really can’t believe an entire year has gone by (BTW: what a fantastic ride it’s been!), but yesterday the Boston Business Journal included us in its top list of Best Places To Work (only 25 in small co. category) for the second year in a row!

We are so honored to be recognized in back-to-back years and listed next to amazing companies that demonstrate dedication to their employees’ professional growth and career development and provide a ROCKIN’ work environment.

The winners ranked highest among hundreds of companies whose employees filled out an anonymous survey with detailed questions regarding their overall work environment and experience.

The list includes global enterprises such as Google, Microsoft, Ritz-Carlton, and Comcast, as well as local startups such as Carbonite and HubSpot.

As Doron said, “We’re committed to achieving great commercial success, but we also strive to create a rewarding and meaningful experience for our 40+ in-house employees. They are the ones who make uTest’s success possible!”

THANK YOU to the entire uTest crew who make uTest not only a best place to work, but simply the best place to be day-to-day. There’s no way we would be where we are without our world-class team.

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Jakob Nielsen – Part I

What an honor it is to have Jakob Nielsen – the “King of Usability” – as our Testing the Limits guest this month. Jakob Nielsen, Ph.D. is principal of Nielsen Norman Group , a research and consulting firm that studies how people use technology. He is the author of many books, including Eyetracking Web Usability and Prioritizing Web Usability. He has invented several usability methods, including heuristic evaluation, and holds 79 United States patents, mainly on ways of making the Internet easier to use. For more, read his official biography.

In part I of our interview, we get his thoughts on the evolution of user experience; the superiority of native apps; tablet usability; the death of PDF files; iPhone vs. Android and other hot topics. Be sure to check back tomorrow for Part II of the interview. Enjoy!

uTest: Like everything, software usability is in a constant state of change. How have you managed to stay on top of a field that seems to get turned upside down every other month?

JN: The users keep me fresh. I don’t really have to know anything, because I can simply see what our test participants use and how they use it.

uTest: It seems to us that software usability is as much a study of human behavior than anything else. What other subjects would you advise people to study who want to learn more about user preferences? Psychology? Sociology? Others?

JN: The main thing I recommend is to study your actual users: invite a handful of representative customers to your location and run them through simple usability studies of your software. One day in the lab is worth a year in university lecture halls, in terms of actionable lessons learned. (And remember that your “usability lab” can be a regular office or conference room —as long as you shut the door.)

That said, it’s still well worth studying all branches of psychology (perceptual, cognitive, social, etc.). One of the most popular courses at the Usability Week conference is called “The Human Mind and Usability” and summarizes the most salient psych findings for designers who don’t have time to go back to school.

It’s also worth studying visual design, even if you’re never going to draw anything yourself. Knowing the concepts and language is helpful when communicating with graphic designers, both to let them know what you want and to understand their ideas.

uTest: In the world of mobile, there’s been a lot written on the subject of native apps vs. the mobile web. What’s your take on this debate? Do both methods have a role to play in the user landscape? And for companies just venturing the mobile realm, where would you tell them to focus their attention?

JN: Apps are superior for 3 reasons:

  • Empirically, users perform better with apps than with mobile sites in user testing.
  • Apps are much better at supporting disconnected use and poor connectivity, both of which will continue to be important use cases for years to come. When I’m in London and don’t feel like being robbed by “roaming” fees, any native mapping app will beat Google Maps at getting me to the British Museum.
  • Apps can be optimized for the specific hardware on each device. This will become more important in the future, as we get a broader range of devices.

Apps have the obvious downside of requiring more development resources, especially to be truly optimized for each device. If a company doesn’t have enough resources to do this right, it’s better to have a nice mobile site than a lame app.

A second downside of apps is that users have to install them. Our testing shows poor findability and usability in Apple’s Application Store, and many users won’t even bother downloading something at all for intermittent use. So ask yourself whether you’re really offering something within the hardcore mobile center of need: time-sensitive and/or location dependent, and whether your offer is truly compelling in this crowded space. Most companies are never going to make it big in mobile. In some cases all they need is to make their main website somewhat mobile-friendly. Many others should deliver a dedicated mobile site but not bother with apps.

uTest: Regarding tablets, we see a lot of companies taking their current iPhone app, increasing the graphic fidelity, and releasing it as an “original” iPad app. In your view, what the biggest mistake being made by companies developing apps specifically for tablet devices?

Continue Reading

Essential Guide to Mobile App Testing

How Google Tests Software: Small, Medium and Large

Google Test Director James Whittaker recently concluded his fantastic “How Google Tests Software” series. We covered Part I a few weeks back, but I wanted to re-direct your attention to his latest post, which deals with the size and scope of their various projects.

Despite the perception that Google’s testing is highly complex and indecipherable to us mere mortals, we find the opposite to be true. As Whittaker explains, testing scope is determined by “emphasizing  scope over form.” Plainly stated, testing comes in three sizes at Google: small, medium and large. Here’s his explanation:

Small Tests are mostly (but not always) automated and exercise the code within a single function or module. They are most likely written by a SWE or an SET and may require mocks and faked environments to run but TEs often pick these tests up when they are trying to diagnose a particular failure. For small tests the focus is on typical functional issues such as data corruption, error conditions and off by one errors. The question a small test attempts to answer is does this code do what it is supposed to do?

Medium Tests can be automated or manual and involve two or more features and specifically cover the interaction between those features. I’ve heard any number of SETs describe this as “testing a function and its nearest neighbors.” SETs drive the development of these tests early in the product cycle as individual features are completed and SWEs are heavily involved in writing, debugging and maintaining the actual tests. If a test fails or breaks, the developer takes care of it autonomously. Later in the development cycle TEs may perform medium tests either manually (in the event the test is difficult or prohibitively expensive to automate) or with automation. The question a medium test attempts to answer is does a set of near neighbor functions interoperate with each other the way they are supposed to?

Large Tests cover three or more (usually more) features and represent real user scenarios to the extent possible. There is some concern with overall integration of the features but large tests tend to be more results driven, i.e., did the software do what the user expects? All three roles are involved in writing large tests and everything from automation to exploratory testing can be the vehicle to accomplish accomplish it. The question a large test attempts to answer is does the product operate the way a user would expect?

Simple enough, yes? So if you’re in charge of testing at a growing company – and wish to follow in the footsteps of giants – you would be wise to read the entire series, starting here.

As a supplement to this series, here’s Whittaker’s last uTest appearance – a webinar titled “More Bang For Your Testing Buck” (after the jump). Enjoy!

Continue Reading

Essential Guide to Mobile App Testing