Software Testing Blog

Two years ago, we got this crazy idea to start interviewing the giants of software testing, for a series now known as Testing the Limits. To kick things off, we shot some questions back and forth with the distinguished testing author/teacher/speaker/consultant…(deep breath)…. James Whittaker!

A frequent guest of the uTest blog – as well as the author and host of several eBooks and webinars – James is known throughout the industry as the Test Director for a little company called Google. If you’ve stopped by the Google Testing Blog at all in the last year or two, chances are you’ve read his posts.

Anyway, we’re extremely excited to have James back for his third exclusive interview – and we’ve got a lot of ground to cover. In the first part of our Q&A, we discuss some recent changes at Google; a possible book deal; the future of cloud computing; testing Chrome OS; the problem with test automation; the upcoming GTAC event and why testing is (believe it or not) getting easier. For the second half of the interview, be sure to check back tomorrow.

**********

uTest: Good to have you back, James. Tell us what you’ve been up to? Anything different in your life at Google?

James: Funny you should lead with that! Yes, my role at Google is changing. I’ve passed the leadership of Chrome and Chrome OS testing to a colleague and taken over Cloud Computing. Lots of amazingly complicated back end/data center testing issues which are new and exciting. Google keeps pushing me outside my comfort zone in the most endearing way. Retirement in place hasn’t really been an option. But hey, one can dream.

uTest: We’ve noticed you’re blogging a lot more of late and you’ve started a Twitter account! How do you manage to find time for these things with the new challenges you’re faced with?

James: Believe it or not, the Cloud stuff is actually smaller than the Chrome/Chrome OS work in terms of the amount of time out of my day. The level of automation is very high and there is a lot we’ve been able to do put some very hard sub-problems on autopilot. My team here is seriously Jedi quality folk. Definitely good fodder for some future talk. But the Google-wide testing efforts in terms of security testing, testing tools, development tools and general evangelism is also officially part of my day job. I own the Google Testing blog and am also running GTAC this year. And yes, I finally succumbed to Twitter. It’s actually a lot of fun so far, 140 characters is so do-able.

uTest: Speaking of the blog, I noticed Pat Copeland mentioned the popularity of the current “How Google Tests Software” series. Any plans on turning that into a book?

James: More than plans, I am under contract with Addison Wesley and am working diligently on it. I am finding that as I dig deeper into Google internal testing processes I have to be more careful about what I publish. I have to avoid talking about confidential technology and innovations created by other Googlers who aren’t ready to discuss their work externally. So there is a backlog of stuff that requires review and once approval occurs, it makes sense to publish it all at once. The book format is ideal for that as opposed to trickling it out in blog form. Besides I love books, they are so cuddly and blogs are so not cuddly. And I also have two co-authors Jeff Carollo and Jason Arbon who I am mentoring through the writing process and that is bringing me a lot of satisfaction.

uTest: You made a comment in one of your blog posts about “automating to within the last inch of the human mind.” Can you tell us what you mean by that.

James: Hey, you’re stealing blog material from me! I was planning on describing this on the Google Testing blog as it did get a number of comments from curious readers. The central idea is that I think that test automation is often built to solve too big a problem. This broad scope makes automation brittle and flaky because it’s trying to do too much. There are certain things that automation is good at and certain things humans are good at and it seems to me a hybrid approach is better. What I want is automation that makes my job as a human easier. Automation is good at analyzing data and noticing patterns. It is not good at determining relevance and making judgment calls. Fortunately humans excel at judgment. So when I am doing exploratory testing, I want automation following me and helping. I can watch the UI (which it can’t) and it can watch the instructions hit the CPU (which I can’t). I can identify usability issues and the automation can trap exceptions and notice system issues. It’s cooperative and symbiotic. We have tools for this and are actively designing more tools where the automation gathers what it can and then presents it to the human for a decision…to within the last inch of the human mind. Get it?

uTest: Absolutely. I am sure the uTest crowd will like the message that the human mind is the most important part of the equation and that automation is here to help!

James: You bet. The human mind is key and uTest provides minds that scale. I may be trying to drive a lot of hard testing problems to extinction, but intelligence at scale is not one of them.

uTest: The theme of last year’s GTAC was “Test to Testability” – where you highlighted several methodologies and tools that could be used to build testability into products. What is the theme of this year’s GTAC conference and what are the implications for testers and testing managers?

James: “Cloudy with a Chance of Tests” It’s Cloud Computing with a twist. At Google we believe the cloud fundamentally changes not only the way testing tools and techniques are applied but also the very nature of testing itself. If this is too cloudy of an answer, stay tuned!

uTest: The world of testing moves incredibly fast at times. That said, is it possible to “future-proof” the theories and methods you write about in your books? Talk about some of the challenges of trying to stay current on the latest testing trends.

James: That’s easy. You stay current by inventing the latest trends not following them. Invention is less about being able to read the future and more about a willingness to experiment and be wrong. You should see our drawing room floor at Google, it’s a graveyard of failed experiments. We purposefully try to do things differently to make sure we’re not missing some great new idea or technique. We’ll try almost anything once and we push hard to fail fast. Staying open minded is the key. If you are willing to experiment and expect everything you try to fail, when something actually works, well, you know its something special.

uTest: Big picture question: With the emergence of the cloud, automated tools and other technologies….is software testing getting easier? Or is it a case of the “more we learn, the less we know”?

James: It is getting easier, way easier. There are so many options now. Computing power that makes testing millions of configurations a matter of a couple hours instead of completely impossible. Crowdsourcing that alleviates the need for expensive hardware acquisition and maintenance and hard to manage off-shoring. Developers who get quality are in larger number now and they write fewer bugs. Newer technology is making old and stupid test ideas and tools obsolete. Gotta love that. The web as a single platform increases test reuse. Dev to test ratios are coming in line with reality and sensibility. Anyone who says testing is getting harder is doing it wrong.

Editor’s note: Check back tomorrow for the second half of our interview with James Whittaker.