Software Testing Blog

It was one year ago (June ’09) that James Whittaker helped us christen our ‘Testing The Limits’ interview series by being our first guest. And for much of the year, he held the distinction of generating the most page views… and then some guy named Patrick Copeland came along and took the lead a few months back.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

What we are working toward is a browser that is capable of testing applications that run inside it. The idea is to build the test framework into the browser, in our case Chrome, so that test artifacts don’t have to be moved around and installed in test labs and on test machines. Also since the browser and automation are one, problems such as Ajax and self-updating web pages won’t confuse automation. This is true cloud-based test automation. What I want is the ability to launch one of these WTF-equipped browsers and point it at the web site/app I want to test and have all the back-end test case storage, bug reporting, progress reporting etc managed for me, automatically, in the cloud. Tests that I wrote or others wrote can be consumed by the browser and applied en mass. These assets all exist in the cloud at Google, why not make them universally available?

I’m betting you want the same thing.

uTest: You’ve argued that the past two decades of software testing innovation have been “the disposable decades” – too tied to the app, the domain and the technology. What does the next decade look like to you? What will surprise us most?

JW: At GTAC 2008 and EuroStar the same year I gave a talk on the Future of Testing and I am returning to GTAC in October 2010 to give an update on my vision for the next decade. I don’t want to preempt that talk, but I will say that my vision is far more aggressive now than it was then. What I thought was 10-20 years out two years ago will be here in less than 5. Things are changing fast and I am anxious to push them faster. We are working unnecessarily hard to test our applications and it is limiting innovation and creating a market full of sub par software. This has got to stop.

The central theme of my vision of the future is getting rid of this disposable nature of our test artifacts. We are reinventing the wheel every time we test wrt planning, test development, automation, reporting and metrics. Let’s get together and stop this. One of the reasons I am so nice to you guys at uTest is that you are working with the idea of leveraging the cloud and the crowd to perform testing. There is much more collective intelligence involved in what you guys are doing and I like that. I am anxious to get our testing tools in the hands of your crowd.

uTest: In your most recent blog post, you describe a (very near) future where test data will be more like a cellophane wrapper around a web app UI, an overlay that would replace the tiresome querying of multiple databases. Just how close are we to achieving this innovative bug and test case overlay and what would be the significance of such a feat?

JW: You keep trying to pull my GTAC talk out of me prematurely! But let me give a bit of a preview. Test managers need data to make decisions about how to optimize their testing and their testing resources. Most technical test managers I know end up creating a set of SQL queries that they use to get information about features, bugs, test cases, test runs and so forth. All the data is there, it’s just hard to access. Video gamers have a similar problem, there is a lot of information in the game and it is in a lot of different places. Our solution in testing should be the solution gamers use: a heads up display – a single location to surface all the information a tester and test manager needs to make the right decisions during a testing campaign. We are building a heads up display that can consume all this information and use it to guide the hand of the tester and test manager. This is part of what I meant earlier when I said that manual testing needs automated assistance. People doing exploratory testing without a HUD are like gamers who play without one. In the latter case, the gamer’s character gets killed, in the former case the tester is less effective.

uTest: You are a busy man, James, having less and less time to write witty blog posts (which we miss)! What has been keeping you most busy and what have your greatest accomplishments been in the past 12 months?

JW: Busy or just having so much fun I can’t take the time to write a lot? My greatest accomplishment happens every day: people who work for me questioning every aspect of our business and pushing innovative solutions to other testers to improve the way they execute. I had a direct tell me two weeks ago that he couldn’t wait for Sunday to pass so he could come to work on Monday. Nothing else I do compares to that.

uTest: One of our top testers, Madhukar Jain, would like to know: Do you see cloud computing as the next evolving challenge in testing? We would take this one step further and also ask if the cloud represents or enables the fourth gen of testing, or testsourcing, you’ve often alluded to? What is the cloud’s potential to empower the crowd (third gen)?

JW: I see cloud computing as the way to solve the testing problem. And yet again you are forcing me to reveal my GTAC talk prematurely. So be it. Imagine for a moment every test case database, bug repository, build script, static analysis tool, metrics engine, and so forth in the cloud. Every tool and scrap of information a tester will ever need stored in the cloud and surfaced in his HUD when it is most relevant. It’s already stored on some server somewhere, putting it the cloud is a small step. Over time (hopefully a short time) we’ll be able to use this information seamlessly. Instead of the Internet, you have the Testernet capable of answering any question you have about testing and being there to guide you through whatever testing tasks you are performing. Other industries are already doing this. We’re behind the curve in testing.

Do you remember Star Trek The Next Generation? My friend and fellow Googler Dan Massey and I like to hold this series up as the type of technology we want to build. Geordie and Data were constantly “reconfiguring the subroutines” to solve some important task. They could reprogram the phasers, the photon torpedos, the holodeck, anything on the ship! Plug and play “subroutines!” They did this from any console on the ship. They did this without installing any back end repositories or test assets because it was all available in the cloud. They did this without running any all-night build scripts or writing new test cases. The build happened automatically, test assets were applied automatically. The reason Geordie and Data could do this programming so quickly is because they only had to solve the problem they were working on, they would ignore all the back-end build and test nonsense better suited for a computer.

I bet Geordie had one hell of a testing HUD and never had to wade through the complexity of partially automated build systems and SQL queries to find the tests and test data he needs, in fact I bet he didn’t even know anything about these backend systems. They were hidden in the cloud, exactly where they belong, allowing him to concentrate on his programming.

Editor’s note: Read Part II of the interview.