More Bang For Your Testing Buck – Follow Up Q&A With James Whittaker

Last week, more than three hundred software professionals from around the world tuned in to “More Bang For Your Testing Buck” – an exclusive webinar with best-selling author and Google Test Director James Whittaker. For those in attendance, we’re happy to post the following reprise of the Q&A session, where James addresses ACC requirements; the levels where testers are needed most; test tours; unit testing and more. See his answers below.

For those that missed the event, we have posted the webinar in its entirety (after the jump). If you’re interested in similar presentations, be sure to check out our webinar archives.

When is the right time to start with ACC in a development project? You just told us that ACC are not requirements. So ACC cannot be defined before requirements are clear? May ACC can even be defined just with the requirements?

Whittaker: Early in a project when there is a lot of flux, features added and cut/unreliable code being written, there isn’t much use in putting a lot of work into any sort of artifact and the ACC is no different. Once the features are locked, this is the time to begin in earnest. Even for large complicated products a good ACC can be built in hours, not days, so it’s unlikely that you’ll get too far behind. What we’ve found helpful is having a single test engineer maintain the ACC early in a product cycle and then when features are no longer in so much flux, that’s when additional testers can be added. Too many testers too early is just not productive. Developers know their code is buggy and it’s pretty unhelpful to keep reminding them of it.

The ACC and the requirements are very similar. In fact, I think the former could be a viable substitute.

It sounds like you think testers are only useful at the level of the UI?  Or did I misunderstand?  I currently work as a tester on an architectural team and code tests to test services and APIs.  Are you saying that is not a useful testing task, or that it should be owned by devs?

Whittaker: I want to make the distinction been early cycle testing, which are things like unit testing and test infrastructure, and late cycle testing which is more end-to-end. I think all late cycle testing should be done by testers and all early cycle testing should be owned by devs. Even within early cycle testing there are devs who specialize in writing such test code. At Google they are called Software Engineers in Test. They are engineers first and testers mostly in the sense that they write test code and not shipping features.

Now late cycle testing or end-to-end testing (if you like that phrase better) can be either UI or API level. At the UI we are stringing together inputs that make up user scenarios. At the API level we are writing code that utilizes the APIs with similar purpose: to test that the APIs can be used in real user scenarios.

The bottom line for me is that if you are thinking about the user, you are a tester. If you are thinking about the code, you are a developer.

Is there added risk that the useful highlighting of problem areas through the  “bad neighborhood tours” prevents someone looking for other problems with the highlighted area?

Whittaker: Only if you don’t also use other tours. The tours are meant to cover functionality as a whole and balance each other out. Each capability will be involved in any number of actual tours and if the right collection of tours is chosen, you should get good coverage. In fact, that’s kind of the whole point: tours are generalizations of test cases. They are a way to package up tests that are related in form, purpose or substance so they can be taken as a whole. So you might make a test pass that, hypothetically, looks like this:

  • Bad Neighborhood Tour: 18 tests total
  • Landmark Tour: 45 tests total
  • Rained-Out Tour: 3 tests total
  • FedEx Tour: 5 tests total
  • Money Tour: 31 tests total

And so forth.

Should we be writing the unit test or just advising on their development?

Whittaker: What advice are you going to offer? This is a good question and I am being serious here. Unless you know the code base and how it was constructed can you really help here? Unit tests are code that test low level boundary conditions and test that specific control structures are right. If you aren’t familiar with the code, you’ll not be much help. That’s the devs domain and his/her job to get it right. Being a crutch for developers weakens their resolve to do proper unit testing, don’t give them the ability to blame their poor unit tests on you.

If a dev is consistently shipping you code that falls down easily then it is clear that they are not doing good unit testing. And you are wasting your time if you are testing such rubbish. Now perhaps you can advise them: write better unit tests so your code is harder for me to break. Unit testing should remove most easy bugs. If you aren’t working hard to find more subtle problems, the devs are not doing their jobs.

At a company that is entrenched in the notion of test plans and doesn’t have the tools that you demonstrated for seamless bug reporting, what are some strategies for making the most of our testing hours?

Whittaker: Be more organized and thoughtful. In fact, it was the result of being more organized and thoughtful that led us to ACC and it’s accompanying toolset. We purposefully questioned every testing activity we were doing and played devils advocate: Are our test plans really being used? What actual value are they providing? Which parts are simply not worth doing? How long are we spending reporting and triaging bugs vs actual testing? Why are our test cases so hard to reuse? Why are tests hard to find? Where do new testers often get confused? And so on. Ultimately we started throwing out things like test plan as documents and focusing more on test planning as an activity. The value was in the activity not in the artifact.

What we ended up with is a process that maximizes the time we spend doing productive work and minimized the time we spent creating throw-away docs. You can do this too. You may end up with something that looks like our process (in which case you can adopt our tools) or you may end up with something that fits your team and its culture. Either way, you end up working smarter.

Editor’s Note: And as promised, here is the full-length version of the webinar.

Essential Guide to Mobile App Testing


  1. sonali says

    I found about this being a very usefull session.Unfortunetly the video is not available now. Where can I see the video?

  2. Mike says

    No objections here Joseph – and I know James feels the same way!

    Glad the webinar was helpful.

  3. joseph Agostini says

    I took snap shots of your drawings to share with my staff. We are often asked to defend the value of test professionals and these are simple, yet very powerful illustrations.

    I hope you don’t object. These will be for internal use only and I will give full attribution.

  4. Josh says

    I asked one of the questions featured in this article. Does this mean I get one of the signed books/t-shirts, and if so, is there somebody in particular I should contact?

  5. Scott Leavitt says

    Thank you Mike for posting the webinar recording and follow-up Q&A. I thoroughly enjoyed Dr. Whittaker’s presentation and appreciate seeing his follow-up here.

Leave a Reply

Your email address will not be published. Required fields are marked *