Testing the Limits with James Bach (part 1)

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

  • A freelance sentry.
  • A small-time warlord.
  • An itinerant geometer.
  • Werewolf.
  • Werewolf hunter.
  • A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
  • Zorro.
  • A gentleman naturalist.
  • A buccaneer.
  • Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

JB: Years ago, Rational hired me to advise them about test processes, and for that reason I took the Rational Unified Process class. I had been working with the RUP team, and I wanted to see what the Rational instructor was saying about RUP. At lunch on the first day, I asked him if it was okay that I was asking challenging questions. He said that was fine. But at lunch on the second day he kicked me out of the class!

Most instructors don’t like me in their classes. So, when you say I’m forced to choose, it’s a question of which class I would prefer to disrupt. (Unless I have the choice of sitting in one of Cem Kaner’s classes at Florida Tech. I would definitely prefer that. Last time I took a class from Cem, I had to play solitaire furiously in the back of the room. It was the only way to stop myself from jumping up and shouting “Yes!” every three minutes.)

I would prefer to disrupt an ISTQB class.

uTest: A few months back you presented “How to Fake a Test Project” at STPCon, where you said “testers may accidentally find bugs because they don’t follow the test scripts precisely.” Are testing managers starting to catch on to this irony of such “best practices”, or are most of them still oblivious to this?

I don’t know. I think most test managers don’t think much about it. As I said in my talk, testing is easy to fake– *even by accident*– because testing is so intangible.

JB: Be honest, you’ve faked a few test projects in your day, right? C’mon you can tell us.

Yes. I remember walking over to my manager’s office at Borland– by this time I’d been doing test management for five years– carrying an estimate of how much testing was required for my project. The estimate was couched partly in terms of “test cases required.” But just as I reached his office, I realized that the numbers meant nothing.

Until that moment, I thought they meant something. Well, they did mean something, just not anything I could defend or explain. Suddenly it dawned on me that I was peddling an empty *story* about testing couched in numbers to make it look rational.

As I write this, that incident happened 17 years, 5 months, and 1 day ago. Since advising my boss that he should ignore my test case estimates at that meeting, I have never used test case counts as a metric on a test project. But that doesn’t explain why, for the 5 years, 1 month, and 21 days before that, I *did* use test case counts.

Of course, the explanation is that I hadn’t much thought about it. I went along with the convention. So, I can forgive people who fake test projects by accident. When we’re young we make mistakes. But we also learn and grow.

uTest: In your new book, Secrets of a Buccaneer-Scholar, you advocate alternatives to traditional schooling and certification – not just for testers, but for people in all professions. It seems to have worked well for you, but why are so many people still reluctant to ignore the certificates and degrees and just start testing? What’s holding them back?

JB: Fear and indoctrination. I recently argued with a fellow about why he was going through a certificate program that bored him and insulted him, instead of quitting that school and being a tester. He claimed that no one would give him a job without the certification. Well, he emailed me a few days ago and told me he just got a job, and that I was right.

Well of course! I’ve been a hiring manager, and the idea that hiring managers are zombies who only look at paper credentials is wrong. SOME managers do that, but you don’t really want to work for someone like, do you?

Seems to me uTest is a mechanism for building your reputation as a rapid tester. I encourage new testers to try it for exactly that reason.

uTest: What’s the most promising trend/development in testing?  And what’s the most dangerous or discouraging trend you’ve seen?

JB: I think the most promising trend is public peer conferences and mentoring. Things like the Weekend Testers, for instance, or the Rat Pack. Some of these people are inspired by guys like me, perhaps, but they are not followers. They are taking it upon themselves to re-invent testing by experiencing and examining it intensively. I’m proud of anything I have done or can do to encourage this sort of creativity.

The most dangerous trend is certification, which is systematically dumbing down our craft and preventing good people from getting work.

A more dangerous thing, which is not a trend but unfortunately a constant, is the astonishing incompetence of technology middle and upper management, who frequently demand stupid metrics and meaningless reports from the people who work for them, or who decide to outsource without having the first clue what they are giving up by dumping knowledgeable testers and sending critical work ten time zones away.

For some reason, test managers complain to me about this, but often don’t complain to their own bosses.

Last year, my brother created a world-class test team at LexisNexis, then someone many levels up from him decided to fire everyone and outsource. They never talked to my brother about this. Apparently they think good testing happens by magic. But here’s the thing, my brother decided that he was going to train the outsourcing company (who had claimed to know a lot about testing, but, surprise, didn’t actually know much at all) to do skilled exploratory testing. It worked. He’s pleased. And now his management at LexisNexis (former management, because they then fired him) must think outsourcing is a great strategy, having no idea that it works only because people like my brother have too much personal pride to let even senior management airheads experience a richly deserved failure.

uTest: We’re all familiar with your work, as well as that of your colleagues like Michael Bolton, Cem Kaner, Jon Bach and others. Are there any “great unknowns” out there in the testing blogosphere to whom we should be paying attention?  Anyone that should be ignored altogether as a dangerous thinker?

JB: I sometimes complain about people I think are dangerous. See my blog.

Basically, I hate bullies. I want a free and open craft. The most dangerous people are those whose personal ambition will mean discouraging innovation (perhaps through establishing some sort of baloney “testing standard” through the auspices of ISO), and encouraging the sort of test management practices that make software both expensive and sucky, while discouraging any smart ambitious person from entering the field in the first place.

As far as great unknowns, there are some. I think Lanette Creamer is someone to watch. Adam White, Ajay Balamurugadas, and Ben Yaroch, as well. There are the Weekend Tester guys, and a shadowy Context-Driven sub-community calling itself “The Rat Pack.” I come across testers, regularly, who are talented, but haven’t taken a step out into public discourse.

I am constantly recruiting for colleagues.

Be sure to check out Part 2 of our interview with James.

Essential Guide to Mobile App Testing


  1. says

    I’ve read a few just right stuff here. Certainly value bookmarking for revisiting. I wonder how so much attempt you put to create this sort of magnificent informative site.

  2. says

    Not-a-fan: Smart move staying anonymous. Once you’re known for being ignorant, it’s hard to get people to take you seriously. No cowardice in being a sniper, just make sure you hit your target.

    Unfortunately, you’re a bad shot. What James left out of his story about me being laid off is that one month after me and most of my team were notified of the layoff, my boss offered me a new job to stay on. Was it because of my ability to influence? I dunno, but I like to think it was due to being a graceful, professional leader who wanted to provide value despite a tough situation.

    Alas, even though I was offered a position at my same salary, I turned it down. It was not QA-focused, it was as a SME on a project where I would no longer be leading testers. So I said no thanks. I finished out my transition period, head held high and a new opportunity met me as I left the building.

  3. Keri S. says

    Thanks for taking the time to post this. The Q&A with James Bach has given me reason to believe that I may not be totally crazy and insane after all. I have always been the rogue tester… the one that didnt necessarily follow all of the methodologies… the one that works hand-in-hand with the developers… the one that finds solutions to issues vs just logging the issue. I always seem to be searching for that “right way” of doing things vs the normal way\the old school way. The time spent on Plans, Cases, Scripts, Scenarios, Matrix’s ~ beauricaratic papers ~ has always seemed to me to be time not-so-well-spent. I do believe that creating test artifacts is necessary but that is where my mind takes off and spins out of control with so many thoughts and opinions and trying to come up with a better way. I have had multiple conversations and presented several new processes with upper management attempting to streamline our processes but have had no luck. Time-to-market is a huge value add and it is because of these old-school methodologies that QA gets a bad wrap for being a road block and might be a direct cause of the push to outsource; but that is a whole seperate conversation. Agile Testing is something we tried at my company and I jumped on the opportunity to be the QA Manager and Tester. This concept is exactly how I like to work: fast and directly with everyone involved. Testing starts in the Dev environment alongside the developers, BA’s, Clients… and you prepare regression scripts to be run in the QA environment for that particular iteration. Bugs are fixed immediately. Unfortunately, our efforts as a whole were not seen as a success. This was more to do with the political barriers (SDLC) and fear. IMO, the QA portion was a huge success. Although we will not likely use Agile for development in the future, I will use many of its processes for QA.
    The concept of uTest is a great one. I havent done a lot of testing in uTest but I have done enough to know that this is a great idea and the fast-paced exploratory-type testing is exactly what I excel at.
    One interesting note: My Management thinks very very highly of Mr Bach and is always sending Bach-quotes out to everyone. I cant wait to send this post to him and have a real discussion on it. Hmmmmmm, maybe I should sign as “anonymous”…naahhh… I’m not scared. (Hi Paul)
    ~ Keri

  4. says

    On James’ comments on numbers and “empty” stories: some people believe that James and I hate numbers. We don’t. We love them, just as we love children and animals. And, therefore, we hate to see them abused, put to evil purposes, enlisted unwillingly into someone else’s army.

    On Not-a-Fan’s comment.: Notice that Not-a-Fan, with some potentially valid criticisms, is too shy to sign. Note also that James, with reasoned responses to those potentially valid criticisms, is brave enough to sign. What’s up with that?

    I’ve noticed this pattern in my own blog, too. For example, note the reaction from Anonymous here: http://www.developsense.com/2009/09/testing-checking-and-changing-language.html

    To me, this is an important issue for the testing community. Why be afraid to speak your truth? And having done so, why be reluctant to identify yourself? Anonymity would make sense if, say, you were making some kind of argument that is directly at odds with your organization’s stated policy, or that threatens some value other than your reputation. In fact, a reasoned criticism threatens neither, and neither “Not-A-Fan” (in James’ case) or “Anonymous” (in mine) have said anything that appears to fall into those categories.

    The only thing that anonymity affords here is the opportunity to say “insane blathering” without having to take responsibility for saying it. That’s not being a critic; that’s being a wimp.

    —Michael B. (brave enough to sign)

  5. says

    That’s a valid point. We have a mantra at uTest: “numbers can be tortured until they tell you whatever you want to hear”. Said differently, numbers — absent proper context — can be dangerous. We’re a numbers-driven company in nearly all of our activities. But incomplete, misdirected or meaningless numbers (eg: devoid of context) give false confidence and contribute to poor decision making.

    In my experience, this extends beyond the world of testing. It’s a truth that applies to most (if not all) disciplines — whether it’s marketing, product, finance or other.

    -Matt J.

  6. says

    I want to disagree with myself. This has been bugging me…

    I wrote: “…an empty *story* about testing couched in numbers to make it look rational”

    It was NOT an empty story. It was a story justified using empty numbers. The difference is important. Actually it’s one of the reasons these numbers continue to be used in our business.

    For instance, you might feel good. You might say “I feel like an 8 on a scale of 1 to 10.” The “8″ represents “good”. The eight does not say anything that “good” does not already say, even though it looks more objective and scientific. The selection of the number 8 is subject to all sorts of unconscious biases.

    The thing about numbers is that they are easily separated from their context. I think that’s one reason people like them. “We have 94% coverage” substitutes for a conversation about what we tested. But as a man who loves mathematics, I cannot in good conscience allow that to happen to my numbers. Therefore, when I use numbers, I keep the context close by. And I don’t use numbers when they contribute nothing material to my point.

    In the example I gave, I was not making up the need for testing, I was making up numbers that communicated nothing more than “I think we need to test this thing, here, and I think that will take some time.” These days, I forego the numbers and have the conversation.

    – James

  7. says

    To Not-A-Fan:

    Why does uTest allow anonymous content-free comments? That’s the better question.

    Of course there is context missing. That doesn’t make it insane blathering, it makes it ordinary human story-telling just like everyone gets to do, including me. I’m asking the reader to accept, based on my long-established credibility, that the essence of the story is as I say it is. I understand that an appropriately skeptical reader may not accept it. That’s perfectly okay. By all means, raise questions about it. That’s fair. But are these serious questions? Do you really think I “don’t know how companies work”, for instance? REALLY? That doesn’t seem over-the-top to you? You find it hard to believe that a test manager in a large company could have his job outsourced by someone many levels up who never observed his team in action, but you find it easy to believe that I, one of the better known consulting software testers in the world, don’t know how companies work?

    My brother just wrote a thoughtful blog entry about his experience. He and I wrote an article about it in STP magazine, as well, and he gave a talk about it at two conference this year. We have put the story out there to be a part of the Great Conversation of testing. If you’d like to criticize it, why not come out, show your face and reveal your good name– like we do– so that YOUR credibility contributes to the points YOU would like to make.

    Yes, it is a bit dangerous. I might, after all, ridicule you or seem to willfully misunderstand you. Just as you are doing to me. But as you see, I’ve been ridiculed through my career, yet still– the people I respect continue to respect me. Perhaps the same will be true for you.

    Come out. Grow up. Be a part of the debate.

    – James

  8. Not-a-fan says

    This in insane blathering. Why is uTest posting this? I’m sure there’s plenty of context here that’s missing. For instance, maybe your brother sucks and built a terrible team. Maybe your brother is unable to influence. Maybe you don’t understand how companies work.

  9. says

    As a professional Consultant I agree with you. I stop getting certifications about 5 years ago and never look back. Thanks God I have a job and company I work now (Comcast) know I’m the best tester they have! BTW, I also stop presenting meaningless matrix and reports 4 years ago!!


Leave a Reply

Your email address will not be published. Required fields are marked *