Software Testing Blog

In the first part of our interview with Michael Bolton, we grilled him on the emergence of the Weekend Testers, sensible metrics, Michael Bolton the pop star and a bunch of other topics. In part “deux” of our interview, we tackle the necessity of tester passion, how emotions affect testing, and the greatest threats to the profession. Check back tomorrow for the final segment.

uTest: There’s a lot of passion amongst testing thought leaders about the best way to test, or the best way to manage or train testers.  Often that passion overflows into heated debates.  How can this passion best be channeled to improve the state of testing?

MB: First of all, we should welcome debate, and we should welcome skilled argumentation as part of the art of construction and practice of persuasion. I’ve found, though, that it helps to remember that we’re exploring and challenging ideas. That means it’s good not to get too personally invested in certain ideas, because we’re always learning more, and because changes in context can mean big changes in what needs to be done.

That said, there are some ideas that seem robust for me. I believe that it’s unethical to dumb down people or the work that they do. I believe that we should focus our craft on learning, and learning how to learn rapidly. How can we improve the state of testing? By recognizing that software development is a web of people who are related in service to each other. That means putting people and social issues first. Get that right, and everything else will follow.

Suggestions are cool.  Standards are something else.  No group should be dictating to other people how they must test unless there are compelling human health and safety reasons for it. Do you really believe that the standards people know anything useful about your business? That the force of government-supported regulation, created by busybodies, should weigh on how you do your daily work? And if your answer is No, what are you going to do to get it stopped?

I talked to one fellow at a conference who said that managers would have no way to filter candidates without certifications.  “No way?” I asked  “Really?”  “Well, what other way is there?” he asked. “What other ways can you think of?” I asked him. He was stumped.  He didn’t seem to know about personal references, well-crafted resumes, networking, interviewing, auditioning, hiring from within and training people.  He did know about those things, but he was so fixated on certification that he couldn’t come up with a single one of them on the spot. Now:  do you really want someone like that, with that kind of limited mindset, setting the standards for how you test your product? Big money and people’s livelihoods are riding on your answer.

uTest: We sat in on your STPCon presentation in October, where you said “Pay attention to your emotions while testing. If something strikes you as being off, then chances are it probably is.” How did you come up with this piece of advice? And can you give testers a brief real-world example?

MB: I fly a lot, and so I often have to book travel using buggy airline Web sites. A couple of years ago, I had to book a flight to Europe. In the course of five minutes, I went through surprise (the site kept revising the dates I had chosen), confusion (why was it doing that?), frustration (the airline’s own Web site didn’t offer flights directly from Toronto to Amsterdam, even I knew though travel agency sites and personal experience that there were two such flights a day), impatience (why is this so hard and taking so long, for crying out loud?) and annoyance (I don’t liked being stymied and having my time wasted). I wrote an article about the experience; you can find it here.  

We all want to do testing better and faster, right? Tests of the business rules and the calculations are really important—no question about that—but emotional responses are important too, and they have the advantage of being immediate, both in the “right now” sense and the “no intermediary” sense; they’re direct experience.  Our customers will have emotional responses to their experiences too, and you never get a second chance to make a first impression.

In the Rapid Testing course that James Bach and I write and teach, we say that a bug is “something that bugs somebody who matters”.  People, including testers, are bugged by stuff that doesn’t work.  Being bugged is a feeling that starts with an emotional reaction.  The feeling is a signal.  It’s data, but not information; you still need to interpret the signal’s meaning and significance.  So instead of saying “If something strikes you as being off, then chances are it probably is,” I’d prefer not to be too certain about that.  Look into what you’re feeling, for sure, but consider the possibility that something else is the source of the feeling.  Maybe what you’re seeing is okay, but you need to learn something about the business rules.  Maybe it’s not consistent with an older version of the product, but the new way of doing things is really better and you’re just not used to it. Maybe it feels like a big deal for you, but it’s okay for your client or your end users.

We teach people to pay attention not only to the product, but also to their testing. If you’re bored while testing, use that feeling as a trigger heuristic to prompt a question:  why am I bored? Is it because my testing isn’t revealing anything new? Is it because I’m doing repetitive and un-engaging work that I could delegate to a machine? Is it because this is useless work that I shouldn’t bother with at all? Am I missing the point or paying attention to the wrong things?  If you’re uncomfortable for some reason—if you feel like you’re being pressured, for example—there’s a possibility that you’ll be distracted, or that you’ll rush, or that you’ll be biased towards noticing some kinds of problem at the expense of others. But maybe the pressure is telling you something about the significance of risk, or maybe it’ll remind you to work rapidly, or something else that’s good in your context.

In Jerry Weinberg’s Quality Software Management Vol. 1, Systems Thinking, he points out that decisions about quality are always political and emotional, but people want to appear to be rational. In fact, that in itself causes an emotional reaction in people who want to preserve their own illusions that they’re rational. At a conference recently, a tester approached me and said that she had been intrigued by the role of emotion in software development. She mentioned this to her boss. He replied, “There’s no room for emotion in software development.”  She said, “Really?  No room at all?” He got very agitated, and repeated, shouting this time, “THERE’S NO ROOM FOR EMOTION IN SOFTWARE DEVELOPMENT!” She told me he was quite serious. Naturally, we laughed about that. Our emotional reaction was a pointer to the incongruence between what he was claiming and how he was feeling.

uTest: What are the greatest threats right now to the software testing discipline?  What are the greatest hopes for a brighter testing future?

MB: The greatest threat at the moment, I think, is a systemic set of misconceptions about testing, especially at the middle management level. We don’t do quality assurance; people with the power to make decisions about the nature of the product and the project do that. We call those people managers. Software testing is not like manufacturing quality assurance, where we’re trying to make a zillion identical copies of something, and then checking to make sure that each one is just like the prototype. Every software development project is an attempt to build something new—otherwise we’d use a product that had been built already. So software testing has more in common with design quality assurance, where we’re trying to figure out the relationships between a product and its stakeholders, and trying to understand the product as we’re designing and building the first instance of it. But even then, we don’t assure quality; we question it on behalf of our clients. The people who are building and managing the product assure quality.

When we model software testing on manufacturing quality assurance, we emphasize checking at the expense of testing.  I’ve written a lot on my blog about the distinction between the two since August of 2009.  (You can see it at http://www.developsense.com/blog.shtml). Checking is important, but it’s mostly a confirmatory activity, verifying what we knew to be true before, or what we hope is still true, making sure that we’re getting the right answers. Testing is a bigger deal, focused on finding new information, identifying new risks, asking new questions.

Lots of people—managers, programmers, and even many testers—conflate testing and checking. Again, checking is really important, and it’s important to do it skillfully.  In particular, I think it’s important to automate it skillfully.  But when people confuse testing and checking, they don’t think clearly about cost, value and risk, and they tend to dumb testing down.  That’s a big problem.

uTest: Looking back over your testing career, you’ve seen a lot of changes to tools and techniques and trends.  What do you think software testing will look like in 2020?

MB: 2020 the year, or 20/20 the vision? I predict that, in 2020, there will have been a lot of changes to tools, techniques, and trends, and that someone will ask a tester to predict what testing will be like in 2030.

At this point in history, no sensible person should make specific predictions about technology beyond dinner time; read The Black Swan for more about that. But as for testing, the fundamentals will still be the same. People will want stuff, and programmers and engineers will design it. There will be miscommunication, misunderstanding, unexpected incompatibilities, delays, mistakes, emergent properties. So we’ll still need to investigate requirements and designs and products and problems skillfully.

It’s like writing. After a few thousand years of development on paper, we’ve got people writing for print and online media and television and radio. People’s work can be prepared and consumed anywhere in the world, 24/7, in any number of languages. But we still have people to manage the writers, and we still have people to help the writers—editors and research assistants and designers and so forth.

Editor’s note: Check back in tomorrow for Part III.