Software Testing Blog

Yesterday we posted Part 1 of our interview with James Bach, where he discussed tester certifications, faking test projects, his latest book and wide range of other topics (including life as a freelance sentry in a parallel universe). Today, for Part 2, we discuss tips for automated checking, what makes a good tester a great tester, his flying lessons and much more. Enjoy!

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.