As described on their award page, this honor will “recognize a person in the software testing and quality community, who inspires others and dedicates their career to industry advancement.” The organizers were looking for someone who has dedicated their career to the betterment of software testing and quality; who has shown exceptional leadership and who has educated, promoted and published on behalf of the industry. In other words, a software testing luminary.

With that type of criteria in mind, we’re not surprised to see Cem Kaner, James Bach and Jerry Weinberg as this year’s finalists. You may know Kaner and Bach from our recent Testing the Limits interviews (Jerry, if you’re reading this, we’d love to have you as a guest as well). But in case you’re unfamiliar with these testing giants, here are clips from their award bios:

Cem Kaner
The focus of Cem Kaner’s career is the satisfaction and safety of software customers, users and developers. His work is multidisciplinary—testing is one of the disciplines.

Kaner sees much of software engineering as applied social science. “We create software for the benefit of people. The essential measure of the value of that software is how well it works for the people it’s supposed to serve.” As he sees it, testing is more importantly about assessing the worth of a product or service than its conformance to documentation. To advance in the profession, Kaner argues that testers must develop and apply technological and cognitive skills, rather than relying on routine processes. He coined the term “exploratory testing” in the 1980’s to describe cognitively-engaged testing and distinguish it from scripted approaches. Continue reading….

James Bach
If Devil’s Advocate were a sport, James Bach would play it professionally. You’d see him on TV promoting Magnetic Analysis Corporation’s Finned Tube Test Coils (“Flux leakage analysis done RIGHT!”). Alas there is no such sport, so he’s a consulting software tester instead.

James owns and operates Satisfice, Inc. He’s a founder and leading voice in the Context-Driven school of testing (one of our industry’s several prominently competing communities of practice). He’s also a founding member of the Association for Software Testing. He has written many articles, co-authored “Lessons Learned in Software Testing”, and wrote “Secrets of a Buccaneer-Scholar”, a book about technical self-education. Continue reading….

Gerald Weinberg
Gerald (Jerry) Weinberg has always loved to create computer systems and programs. He’s a programmer through and through. And he has never stopped programming; he’s still at it. When one domain has been exhausted of interesting (i.e. hard) problems, he’s moved on to another until he reached domains where the problems are very hard, very compelling. There he continues to do requirements, analysis, design, code, and test to this day. Those still very interesting problem domains are himself, others, and organizations.

Jerry is author or co-author of several hundred articles and more than 30 books. His earliest published work was on operating systems and programming languages, but the 1971 publication of The Psychology of Computer Programming is considered by many the beginning of the study of software engineering as human behavior. Continue reading….

The polls will be open until September 2nd, so make sure you cast your vote before then. The winner will be announced and presented at the Software Test Professionals Conference & Expo 2010 in Las Vegas, NV October 19th – 21st.

Good luck to all the candidates!

Much like our Top 20 Software Testing Tweeps post, we need your help in letting us know if we’ve accidentally missed any good ones. Here they are in order of occurrence:

1. QUEST-Quality Engineered Software & Testing Conference (Apr 19-23, 2010: Dallas, TX)
2. Rapid Software Testing-By DevelopSense (Jul 5-7, 2010: Amsterdam, NL)
3. STANZ-Software Testing Australia/New Zealand (Aug 23-24: NZ & Aug 26-27: AU)
   
4. CAST-Conference of the Association for Software Testing (Aug 2-4, 2010: Grand Rapids, MI)
5. STAREAST (passed) & STARWEST-Software Testing Analysis & Review (Sept 26-Oct 1, 2010: San Diego, CA)
6. iqnite events-Next one in UK-formerly Software & Systems Quality (Oct 4, 2010: London, UK)
7. STPCon-Software Test Professionals Conference (Oct 19-21, 2010: Las Vegas, NV)
8. GTAC-Google Test Automation Conference (Oct 28-29, 2010: Hyderabad, India)
9. Expo: QA (Nov 16-18, 2010: Madrid, Spain)
10. EuroSTAR (Nov 29-Dec 2, 2010: Copenhagen, Denmark)

Have we omitted any noteworthy testing conferences you’ve recently attended? Please add your recommendations in the comments and they’ll be placed in the running to join the top events list. Maybe we can make this list a Top 15!

UPDATE: So far, some really great recommendations from our community include O’Reilly Velocity, Bangalore Workshop on Software Testing and VISTACON 2010 (the first Vietnam International Software Testing & Automation Conference).

Conversely, this recent article from The Economist could just as well be about today’s software companies, but it isn’t. It is about Toyota’s recent problems.

Like everyone else, the author wants to know how the auto giant could so quickly lose its reputation for safety and quality (things that can happen to ANY company if they are not careful). The culprit? You guessed it: software bugs.

Instead (of trying to keep pace with competitors), two recent trends, both software related, hint at the reason behind Toyota’s unexpected decline. One is the shortening of product-development cycles generally in the car industry. These are down from a typical four or five years to little more than 15 months, thanks to computer-aided design and manufacturing, and the virtual simulation of the resulting products. To save money and time, Toyota has even dispensed on occasion with building test “mules” and other engineering prototypes.

The other trend is the wholesale replacement of mechanical components with electronic controls. It started with ignition systems, then spread to air-conditioning, cruise-control, engine-management, throttle linkages, transmissions, and now the steering and braking systems. Drive-by-wire is not cheap, but it reduces the number of components needed to do the job. It also allows them to do extra things as well as to compensate for wear and changes in driving style and road conditions.

But software is not hardware, and software “engineers”, despite their appropriation of the name, are a different breed from the sort that bash metal. Programming digital controllers is not one of Toyota’s core competences. Even with the most diligent of testing, bugs will always find their way into software. Right now, it seems Toyota is learning that lesson the hard way.

But Toyota isn’t the only one learning lessons the hard way. As it turns out, the NHTSA does not have one single software engineer on the payroll to analyze the Toyota situation properly. CarConnection.com puts this into context:

If it cannot properly analyze those systems, or even understand at a deep-code level how they work, then the agency is useless at overseeing the entire “Safety” part of its mandate.

The agency has an annual budget of more than $800 million, and it employs 635 thousands of people. That not a single one of them is an EE or software engineer borders on the criminally insane.

So what have we learned from all this? Basically, that it’s okay to launch on extremely short development cycles if – and only if – the product has been tested extensively beforehand. Also, since there is no safety net, you are on your own when it comes to ensuring a quality. No one else can do it for you.

*By the way, Mike Mullane – a terrific writer and lecturer -  was the keynote speaker at STPCon this past Fall.  For a primer on his career and general philosophy, you should watch his appearance on The Daily Show.

In the third and final part of the Michael Bolton trilogy, we cover advice for new testers, his hypothetical banishment from Software Land, the blogs he reads and more. Did you miss our earlier interviews? Here’s Part I and Part II.

uTest: Hypothetical: You’ve been banished from testing – nay, ALL software-related activities – for the rest of your days. What will you to earn a living?  What hobbies would you pick up to fill the intellectual void?

MB: Who knows?  For fun, I’d keep playing mandolin, probably. Teach, maybe. Write. I’ve worked in theatre stage management, been a book-keeper, tended bar, worked in a comedy club. In high school I worked in mail rooms during the summer. Whatever I’ve picked up in life, it was because something needed to be done and I was there to do it.  If it didn’t seem like much at first, I started to learn about it quickly. When you invest a little bit of effort to figure out your job, you learn how to makes it faster and better and more interesting. It turns into this great feedback loop. Any job can be more fun when you set out to master it.

uTest: Tell our testing community something about you that your most avid readers don’t know.

MB: While walking through the woods on an island near Vancouver recently, I found myself being quiet and brief, which I like from time to time. Practically nobody knows that.

Lots of people probably don’t know how much I’m eager to help people out. All of my work—courses, articles, conference presentations, this interview—comes with lifetime free technical support. Have a question? Just ask. I might not answer right away—supporting the family with paying work takes precedence over supporting the community—but I’ve never knowingly turned anybody down, so if I don’t answer right away, be persistent. James Bach makes the same offer, by the way. We’ve found that it’s a great way not only to help people, but also to explore problems and come up with solutions and learn things that can help our clients.

uTest: If you were talking to a newbie tester, what advice would you give him or her to set their professional journey off on the right foot?  How about for a 10-year veteran tester?

MB: To both: learn, and don’t stop learning. Practice in your work, but also on open source projects, and in your life generally. Study testing, but note that the great ideas are likely to come from outside the field—at least the narrow vision of the field that the process enthusiasts and “certificationists” present. Testing is a marvelously interdisciplinary craft.  One implication is that whatever you bring to the table from your life and your experience and your education can inform new ideas about testing.  Join or start communities of skilled testers in your area. Join online groups; I like the Software Testing Mailing List and the Software Testing Club particularly. Learn about your craft and build your reputation by asking questions and seeking answers. As with testing itself, the really excellent work starts with asking questions.

uTest: In your opinion, what characteristics, skills or experience separate a good tester from a great tester?  How about in a testing manager?

MB: My colleagues and I have been talking about skills of great testers lately.  (In term of public postings, James and Jon Bach have podcast a nice pair of conversations which you can find here. James also did a CodingQA podcast here. One key skill is the ability to identify context quickly, which helps to avoid all kinds of pathologies—wasting time, failing understand the mission, testing the wrong stuff, investigating unimportant risks, and so forth.  Get the context right, and you’ve got a better shot at doing excellent testing. We teach people to consider and ask questions about the test lab, the test team, the development model and the programmers, the requirements, and the mission to help with that.

Another skill—which we also teach—is something that James has named test framing, which is the same kind of thing in the other direction. It’s the high-level skill—a skill set, really—associated with linking the context, your motivation, the actions you perform, the observations you make, the inferences you draw, and the story you tell about all that stuff.  It’s the ability to follow and describe the threads of logic forward and backward, from the mission to the outcome of the test and back again. It’s the ability to competently address and answer the questions “How, specifically, does this test relate to your mission?” as you prepare to test and “How does the outcome fulfill your mission?” after you’re done.

We’ve seen some good testers whom we can’t quite call great, because they have a hard time with some aspect of that skill set. They seem to lose the ability to link premises and conclusions at some point.

I think there’s a strong relationship to the skills of a great test manager there. A great test manager needs to be able to do the same kind of framing with respect to the test project and the test strategy. A manager that can frame the project can negotiate for people and resources more effectively, can train and deploy her people more efficiently, can explain the work of her team, can defend herself skillfully, can adapt to rapidly changing circumstances. That’s really important, because most projects are messy and confusing. If they weren’t we wouldn’t need testers.

Here’s an example: Test managers get asked questions like “When is testing going to be done?” That question is hard for some test managers because it’s not the test manager’s job to make the decision. They feel pressured to provide an answer, but they can’t frame the problem. A great test manager can deal with that pressure. First: Problems in a product get introduced almost entirely by people other than the testers. Second: Neither those people nor the testers know what problems are there, or where they are. Third: Testing is done in cycles that reveal as-yet unknown information about problems that may or may not need to be fixed. Fourth: The decision to fix the problems depends on the programmers and on the product owner. Fifth: Testing isn’t done until the problems in the product are fixed to the product owner’s satisfaction.

If those things are true (and they are), the amount of investigation and reporting and bug verification testers have to do is unpredictable until you’ve actually tested the product, made decisions about what’s to be fixed, and fixed the problems. Well… it’s predictable, but the prediction doesn’t have any real validity, because there are too many variables that could throw the whole prediction off. “Accurate” predictions in this business come mostly self-fulfilling prophecy—and from ignoring changes in scope or staffing or schedules or budgets or tools that just happened to make the prediction come true. If you set only a couple of data points for your targets, all the other stuff can change without people noticing much.

So testing cycles can be scheduled, but testing cannot be done until the last problem deemed important by the product owner is fixed, by the programmers, to his satisfaction. The great test manager can help the product owner to identify and decide upon what information he needs to know about the product. His goal is to be satisfied in his technical awareness of the product—in relation to the business needs—such that it makes business sense to ship or deploy the product. The test manager and the product owner collaborate in identifying the testing tasks that might help in providing that information.  That turns into the mission for that cycle of testing, test manager and her team can then set about fulfilling the mission—and adapting it when the product owner changes his mind, or when other things don’t go according to plan.  But in the end, the question of when testing will be done depends very little upon the test manager, and far more upon the programmers, the product owner, and a bunch of information that is presently unknown.  A great test manager can explain that in a way that satisfies the product owner, without being trapped into an unachievable commitment.

uTest: At STPCon, you also said that “Testing can be assisted by automation, but automation is not the arbiter of value. We are.” Does this mean that testing will never be replaced by machines? And does this include cyborgs?

MB: Automation is a tool; it’s a means to an end.  It’s a medium, in the McLuhan sense—it extends or enhances or enables or accelerates or intensifies some human capability, but it doesn’t replace that capability. Writing and email and wikis extend our ability to communicate, but they don’t replace talking.

Machines can’t make decisions about what’s best for us, because they’re only extensions of some human faculty. Value is about what humans want, what they like, what they’re willing to pay or do. People don’t even like other people making those decisions for them.

uTest: You’re an active blogger, teacher and speaker – do you enjoy these things more than testing itself?

MB: That’s a little like asking about someone’s favorite Beatle album. I love testing, I love teaching it, I love talking about it. It’s a feedback loop. I wouldn’t be able to do the other stuff if I didn’t test, and the more testing and research about testing that I do, the more I find that’s interesting to talk about. When we teach Rapid Testing, we offer a free day of hands-on testing and coaching. We offer that separately too, and it’s always a source of great learning and great fun.  I’d be delighted for clients to involve me more directly more often.

uTest: Who do you read in the world of testing (blogs, sites, journalists, et al)?

MB: I have a very expansive view of the world of testing. To me, the world of testing is the world of human knowledge and inquiry, of critical thinking, of science. That includes software-related stuff, but it includes a ton of other disciplines too:  medicine (Jerome Groopman, Atul Gawande), aviation (Patrick Smith on Salon.com), education and other social issues (David Cayley), advertising and marketing (Terry O’Reilly), economics (James Surowiecki, John Cassidy, Herbert Simon), security (Bruce Schneier), risk (Nassim Taleb).  All those folks and their specialties are Googleable. I really like Malcolm Gladwell’s writing, too; he’s harder to pigeonhole, but he’s good at finding intriguing sides to the basic story. I first encountered most of the writers via The New Yorker and most of the broadcasters via CBC Radio. I like starting with very accessible stuff, and then if something interests me, I follow the lines of references, bibliographies and notes.

A couple of testing books that came out recently have few notes and no bibliography. That’s like publishing malpractice, so far as I’m concerned. You’ll notice that I’ve mentioned a lot of names already. That’s specifically so that people who are reading this interview can check out the stuff that’s fascinated me and helped to spark new ideas. I hope it helps to inspire people to talk about their own discoveries. Thank you for this opportunity to chat about mine!

Editor’s note: We hope you’ve enjoyed our three-part chat with Michael Bolton. If there’s a question you wanted us to ask that we didn’t get to, you can ask Michael directly at: michael@developsense.com.

Have suggestions for our next interview? Send those along as well!

In the first part of our interview with Michael Bolton, we grilled him on the emergence of the Weekend Testers, sensible metrics, Michael Bolton the pop star and a bunch of other topics. In part “deux” of our interview, we tackle the necessity of tester passion, how emotions affect testing, and the greatest threats to the profession. Check back tomorrow for the final segment.

uTest: There’s a lot of passion amongst testing thought leaders about the best way to test, or the best way to manage or train testers.  Often that passion overflows into heated debates.  How can this passion best be channeled to improve the state of testing?

MB: First of all, we should welcome debate, and we should welcome skilled argumentation as part of the art of construction and practice of persuasion. I’ve found, though, that it helps to remember that we’re exploring and challenging ideas. That means it’s good not to get too personally invested in certain ideas, because we’re always learning more, and because changes in context can mean big changes in what needs to be done.

That said, there are some ideas that seem robust for me. I believe that it’s unethical to dumb down people or the work that they do. I believe that we should focus our craft on learning, and learning how to learn rapidly. How can we improve the state of testing? By recognizing that software development is a web of people who are related in service to each other. That means putting people and social issues first. Get that right, and everything else will follow.

Suggestions are cool.  Standards are something else.  No group should be dictating to other people how they must test unless there are compelling human health and safety reasons for it. Do you really believe that the standards people know anything useful about your business? That the force of government-supported regulation, created by busybodies, should weigh on how you do your daily work? And if your answer is No, what are you going to do to get it stopped?

I talked to one fellow at a conference who said that managers would have no way to filter candidates without certifications.  “No way?” I asked  “Really?”  “Well, what other way is there?” he asked. “What other ways can you think of?” I asked him. He was stumped.  He didn’t seem to know about personal references, well-crafted resumes, networking, interviewing, auditioning, hiring from within and training people.  He did know about those things, but he was so fixated on certification that he couldn’t come up with a single one of them on the spot. Now:  do you really want someone like that, with that kind of limited mindset, setting the standards for how you test your product? Big money and people’s livelihoods are riding on your answer.

uTest: We sat in on your STPCon presentation in October, where you said “Pay attention to your emotions while testing. If something strikes you as being off, then chances are it probably is.” How did you come up with this piece of advice? And can you give testers a brief real-world example?

MB: I fly a lot, and so I often have to book travel using buggy airline Web sites. A couple of years ago, I had to book a flight to Europe. In the course of five minutes, I went through surprise (the site kept revising the dates I had chosen), confusion (why was it doing that?), frustration (the airline’s own Web site didn’t offer flights directly from Toronto to Amsterdam, even I knew though travel agency sites and personal experience that there were two such flights a day), impatience (why is this so hard and taking so long, for crying out loud?) and annoyance (I don’t liked being stymied and having my time wasted). I wrote an article about the experience; you can find it here.  

We all want to do testing better and faster, right? Tests of the business rules and the calculations are really important—no question about that—but emotional responses are important too, and they have the advantage of being immediate, both in the “right now” sense and the “no intermediary” sense; they’re direct experience.  Our customers will have emotional responses to their experiences too, and you never get a second chance to make a first impression.

In the Rapid Testing course that James Bach and I write and teach, we say that a bug is “something that bugs somebody who matters”.  People, including testers, are bugged by stuff that doesn’t work.  Being bugged is a feeling that starts with an emotional reaction.  The feeling is a signal.  It’s data, but not information; you still need to interpret the signal’s meaning and significance.  So instead of saying “If something strikes you as being off, then chances are it probably is,” I’d prefer not to be too certain about that.  Look into what you’re feeling, for sure, but consider the possibility that something else is the source of the feeling.  Maybe what you’re seeing is okay, but you need to learn something about the business rules.  Maybe it’s not consistent with an older version of the product, but the new way of doing things is really better and you’re just not used to it. Maybe it feels like a big deal for you, but it’s okay for your client or your end users.

We teach people to pay attention not only to the product, but also to their testing. If you’re bored while testing, use that feeling as a trigger heuristic to prompt a question:  why am I bored? Is it because my testing isn’t revealing anything new? Is it because I’m doing repetitive and un-engaging work that I could delegate to a machine? Is it because this is useless work that I shouldn’t bother with at all? Am I missing the point or paying attention to the wrong things?  If you’re uncomfortable for some reason—if you feel like you’re being pressured, for example—there’s a possibility that you’ll be distracted, or that you’ll rush, or that you’ll be biased towards noticing some kinds of problem at the expense of others. But maybe the pressure is telling you something about the significance of risk, or maybe it’ll remind you to work rapidly, or something else that’s good in your context.

In Jerry Weinberg’s Quality Software Management Vol. 1, Systems Thinking, he points out that decisions about quality are always political and emotional, but people want to appear to be rational. In fact, that in itself causes an emotional reaction in people who want to preserve their own illusions that they’re rational. At a conference recently, a tester approached me and said that she had been intrigued by the role of emotion in software development. She mentioned this to her boss. He replied, “There’s no room for emotion in software development.”  She said, “Really?  No room at all?” He got very agitated, and repeated, shouting this time, “THERE’S NO ROOM FOR EMOTION IN SOFTWARE DEVELOPMENT!” She told me he was quite serious. Naturally, we laughed about that. Our emotional reaction was a pointer to the incongruence between what he was claiming and how he was feeling.

uTest: What are the greatest threats right now to the software testing discipline?  What are the greatest hopes for a brighter testing future?

MB: The greatest threat at the moment, I think, is a systemic set of misconceptions about testing, especially at the middle management level. We don’t do quality assurance; people with the power to make decisions about the nature of the product and the project do that. We call those people managers. Software testing is not like manufacturing quality assurance, where we’re trying to make a zillion identical copies of something, and then checking to make sure that each one is just like the prototype. Every software development project is an attempt to build something new—otherwise we’d use a product that had been built already. So software testing has more in common with design quality assurance, where we’re trying to figure out the relationships between a product and its stakeholders, and trying to understand the product as we’re designing and building the first instance of it. But even then, we don’t assure quality; we question it on behalf of our clients. The people who are building and managing the product assure quality.

When we model software testing on manufacturing quality assurance, we emphasize checking at the expense of testing.  I’ve written a lot on my blog about the distinction between the two since August of 2009.  (You can see it at http://www.developsense.com/blog.shtml). Checking is important, but it’s mostly a confirmatory activity, verifying what we knew to be true before, or what we hope is still true, making sure that we’re getting the right answers. Testing is a bigger deal, focused on finding new information, identifying new risks, asking new questions.

Lots of people—managers, programmers, and even many testers—conflate testing and checking. Again, checking is really important, and it’s important to do it skillfully.  In particular, I think it’s important to automate it skillfully.  But when people confuse testing and checking, they don’t think clearly about cost, value and risk, and they tend to dumb testing down.  That’s a big problem.

uTest: Looking back over your testing career, you’ve seen a lot of changes to tools and techniques and trends.  What do you think software testing will look like in 2020?

MB: 2020 the year, or 20/20 the vision? I predict that, in 2020, there will have been a lot of changes to tools, techniques, and trends, and that someone will ask a tester to predict what testing will be like in 2030.

At this point in history, no sensible person should make specific predictions about technology beyond dinner time; read The Black Swan for more about that. But as for testing, the fundamentals will still be the same. People will want stuff, and programmers and engineers will design it. There will be miscommunication, misunderstanding, unexpected incompatibilities, delays, mistakes, emergent properties. So we’ll still need to investigate requirements and designs and products and problems skillfully.

It’s like writing. After a few thousand years of development on paper, we’ve got people writing for print and online media and television and radio. People’s work can be prepared and consumed anywhere in the world, 24/7, in any number of languages. But we still have people to manage the writers, and we still have people to help the writers—editors and research assistants and designers and so forth.

Editor’s note: Check back in tomorrow for Part III.

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

JB: Years ago, Rational hired me to advise them about test processes, and for that reason I took the Rational Unified Process class. I had been working with the RUP team, and I wanted to see what the Rational instructor was saying about RUP. At lunch on the first day, I asked him if it was okay that I was asking challenging questions. He said that was fine. But at lunch on the second day he kicked me out of the class!

Most instructors don’t like me in their classes. So, when you say I’m forced to choose, it’s a question of which class I would prefer to disrupt. (Unless I have the choice of sitting in one of Cem Kaner’s classes at Florida Tech. I would definitely prefer that. Last time I took a class from Cem, I had to play solitaire furiously in the back of the room. It was the only way to stop myself from jumping up and shouting “Yes!” every three minutes.)

I would prefer to disrupt an ISTQB class.

uTest: A few months back you presented “How to Fake a Test Project” at STPCon, where you said “testers may accidentally find bugs because they don’t follow the test scripts precisely.” Are testing managers starting to catch on to this irony of such “best practices”, or are most of them still oblivious to this?

I don’t know. I think most test managers don’t think much about it. As I said in my talk, testing is easy to fake– *even by accident*– because testing is so intangible.

JB: Be honest, you’ve faked a few test projects in your day, right? C’mon you can tell us.

Yes. I remember walking over to my manager’s office at Borland– by this time I’d been doing test management for five years– carrying an estimate of how much testing was required for my project. The estimate was couched partly in terms of “test cases required.” But just as I reached his office, I realized that the numbers meant nothing.

Until that moment, I thought they meant something. Well, they did mean something, just not anything I could defend or explain. Suddenly it dawned on me that I was peddling an empty *story* about testing couched in numbers to make it look rational.

As I write this, that incident happened 17 years, 5 months, and 1 day ago. Since advising my boss that he should ignore my test case estimates at that meeting, I have never used test case counts as a metric on a test project. But that doesn’t explain why, for the 5 years, 1 month, and 21 days before that, I *did* use test case counts.

Of course, the explanation is that I hadn’t much thought about it. I went along with the convention. So, I can forgive people who fake test projects by accident. When we’re young we make mistakes. But we also learn and grow.

uTest: In your new book, Secrets of a Buccaneer-Scholar, you advocate alternatives to traditional schooling and certification – not just for testers, but for people in all professions. It seems to have worked well for you, but why are so many people still reluctant to ignore the certificates and degrees and just start testing? What’s holding them back?

JB: Fear and indoctrination. I recently argued with a fellow about why he was going through a certificate program that bored him and insulted him, instead of quitting that school and being a tester. He claimed that no one would give him a job without the certification. Well, he emailed me a few days ago and told me he just got a job, and that I was right.

Well of course! I’ve been a hiring manager, and the idea that hiring managers are zombies who only look at paper credentials is wrong. SOME managers do that, but you don’t really want to work for someone like, do you?

Seems to me uTest is a mechanism for building your reputation as a rapid tester. I encourage new testers to try it for exactly that reason.

uTest: What’s the most promising trend/development in testing?  And what’s the most dangerous or discouraging trend you’ve seen?

JB: I think the most promising trend is public peer conferences and mentoring. Things like the Weekend Testers, for instance, or the Rat Pack. Some of these people are inspired by guys like me, perhaps, but they are not followers. They are taking it upon themselves to re-invent testing by experiencing and examining it intensively. I’m proud of anything I have done or can do to encourage this sort of creativity.

The most dangerous trend is certification, which is systematically dumbing down our craft and preventing good people from getting work.

A more dangerous thing, which is not a trend but unfortunately a constant, is the astonishing incompetence of technology middle and upper management, who frequently demand stupid metrics and meaningless reports from the people who work for them, or who decide to outsource without having the first clue what they are giving up by dumping knowledgeable testers and sending critical work ten time zones away.

For some reason, test managers complain to me about this, but often don’t complain to their own bosses.

Last year, my brother created a world-class test team at LexisNexis, then someone many levels up from him decided to fire everyone and outsource. They never talked to my brother about this. Apparently they think good testing happens by magic. But here’s the thing, my brother decided that he was going to train the outsourcing company (who had claimed to know a lot about testing, but, surprise, didn’t actually know much at all) to do skilled exploratory testing. It worked. He’s pleased. And now his management at LexisNexis (former management, because they then fired him) must think outsourcing is a great strategy, having no idea that it works only because people like my brother have too much personal pride to let even senior management airheads experience a richly deserved failure.

uTest: We’re all familiar with your work, as well as that of your colleagues like Michael Bolton, Cem Kaner, Jon Bach and others. Are there any “great unknowns” out there in the testing blogosphere to whom we should be paying attention?  Anyone that should be ignored altogether as a dangerous thinker?

JB: I sometimes complain about people I think are dangerous. See my blog.

Basically, I hate bullies. I want a free and open craft. The most dangerous people are those whose personal ambition will mean discouraging innovation (perhaps through establishing some sort of baloney “testing standard” through the auspices of ISO), and encouraging the sort of test management practices that make software both expensive and sucky, while discouraging any smart ambitious person from entering the field in the first place.

As far as great unknowns, there are some. I think Lanette Creamer is someone to watch. Adam White, Ajay Balamurugadas, and Ben Yaroch, as well. There are the Weekend Tester guys, and a shadowy Context-Driven sub-community calling itself “The Rat Pack.” I come across testers, regularly, who are talented, but haven’t taken a step out into public discourse.

I am constantly recruiting for colleagues.

Be sure to check out Part 2 of our interview with James.

uTest:  We loved the SpeedGeeking session you led at STPCon, so we’re going to flip it on you – If you had just five minutes to teach, motivate or inspire the uTest audience about software testing, what would you say?
MH: Well, I’d start by asking the audience what they are doing today – what’s the greatest point or opportunity they feel – and asking what options they see to improve. Most of the time, I hear that testing is “too slow” or “the bottleneck” or something like that.

So I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting – it is very important the team stop the time tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the numbers for improvement. For example, many of the people I talk to can spend 80% of their time or more in meetings, working on documentation, working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from 80/20 to 60/40 will double the amount of time the team spends actually doing testing.

Another thing to look at is the amount of time spent trying to reproduce defects, document defects, file bug reports, “verify” fixes, and so on. We think of these activities as testing, and they can take a substantial chunk of that 20% – but they are really accidental. That’s not a testing bottleneck – it is a development bottleneck. If test can work with development to improve the quality of the software prior to code complete, that will improve the speed of the whole system. Realizing this, and having a little bit of data to “prove” it, can help the entire system improve.

So if I had five minutes, I would say start with measuring how you track your time, and ask yourself if this is the best use of your time and what can change. Sometimes, the big boss will say “no, we absolutely need you to fill out all seven pages of documentation per test run”, and you can say “ok.”  Six months from now, when someone asks why the big project is late, you can point out that the business made an explicit decision to pay the full price of defined process. You presented options and those were not accepted.

That won’t save this project — but it might save the next.  It also turns out that actually testing tends to be much more fulfilling than documentation and compliance activities. Who could have guessed?

Lots of contrasting opinions at last month’s STP Conference. While there were no fist fights (that we heard about anyway), what did you see as the most contentious issue? And where do you fall on this issue?

MH: I’m a little sad that we keep talking about best practices vs. context, scripting vs. exploratory. Of course we need a balanced breakfast of approaches to testing, and of course, “best practices” are a marketing term that do not exist in the engineering literature. Personally, I’m most embarrassed that you could get the crib notes from any conference from 2004 – or possibly 1999 – and see the same arguments.

Some of the problems are due to personalities. There are a few people who just don’t give past credit for ideas, or make wild claims without having actually done much software testing. Where do I fall on the issues? Well, let me first say, if anyone claims to be an expert on software testing, one place to start is to go to LinkedIn, look at what they actually claim to have done, and send out a few emails and try to verify those claims. If you can’t verify those claims, or realize they are written in a very specific way as to be non-falsifiable, well, that tells you a lot.

I was impressed that we managed to not fight about certification at the conference. There was no certification course before, after, or during, and we didn’t have to spend our time debating it’s merits or lack thereof – we mostly talked about, you know, actually testing and stuff. In that, I was pleased.

Congrats on getting your blog “Testing at the Edge of Chaos” exclusively featured on the STP Collaborative. What does it mean to you to test at the edge of chaos? What ideas are you most interested in getting across to the tester community?
MH: Uh, I think my blog has something to do with testing. And, um, like, something to with Chaos, or something.  Seriously, started blogging in 2001 or so to express myself and my ideas. The last iteration of that was my “Creative Chaos” Blog, where I talked covered creativity and innovation in the software process. “Testing at the Edge” is a little more tester-focused, with the goal of covering skills and innovation in the software testing space.

It seems to be that every year’s a new batch of graduates come out of MIT, Carnegie-Mellon, and the University of California at Berkely that know how to automate defined business processes – and take a look at testing and say “gee, there is a defined process, we should automate it.”

Four or five years later, they’ve learned a bit and turn around and say things like “gee, some of testing can be automated, but part of testing an investigative, feedback-oriented process.”  Then every May a new batch graduates and we start all over again.

I think “Creative Chaos” did a good job in reaching out to that audience, and with the publication of “Beautiful Testing”, we may be finally out-growing it. With the new blog “Testing at the Edge”, I hope to move into specific examples of good testing, how to get better at it, and how to set some appropriate boundaries around the testing activity – and discuss what those should be.

What’s the deal with your Testing Challenges? Are they still on-going, or have you simply run out of apps to test?
MH: I generally run test challenges in private, sometimes electronically, as part of a mentoring or training program. I do this both commercially and non-commercially, as part of my zero-profit “Miagi-Do School of Software Testing.” Recently, I’ve been asking people for what they want, and the idea of running test challenges publicly – and sharing the answers – keeps coming up. I ran one in October (link) that was well-recieved. Sure, I can do more of them if there is interest.

As I work full-time for software product company, I don’t see us running out of apps to test anytime soon. (Laughs)

A quick hypothetical: You’re banished from the software testing industry for five years. What do you do during that time? And don’t say developer.
MH: I’d probably challenge the authority of who it is that is trying to banish me! But I will take your question in the spirit you intended it, and answer what my next career choice would if, for some reason, I chose not to test. That would likely be writing about technology or business.

When you think about it, the investigative journalist shares a lot with the tester. Journalists are paid to look around, find something that seems correct on it’s surface but has a problem, and uncover information and evidence. They take careful notes, they work on projects that are different every time, think critically, and are guided by rules of thumb.

A surprising number of the leaders in the testing industry have an education of background in journalism – both Karen Johnson and Jonathan Bach were trained as journalists before they became testers, and Dr. Cem Kaner has a law degree as well as his PhD in Psychology.

What would the Matt Heusser from ten years ago think about today’s software testing landscape?
MH: Well let’s see … ten years ago Extreme Programming was a crazy idea that Kent Beck was trying at Chrysler, “agile” was spelled lower case and was an adjective that meant ‘bendy.’ The software development landscape was full of RUP and patterns and generalizations and abstractions, and testers were counting test cases. James Bach and Cem Kaner were using the term exploratory testing, but it was far from popular. Today, I think testers have more voices to listen to – the standard school, the context-driven school, and the Agile School all understand each other a bit better and have more clear comparisons. Also, test managers have more options, like the crowdsourced approach or perpetual beta.

Overall, there are more options, the profession is taken more seriously, and we are finally starting to evaluate ideas based on consequences and outcomes instead of ideology. It’s a great time to be a tester. I wouldn’t want to go back.

Check back on Monday for part two of our chat with Matt Heusser.  We’ll cover topics like what OS and browser HE uses, how mobile app testing differs from web or desktop testing, and whether great testers are born or trained.

This event gave us a chance to reconnect with our friends from STP Collaborative, as well as sit in on sessions from top-shelf testing thinkers like James Bach, Jon Bach, Michael Bolton and Matt Heusser.  We also got to connect with about a half dozen local uTesters who took us up on our invitation to attend the reception.

We had a few people from uTest in attendance on Thursday and they gave the presenters high marks overall.  The two that I heard the most comments on, however, were from James Bach and Michael Bolton.  Bach tackled the provocative subject of “How to Fake a Test Project.”  Bolton took on the topic of “Rapid Software Testing.”  We’ll see if we can get either of their presentations and share them here. Also of great interest was Friday morning’s SpeedGeeking session with Matt Heusser, which put the speakers on the spot, giving them only five minutes to get right to their point. Talk about fighting your natural instincts!

For anyone who attended, chime in and share your most or least favorite moments from STPCon?  What surprised you, frightened you, entertained you or just generally pissed you off? Sound off in the comments or drop us a line.

Meanwhile, the uTest paparazzi was present and ready to capture some of the scenes from the evening.

Matt Heusser and James Bach take a timeout from their testing debate for a quick photo

Matt Johnston and James Bach deep in discussion about the art & science of testing (and the difference between the two)

Jennifer Moebius and STP chief, Andy Muns take a break from the exhibits to pose for a pic

We will be co-hosting a free tester meetup with STP (Software Test & Performance) as part of the kickoff reception for their big event, STPCon 2009 at the Hyatt Regency Cambridge.  This meetup will be Wednesday, October 21 at 5:30pm.

Join us for a great evening of networking that will be held in the STPCon exhibits area. There, you’ll have the opportunity to connect with your peers, connect with execs from uTest and STP, discover new products and features and talk to the experts who created them.

Another great perk for attendees is that you’ll have the opportunity to discuss the latest and greatest trends with industry leaders such as James Bach and Michael Bolton.

If you’re around, it would be great to meet you in person!  To register, please visit: http://utest2009stpcon.eventbrite.com/.