Software Testing Blog

In part III of our of Testing the Limits interview with Cem Kaner, we discuss why “best practices” is merely a marketing tool; Silicon Valley of yesteryear; his upcoming CAST lecture on investment model and exploratory test automation; the blogs he reads and much more. To recap, here’s part I and part II.

uTest: What’s surprised you the most about the testing industry since you’ve been in the game?

Kaner: Context-dependence was a big surprise to me. It took me about ten years before I reluctantly accepted the idea that my favorite test techniques, attitudes, and life-cycle models were appropriate in situations similar to the ones where I developed my preferences, but not so appropriate in situations that were quite different.

It took me a long time to learn that developing software under a well-specified contract is a common and respectable activity that requires different tradeoffs from mass-market software sold to people who had no say in its design and buy only after it is ready to deliver and only if they like it. It took me a long time to sort out differences between scientific programming (where I started) and consumer software development. It took me even longer to see that testers working in an independent test lab operate under fundamentally different constraints from testers working inside the development company and provided different types of information. And it took well over a decade before I accepted the idea that two different development managers, running essentially the same project, could legitimately want different information from their testers and that it could make sense and be ethical for the two different test groups to structure their work differently, finding or being blind to or ignoring different classes of bugs, in order to satisfy the information needs of their key stakeholders.

The last major event in this chain happened 14 years and two books after I came to Silicon Valley. A client paid me to tour a lot of companies in California, Oregon and Washington. I gave a talk at each place, but I also talked with them about their business models, their testing challenges and methods.

Most of these were good companies with competent testing staff but they did things very differently from each other, often very differently from what I (in my ignorance) would have recommended, but in ways that addressed the risks they were trying to manage. It was already my practice to try to understand what worked at a client site, and why it worked, rather than to evaluate what they were doing against my prejudged ideas. But the diversity in this series of clients was overwhelming. It caused me to abandon many of my favorite ideas about development and testing—not as bad ideas, but as good ones only under suitable circumstances.

uTest: You’re shaping the minds of future testers… so how do you “future-proof” your teachings?  And what major changes do you think will impact software testing by the end of this decade?

Kaner: What I DON’T do is try to slow the field down.

I don’t pretend that there are One True Definitions for any testing terms, because different groups of testers see the craft differently and they use their language accordingly. If two testers have different ideas about the purpose and goals of testing, they are likely to have different meanings, or at least different nuances, in their definitions of “test case.” I don’t go to standards boards to try to legislate my favorite meaning as the official one. This effort to lock down a field that is still in motion, still finding itself, will primarily benefit people selling certification courses. In terms of helping their students prepare for the future, I think that this gives an illusion of certainty and uniformity to people who should be training to embrace questionability and diversity.

Similarly, I think the idea of “best practice” is a security blanket for babies or a marketing tool. In my experience, context matters a lot. A “practice” (a something that I use or do) can work very well under some circumstances and poorly under others. Teaching about “best practices” is another way to sabotage the critical thinking  of the students. As the context changes, people with this training have a poor foundation for adapting, or at least for adapting in any other way that paying a consultant to tell them what to think and do next.

As I see it, we test in order to find out useful information. We use technology (usually) to ask machines questions about how they work and humans questions about how the machines should work. We ask the questions in order to reveal information about the quality of what we are studying, and we usually do that because someone is paying us to get that information.

So, the “future-proof” skills and knowledge are the critical thinking skills, the questioning skills, and the understanding of full-lifecycle scientific method (including the qualitative research needed to develop ideas that lead to potentially-valuable hypotheses and the formal methods useful for focusing and assessing (typically refuting) those hypotheses.) The rest is going to change.

I can’t future-proof my ideas. That illusion is for the standardizers. Many of my ideas have changed enormously over the years. Take a look: http://www.kaner.com/pdfs/TheOngoingRevolution.pdf. Until I stop learning, my ideas will change even more.

uTest: Your latest presentation revolves around investment modeling and exploratory test automation. Give our readers a sneak preview of what to expect (also, where and when to expect it).

Kaner: This is a pair of talks at CAST (the conference of the Association for Software Testing). CAST is at Grand Rapids, Michigan from August 2-4. See http://conferences.associationforsoftwaretesting.org/CAST2010

The first talk is with Doug Hoffman on Exploratory Test Automation (ETA). Mainly, we will review a variety of examples of projects that used automated tests to learn new things about products rather than trying to confirm that old knowledge was still correct. Many of these involve automated creation of enormous sets of tests, scanning the program in ways that a human can’t, looking for potentially interesting results that would be worth a human’s time to investigate further. Several other people are having insights that a lot of (partially) automated testing that they do is not regression testing and is useful for learning new things. They’re presenting their ideas as ETA too, at CAST and at other conferences. We’re just adding our list to the mix and suggesting a framework for sorting out the differences and similarities among this large family of ideas.

My experience with teaching people about ETA in the past is that the examples require technical sophistication to understand (e.g., what does printer firmware really do and what are we trying to learn when we test it?). An hour-long talk is too short for some people. The good news is that at CAST (like at the LAWSTs), we are not limited by the time allotted for the talk. CAST has a breakout room. If people want to argue with us or ask detailed questions after the talk, we aren’t allowed to say, “I’m sorry, we only have 1 minute left to answer your question.” At CAST, we move to the breakout room and stay with the questioners until our discussion is done. Probably this hour-long presentation will take 3 to 4 hours (with questioning) before it is actually done. And hopefully, the people who stay with us to question and argue will walk away understanding what we’re trying to teach.

The second talk is my keynote, at the end of the conference. Building on what Doug and I presented, I present a case study, explaining how I use automated exploratory tests to assess software that implements investment models. Many people find discussions of investing easier to understand and more interesting. See http://adam.goucher.ca/?p=1492 for a reaction to a draft of the talk.

uTest: How do you effectively bridge the gap between academic theory and commercial practice?

Kaner: There are thousands of gaps, in all directions. We don’t ever bridge “the” gap. We bridge specific gaps, one at a time.

The first challenge is to think across paradigms, across ways of looking at the world. If I want to introduce my ideas on design to psychologists, I need to learn to think enough like the psychologists who I am trying to influence to make my presentation influential. I have to understand what questions they are trying to answer or what problems they are trying to solve, and then tie my ideas to their interests or needs. That takes work on my part.

Similarly, if I am trying to learn lessons from another field, I have to learn about that field. A new technique, a new idea, a new experimental method or a new insight (and accompanying wisdom) – these are rarely packaged in a self-contained way that lets me grab one and use it. What is this technique used for? Why? What does it replace? What makes it better? What are its limits?  To understand these things, I need to be able to understand the literature (books, articles, conference talks, demonstrations) that present the new knowledge. And all the time, I am asking, how does that apply to what I might be interested in? Again, that takes work on my part, a lot of it.

I host an annual meeting called the Workshop on Teaching Software Testing. This is one of the LAWST-style workshops (http://www.kaner.com/lawst.htm). We recently had our 9^th one (http://www.wtst.org/). The LAWSTs generally allow up to 20 people in the meeting. We schedule presentations, but we work through a presentation until we’re done with it. Thus, a 1-hour presentation (according to the schedule) might trigger one minute’s worth of discussion or a full day of discussion (we’ve gone to both extremes in the LAWSTs). The questions are often detailed. Sometimes there are debates and one participant will create slides to make a counter-presentation. In a meeting like this, 10 to 20 people work together to build bridges to each other’s ideas. At WTST, we bring together teachers—some academic, some commercial—and we share ideas on how to teach topics in testing.  This is my most effective vehicle for building bridges.

uTest: Name some other testing pundits, zealots, teachers and thought-leaders whom you respect. Who should we interview next month to share their wit and wisdom with the testing world?

Kaner: I’m surprised you haven’t talked with Elisabeth Hendrickson yet. And talk to Robert Austin.

Rapid fire Kaner Q&A:

• Last book read: Harry Dormash, Fire Your Stock Analyst. (This one was so good, I’m adopting it as a textbook in my investment modeling in software engineering course.)
• Last movie watched: Karate Kid
• Mobile device of choice: Amtrak
• Mac or PC: OK
• Techcrunch or Wired: Digg.com, Huffington Post, Automated Trader magazine, Bulletin of the ACM Special Interest Group on Computer Science Education, Law.com, Psychological Science, Bloomberg News, CNBC (especially Cramer) and Rachel Maddow. Ask me next year and the list will be different.
• Whiteboard or chalkboard: projector
• Best video game of all time: No idea. But my personal favorite was Wizardry.

Editor’s note: We hope you enjoyed our latest Testing the Limits interview. If you have a suggestion fir our next guest (or have a topic you want covered) just let us know.