*********

uTest: Let’s go back in time for a second: How did you get into the craft? What was the first application you tested? What was testing like back in the early 90s?

Simo: Providence. It was providence that got me into testing.

I was young, in love, and planning to get married.  I had been doing some part time database development work, but needed a full time job before the wedding.  I submitted letters and resumes to dozens of companies. I was willing to do almost anything that would pay the rent.  I lived in a city where the local job market was dominated by defense contractors. I quickly learned that many of them called nearly everyone who applied for anything in for an interview; so they could learn about people and add them to databases of potential hires for matching to work they did not yet have.  These companies would then present these people to the government as their available workforce when bidding on contracts. This made it frustrating for those of us looking for work. It often wasn’t clear, when going in for an interview, if it was for a real job or for a potential position that might come at some time in the future if that company were to be awarded a government contract.

I interviewed with the company for which my fiancé (now my wife of 19 years) Sophie worked. It appeared to be one of those information gathering interviews without an actual position to fill. I was asked a lot of questions but none seemed related to a specific opening.  At the end of the interview, the interviewer said he’d be calling me.  Time went by without any more contact. Nearly a month later, I got a call asking if I could start the next morning.

I was one of a handful of people hired to work as “data collectors”; to execute tests and collect results for an interoperability demonstration of digital imagery systems.  These systems, from a variety of different vendors, were used by the military for manipulating and exchanging digital photographs.  This test was the second phase of an effort to validate a new communications protocol standard. Our mission was to assess to what extent systems made by different companies could exchange information over a variety of communications systems. We were given a huge matrix of imagery systems and communications equipment (from encrypted telephones to push-to-talk radios) combinations to test. We quickly learned that this required we learn the communications systems, the imagery systems, and the protocol used to exchange data.  I enjoyed this challenge. By the end of the project, I found myself in the role of being the writer of the test report and numerous interface documents. I also learned that I enjoyed testing.

If I hadn’t received that call which led to becoming a tester, I would have likely taken a job as a pay telephone technician. We all know how cell phones have since killed the pay telephone business.

So, what was testing like in the early 90s?  For me, working for an independent fee-for-service testing organization, it was a great learning experience.  I learned that my role as a tester was to help our clients produce better products; not to just pass or fail their products.  I learned that every company followed different development and testing practices – regardless of what labels they gave them.  I learned to seek guidance from others who could mentor me. I learned that continual learning is essential in testing. I learned that testing is a better assessment tool than an assurance tool. I learned to plan and script testing only to the level of detail that made sense; understanding that some detail cannot be known up front.  I learned that testing is exploratory in nature.  I learned to communicate with stakeholders and developers. I learned that the real requirements aren’t necessarily the things documented in a technical requirements document.  I learned to create tools to help me accomplish tedious tasks and enable doing things I could not do manually. While I didn’t realize it at the time, I was becoming a context-driven tester.

After nearly a decade, I moved to working for commercial software development firms — with fear and trepidation.  I wasn’t sure that my skills would transfer to being an in-house tester. I soon discovered that they transferred well.  I spent a short time with some misguided “quality cop” thinking after I was given a “Quality Assurance” title before better understanding and returning to my context-driven roots.

Rapid Fire Q&A:

• Favorite movie of all time: My favorite movie of all time would have to be “Star Wars, Episode IV: A New Hope”. It is both one of my earliest and favorite childhood movie memories, and I still love it

• Favorite movie… that’s about testing or taught you something about testing: I don’t know about it being a favorite, but “Man of the Year” comes to mind. The movie was disappointing in that it wasn’t the comedy it was advertised to be; and the software “bug” on which the plot was based made little technical sense; however, it illustrates the potential risks of software not working as it should – either due to oversight or ill intent.

• Mac or PC: Commodore. I still do everything on my Commodore 64. Seriously, I’m a PC. I’ve not used a Mac in over 15 years.

• Browser of choice: I’ve recently come to like Chrome for most of my personal browsing. However, I prefer Firefox for testing. There are many cool add-in tools for Firefox.

• Brand of smartphone: Motorola Droid X. After seven years of using Windows Mobile devices, I just switched to Android.

• Favorite US president (no fair picking Millard Fillmore… he’s my fav): Teddy Roosevelt. I can’t help but love an adventurer.

• Top tech innovator of all-time (Edison, Gates, Jobs, Andreesen, et al): Apart from the modern computer, Thomas Edison has likely had a great impact on humanity.  So, if I had to pick only one, I’d say Edison. When it comes to computers, Charles Babbage and Steve Wozniak come to mind. The ideas of Charles Babbage led to the programmable computers we have today. And Steve Wozniak was essential in making computers personal and bringing them into our homes

• Best concert you’ve ever seen in person: I haven’t been to a real concert in over 20 years. However, I have fond memories of Barry McGuire sitting on the edge of a stage singing songs to the last couple hundred stragglers who remained.after someone else’s concert.

• Hobby that would surprise our readers: Since I recently quit my job and am looking for new work, I could say that software testing is now a hobby for me. It will hopefully be paying work again, real soon. I could call my obsession with old programming books a hobby. I collect and read old programming and testing books to better understand the history of software development.

Photography and exploring Colorado are two things I could also call hobbies; but those won’t surprise anyone who follows me on Twitter. Also, we’ve got a bit of a zoo at our house with cats, dogs, and fish. That too should not be a surprise.

What may be a surprise is that I’m a recovering collector of NASCAR die cast cars. Yes, I was once an addict with new cars regularly arriving in the mail.  I’ve now been clean, and not added any new cars to my collection, for well nearly two years.

uTest: If Al Gore had never invented the inter-web and software testing didn’t exist… what would you be doing for a living?

Simo: I’d likely be a starving photographer. When I was eighteen, I was considering pursuing a career in computer programming or photography.  My father advised me that he’d encountered more starving photographers than computer programmers. So, I pursued software over photography.

Editor’s note: We hope you enjoyed our Testing the Limits interview with Ben Simo. Who should be our next guest? You tell us.

In part II of our Testing the Limits interview with Ben Simo, we’ll discuss whether you should trust automated testing tools; the proliferation of testers on Twitter; the true meaning of “QA”; how testing evolves differently in each company; the long lost Bach brothers and much more. You can catch up on the conversation by reading part I. We’ll wrap things up tomorrow with part III.

********

uTest: Jon Bach mentioned that changing the meaning of “QA” to Quality Assistance would help outsiders (engineers, executives, et al) better understand the role of this discipline.  Agree or disagree?

Simo: I believe I first heard  “Quality Assistance”  from Cem Kaner.  I agree with Jon. When testers bear the title Quality Assurance, it often implies that they actually assure the quality of other people’s work. Testers are in a position to help assist quality; not assure it. Let’s not assist the setting of unrealistic expectations with inappropriate titles.

uTest: While we’re on the subject, are you anyway related to James and Jon Bach? The resemblance is uncanny.
Simo: I don’t think so. I’m available for adoption if the Bach family is interested.

uTest: You’ve said that you frequently use automated tools, but that you don’t trust them entirely (back to that whole defensive pessimist thing again). What advice do you have for testers and managers wanting to strike a healthy balance? And what’s currently in your arsenal of automated tools?

Simo: My mistrust in tools is based on the fact that tools can’t think for me. Automated checking can only process whatever decision rules someone thought to program when the checks were created. Automation will consistently do what it is programmed to do and consistently not do what it is not explicitly programmed to do. I find test automation to be useful. In fact, there are some things I’d not want to even try to do manually. I do, however, distrust the green bar. When automated checking passes, I ask myself what the automation does not tell me. I also try to keep aware that people who don’t understand what the automation does are likely to assume that it does more than it does.

Tools are much more than test automation. Tools are essential for testing. I don’t want to test without tools. I have some old programming books that promote testing in which a programmer manually executes code, step-by-step, with pencil and paper in order verify that the code works as expected. This is manual testing. This is a testing practice that came from a time when computer time was rare and cost more than people. We’d now laugh at someone proposing testing in this manner.

Today, whether we call it “manual” or “automated”, tools and automation are part of coding and testing software.
Programmers use fancy integrated development environments that do a great deal of the tedious work of managing and checking code. Programmers have unit testing tools that allow them to write tests as they write the code. We use tools to track and version source code. We have continuous integration tools that automatically run pre-defined tests and report results when programmers save code into shared code repositories. Today, programmers rely on interactive tools to do work that used to be done manually. These tools support people coding software rather than trying to replace them with batch processes.

Many of the tools built for testers seem to still be stuck in the world of batch processing. Rather than assist testers by helping with tedious tasks, they try to replace test execution. Some of this comes from the design-all-tests-up-front-in-procedurally-scripted-detail thinking that made sense when computer time cost more than people.

I worked as a tester for nine years before I encountered a record and playback type test automation tool. In those nine years, tools played an essential role in my testing. We built all our tools in-house. We wrote macros and scripts to do tedious data processing. We wrote code to verify data formats. We created test interfaces for databases. We created test management systems. We depended on tools but did not have a single test case that was fully automated. This automation made testers both more efficient and effective.

Rather than look at testing as something to be either manual or automated, I encourage people to look at individual tasks that are part of testing and try to identify ways that automation can help testers evaluate software. Rather than limit tools to automation of test execution, I look for ways that automation can assist testers – to turn them into powerful cyborgs.

These tools include automated test execution tools, data setup and teardown scripts, communication tools, test management tools, test logging and reporting tools, test data manipulation tools, and more. Along with test execution automation tools (take your pick), my most-used testing tools are Excel, Picasa (for screenshots), a screen recorder, and the GNU Unix utilities (also available for Windows).

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

Simo: I’ve not noticed great distinctions in testing practices based on industry.

What I have noticed is that testing practices seem to evolve within individual companies and test groups. Rather than learn from what others have done, it often appears that each tester, each test organization, and each company starts in some stone age and evolves on its own – making the same mistakes that others did long before.  Lessons learned don’t seem to be shared much across companies.

I’ve also noticed that “quality assurance” groups in large commercial businesses, regardless of industry, tend to be dominated by factory school thinking. Instead of viewing testing as cognitively complex work that requires critical thinking, good communication, and continual learning: testing is thought to be predictable repeatable validation work that can be done by automation or cheap labor.  These companies tend to view quality and process as things to be scripted, controlled, and quantitatively measured. In doing so, I believe these companies dehumanize their people and their customers.  These companies tend to have what Jerry Weinberg called “overstructured management” in a paper written nearly 30 years ago; management which attempts to manage people like code: with sequential work models, binary either-or choices, and mechanical repetition. These things may be good for computers, but aren’t great for people.

While many companies are still recursively calling on the mistakes decades past, others are recognizing that people aren’t to be controlled like machines.  While they may not be familiar with either document, these companies exhibit the values expressed in the Agile Manifesto and practice the principles of the Context-Driven School. These companies value people over systems.

uTest: Who should testers be following on Twitter to stay current on the latest industry trends (I mean, besides you and us, of course)?

Simo: I’m not as interested in trends as much as I am in experience reports, and new ideas and challenges. Some of my favorites who are active on Twitter are Pradeep Soundararajan (@TesterTested), Shrini Kulkarni (@shrinik), Matt Heusser (@mheusser), Lanette Creamer (@lanettecream), Elisabeth Hendrickson (@TestObsessed), Michael Bolton (@MichaelBolton), Jon Bach (@jbtestpilot), and James Bach (@JamesMarcusBach). Beyond that, I suggest people see who these people, and I, follow and interact with on Twitter. Use Twitter’s search feature to find people tweeting about things that interest you. Don’t limit yourself to only testers or those people you like. We can learn a great deal from those outside our field or with whom we disagree.

Editor’s note: Read part III of the Ben Sim Testing the Limits trilogy.

In part I of our interview, we get his thoughts on the Worst Bug Ever; his testing philosophy; what it means to be a defensive pessimist; testing certifications, the state of the industry and more. Be sure to check tomorrow for part II.

**************

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

I’d like to be able to say that I carefully selected the name and that it signifies that I care about quality and I’m amphibious like a frog. I’d like to say something along the lines that I started life as a tadpole in the waters of programming and later grew legs to live on land and be a tester.  I could even say that as a Quality Frog, I now eat bugs for breakfast and help keep the waters clear. While such thoughts may have come to mind, the truth is that I came up with Quality Frog while pairing a variety of words with Quality in search of an available domain name. Frog came to mind as something that ate bugs and the domain name was available. Since then, I’ve continued to use it as a handle.

uTest: You’re a self-described “defensive pessimist”, which seem like good qualities for a tester to have. What other attributes come in handy in this line of work?

Simo: The term “defensive pessimist” comes from Dr. Julie Norem, a psychology professor at Wellesley College.  In her book, “The Positive Power of Negative Thinking”, Dr Norem describes defensive pessimists as people who typically perform worse when pressured to look on the bright side and be optimistic about things that concern them.  Rather than trying to think happy thoughts and only look at the positive, defensive pessimists imagine the worst case scenario; not to get depressed and become immobilized, but to develop solutions for what might go wrong in order to be better prepared. Defensive pessimists can make great testers, and are likely to annoy many optimists.

The third guiding principle of the Association for Software Testing states “AST views software testing as a cognitively complex activity that requires critical thinking, effective communication, and rapid self-directed learning.”  I fully agree with this.  Therefore, I find it essential that testers be critical thinkers, effective communicators, and self-educators.  Any one of these three things without the others will make us less effective as testers.

This doesn’t mean that every tester must master all three of these on their own.  My ideal test team would be comprised of people with a diversity of aptitudes, skills, and experience. I don’t want a team of clones.

uTest: Much like our previous Testing the Limits’ guests, you’re a critic of testing certifications. Yet some still see certifications as the only way to stand out from the “unskilled labor” crowd. Tell us a bit about why you’re a skeptic/critic of certs – and how they could be improved and made more relevant/useful/predictive.

Simo: I am not against certifications per se. I am against bad certifications.  I am against certifications that are presented to be something other than what they are. I am against certification bodies and trainers that prey upon people’s desire to stand out and tell people they can improve and certify their competence as testers with few days of training and a multiple-false test. In his keynote at the Conference of the Association for Software Testing (CAST) this year, Cem Kaner stated that If one can become certified in their profession in three days, they are a commodity, and don’t deserve much more pay than unskilled labor. I agree with Dr. Kaner. Rather than educate and help people stand out from the unskilled labor crowd, such certifications trivialize testing and encourage wrong thinking that testing is unskilled labor.  I want testers to be more than a commodity.

Many IT certifications, including testing certifications, are more about marketing than education.  These certifications are not good measures of skill, competency, professionalism, quality, or any of the many things those on the receiving end (of the marketing) care about. In my experience interviewing job candidates, tester certifications have not been an indicator of applicants’ testing abilities.

Software testing is a rich and diverse field. It is also a young field. Rather than feign maturity and simplicity where there is none, let’s embrace the diversity and youth. Let’s continue to learn. Let’s not lock in a set of context-free definitions and practices and make them a standard. Such standards will hurt the quality of software, not improve it.

Rather than pursue a certification, I encourage testers to get involved in a professional community. Find colleagues that challenge you and help you learn.  Seek real education that comes through interaction and doing over memorizing information useful in passing a multiple-false test. The Association for Software Testing offers a series of online software testing courses that facilitate deeper learning that you are likely to find in training focused on helping you pass a certification exam.

Editor’s note: We hope enjoyed part I of our interview with Ben Simo. Here’s part II.

In part III of our of Testing the Limits interview with Cem Kaner, we discuss why “best practices” is merely a marketing tool; Silicon Valley of yesteryear; his upcoming CAST lecture on investment model and exploratory test automation; the blogs he reads and much more. To recap, here’s part I and part II.

uTest: What’s surprised you the most about the testing industry since you’ve been in the game?

Kaner: Context-dependence was a big surprise to me. It took me about ten years before I reluctantly accepted the idea that my favorite test techniques, attitudes, and life-cycle models were appropriate in situations similar to the ones where I developed my preferences, but not so appropriate in situations that were quite different.

It took me a long time to learn that developing software under a well-specified contract is a common and respectable activity that requires different tradeoffs from mass-market software sold to people who had no say in its design and buy only after it is ready to deliver and only if they like it. It took me a long time to sort out differences between scientific programming (where I started) and consumer software development. It took me even longer to see that testers working in an independent test lab operate under fundamentally different constraints from testers working inside the development company and provided different types of information. And it took well over a decade before I accepted the idea that two different development managers, running essentially the same project, could legitimately want different information from their testers and that it could make sense and be ethical for the two different test groups to structure their work differently, finding or being blind to or ignoring different classes of bugs, in order to satisfy the information needs of their key stakeholders.

The last major event in this chain happened 14 years and two books after I came to Silicon Valley. A client paid me to tour a lot of companies in California, Oregon and Washington. I gave a talk at each place, but I also talked with them about their business models, their testing challenges and methods.

Most of these were good companies with competent testing staff but they did things very differently from each other, often very differently from what I (in my ignorance) would have recommended, but in ways that addressed the risks they were trying to manage. It was already my practice to try to understand what worked at a client site, and why it worked, rather than to evaluate what they were doing against my prejudged ideas. But the diversity in this series of clients was overwhelming. It caused me to abandon many of my favorite ideas about development and testing—not as bad ideas, but as good ones only under suitable circumstances.

uTest: You’re shaping the minds of future testers… so how do you “future-proof” your teachings?  And what major changes do you think will impact software testing by the end of this decade?

Kaner: What I DON’T do is try to slow the field down.

I don’t pretend that there are One True Definitions for any testing terms, because different groups of testers see the craft differently and they use their language accordingly. If two testers have different ideas about the purpose and goals of testing, they are likely to have different meanings, or at least different nuances, in their definitions of “test case.” I don’t go to standards boards to try to legislate my favorite meaning as the official one. This effort to lock down a field that is still in motion, still finding itself, will primarily benefit people selling certification courses. In terms of helping their students prepare for the future, I think that this gives an illusion of certainty and uniformity to people who should be training to embrace questionability and diversity.

Similarly, I think the idea of “best practice” is a security blanket for babies or a marketing tool. In my experience, context matters a lot. A “practice” (a something that I use or do) can work very well under some circumstances and poorly under others. Teaching about “best practices” is another way to sabotage the critical thinking  of the students. As the context changes, people with this training have a poor foundation for adapting, or at least for adapting in any other way that paying a consultant to tell them what to think and do next.

As I see it, we test in order to find out useful information. We use technology (usually) to ask machines questions about how they work and humans questions about how the machines should work. We ask the questions in order to reveal information about the quality of what we are studying, and we usually do that because someone is paying us to get that information.

So, the “future-proof” skills and knowledge are the critical thinking skills, the questioning skills, and the understanding of full-lifecycle scientific method (including the qualitative research needed to develop ideas that lead to potentially-valuable hypotheses and the formal methods useful for focusing and assessing (typically refuting) those hypotheses.) The rest is going to change.

I can’t future-proof my ideas. That illusion is for the standardizers. Many of my ideas have changed enormously over the years. Take a look: http://www.kaner.com/pdfs/TheOngoingRevolution.pdf. Until I stop learning, my ideas will change even more.

uTest: Your latest presentation revolves around investment modeling and exploratory test automation. Give our readers a sneak preview of what to expect (also, where and when to expect it).

Kaner: This is a pair of talks at CAST (the conference of the Association for Software Testing). CAST is at Grand Rapids, Michigan from August 2-4. See http://conferences.associationforsoftwaretesting.org/CAST2010

The first talk is with Doug Hoffman on Exploratory Test Automation (ETA). Mainly, we will review a variety of examples of projects that used automated tests to learn new things about products rather than trying to confirm that old knowledge was still correct. Many of these involve automated creation of enormous sets of tests, scanning the program in ways that a human can’t, looking for potentially interesting results that would be worth a human’s time to investigate further. Several other people are having insights that a lot of (partially) automated testing that they do is not regression testing and is useful for learning new things. They’re presenting their ideas as ETA too, at CAST and at other conferences. We’re just adding our list to the mix and suggesting a framework for sorting out the differences and similarities among this large family of ideas.

My experience with teaching people about ETA in the past is that the examples require technical sophistication to understand (e.g., what does printer firmware really do and what are we trying to learn when we test it?). An hour-long talk is too short for some people. The good news is that at CAST (like at the LAWSTs), we are not limited by the time allotted for the talk. CAST has a breakout room. If people want to argue with us or ask detailed questions after the talk, we aren’t allowed to say, “I’m sorry, we only have 1 minute left to answer your question.” At CAST, we move to the breakout room and stay with the questioners until our discussion is done. Probably this hour-long presentation will take 3 to 4 hours (with questioning) before it is actually done. And hopefully, the people who stay with us to question and argue will walk away understanding what we’re trying to teach.

The second talk is my keynote, at the end of the conference. Building on what Doug and I presented, I present a case study, explaining how I use automated exploratory tests to assess software that implements investment models. Many people find discussions of investing easier to understand and more interesting. See http://adam.goucher.ca/?p=1492 for a reaction to a draft of the talk.

uTest: How do you effectively bridge the gap between academic theory and commercial practice?

Kaner: There are thousands of gaps, in all directions. We don’t ever bridge “the” gap. We bridge specific gaps, one at a time.

The first challenge is to think across paradigms, across ways of looking at the world. If I want to introduce my ideas on design to psychologists, I need to learn to think enough like the psychologists who I am trying to influence to make my presentation influential. I have to understand what questions they are trying to answer or what problems they are trying to solve, and then tie my ideas to their interests or needs. That takes work on my part.

Similarly, if I am trying to learn lessons from another field, I have to learn about that field. A new technique, a new idea, a new experimental method or a new insight (and accompanying wisdom) – these are rarely packaged in a self-contained way that lets me grab one and use it. What is this technique used for? Why? What does it replace? What makes it better? What are its limits?  To understand these things, I need to be able to understand the literature (books, articles, conference talks, demonstrations) that present the new knowledge. And all the time, I am asking, how does that apply to what I might be interested in? Again, that takes work on my part, a lot of it.

I host an annual meeting called the Workshop on Teaching Software Testing. This is one of the LAWST-style workshops (http://www.kaner.com/lawst.htm). We recently had our 9^th one (http://www.wtst.org/). The LAWSTs generally allow up to 20 people in the meeting. We schedule presentations, but we work through a presentation until we’re done with it. Thus, a 1-hour presentation (according to the schedule) might trigger one minute’s worth of discussion or a full day of discussion (we’ve gone to both extremes in the LAWSTs). The questions are often detailed. Sometimes there are debates and one participant will create slides to make a counter-presentation. In a meeting like this, 10 to 20 people work together to build bridges to each other’s ideas. At WTST, we bring together teachers—some academic, some commercial—and we share ideas on how to teach topics in testing.  This is my most effective vehicle for building bridges.

uTest: Name some other testing pundits, zealots, teachers and thought-leaders whom you respect. Who should we interview next month to share their wit and wisdom with the testing world?

Kaner: I’m surprised you haven’t talked with Elisabeth Hendrickson yet. And talk to Robert Austin.

Rapid fire Kaner Q&A:

• Last book read: Harry Dormash, Fire Your Stock Analyst. (This one was so good, I’m adopting it as a textbook in my investment modeling in software engineering course.)
• Last movie watched: Karate Kid
• Mobile device of choice: Amtrak
• Mac or PC: OK
• Techcrunch or Wired: Digg.com, Huffington Post, Automated Trader magazine, Bulletin of the ACM Special Interest Group on Computer Science Education, Law.com, Psychological Science, Bloomberg News, CNBC (especially Cramer) and Rachel Maddow. Ask me next year and the list will be different.
• Whiteboard or chalkboard: projector
• Best video game of all time: No idea. But my personal favorite was Wizardry.

Editor’s note: We hope you enjoyed our latest Testing the Limits interview. If you have a suggestion fir our next guest (or have a topic you want covered) just let us know.

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

• http://kaner.com/pdfs/FITcolloquiumBuildingFreeCoursewareCommunity.pdf

I describe the challenges in a little more graphic detail here:

• http://kaner.com/pdfs/nsfKanerFiedlerBarber.pdf

And I walk through our instructional model in a lot more detail here:

• http://kaner.com/pdfs/DevelopingInstructor-CoachedActivitiesNSF2008.pdf

At university and in practitioner training, my preferred teaching style is to leave the lectures to the videos, to use quizzes to drive students to watch the videos (and read the assigned readings) carefully,  so that they get the basic knowledge that I’m trying to convey, and then to use classroom time (in a live class or most of the course time in an online class) to do tasks based on the materials covered in the videos and readings. The students can apply and argue with the videos and readings in these tasks. I believe we foster a deeper level of learning this way.

As to “curriculum,” I think there are many different, valid, instructional objectives.

In an undergraduate university curriculum, the goal is not to train people for a specific type of future job. It’s to prepare people for the next 10 years of their career, with the expectation that they’ll want to switch jobs, and probably careers, a few times in that period. Undergraduate education has to be about teaching people learning skills as much as technical skills, and enough fundamental education that they are not blocked when they approach materials outside of their core domain. This drives us to a hard-to-optimize conflict between “general education” and technically-focused education in the field of the student’s choice. The broad curricular tradeoffs are too complex for me to understand how to resolve.

In terms of software testing, most universities offer little education directly focused on testing, and much of what is taught at most schools (that teach testing) is thoughtful code verification blended with applied mathematics, rather than skeptical assessment of product quality. Much of this contributes valuably to general education and to programming skill. But not so much to the craft of testing.

Mainly, our field relies on on-the-job experience and commercial training. The training courses rarely get much beyond the basics because the average student hasn’t gotten much past the basics. In addition, there is a critical difference between university education (done well) and commercial training: projects and homework. Students in my Testing 2 class (programmer testing) write take-home final exams of up to 5000 lines of code. Other courses at Florida Tech demand more than this. Most projects are much smaller, but each of them provides experience, not just vocabulary. Active engagement with course material is what fosters deeper learning, and we don’t have time for that in commercial training.

As a result, instead of starting from a well-built-up foundation, testing practitioners learn most of what they learn about testing on the job and in relatively simplistic training courses. They reinvent the field, each for herself or himself, learning the same lessons that we have learned 10, 20 or 30 years before. This is not a path to significant progress in the field.

This low level of common knowledge in the field is convenient for certification courses. They can sell a few days of training (at a high price) that is enough for students to pass a relatively trivial exam. Think of the absurdity of this. The current certifications are essentially advertisements that you (the certificate holder) know as much as someone with no experience and three days of training. Would you hire someone who was proud of that low a level of knowledge?

I keep meeting people who tell me that they send their staff to these kinds of courses because the course teaches everyone a common vocabulary, and our field needs a common vocabulary. I don’t personally think that a common vocabulary has much value. What is more valuable is learning to ask someone else what they mean when they say something.

Thirty years ago, a “big” program was 10,000 lines of code. Today, even cell phones have millions of lines of code. This enormous change in productivity comes from revolutions in the practice of programming and software design. But we aren’t seeing revolutions in testing, not like the change into structured programming or out of structured programming and into object-oriented. No one would think of teaching the programming methods of the 1970’s or 1980’s as state of the art today. But the current certifications in testing (and many of the “applied” university and college courses) seem to me to be based on “best practices” and attitudes that aren’t much different from what I started writing Testing Computer Software to rebel against back in 1983.

We need to make progress on the education problem, or we will dig deeper and deeper into the same old ruts.

uTest: How well are companies adhering to The Software Consumer Bill of Rights (which you wrote in 2007)? Have conditions improved for the average software consumer?

Kaner: The Principles of the Law of Software Contracts have finally been adopted by the American Law Institute. That will probably reflect itself in judicial opinions over the next few years.

In terms of legislation, the United States is in a dysfunctionally polarized period. Very little legislation of any kind is being passed. We might not see useful statutes (laws passed by legislatures) on computer law for several more years. Until then, companies will start acting more responsibly after some successful lawsuits or enforcement actions.

uTest: James Whittaker has made the jump from academia to corporate America.  Will we see you take your considerable talents “across the aisle” at some point (assuming that you aren’t doing so already)?

Kaner: For me, this draws a false dichotomy.

As I said in my last note, programming has been evolving in powerful ways. Testing: not so much. Developing better education/training for testers is one part of the way out of this hole. That’s what I’m currently working on. I’m working at a university because universities house the cutting edge for educational practice.

On an ongoing basis, I ask myself whether I am working on the projects that have the potential to make the greatest positive impact and whether I still think I can realistically make the kind of contribution that is needed. For now, I’m in the right place doing what I think are the right things for me to be doing. Later, that will certainly change. When? How? I don’t know.

Editor’s note: Check back tomorrow for part III of the interview.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

My dissertation focused on measurement of perceptual experiences (a field called psychophysics). We asked questions like “How loud is this sound?” Hearing is an experience. The physics of the sound can help us predict some aspects of that experience, but loudness is about what it sounds like to you, not the physics.

It’s pretty easy to think about how to measure the length of a table. Use a ruler. But what ruler do we use for sweetness? In order to think clearly about this, we need to clarify for ourselves the construct (the idea of) “sweetness.” What does someone mean when they say that X is sweeter (to them) than Y? Then we have to imagine a measuring instrument (like a ruler) that would give us low numbers for not-sweet, high numbers for very-sweet, and reasonable distances between different amounts of sweetness.

“How do we know that this (ruler) measures that (construct)?” is one of the fundamental questions of measurement. It even has a name, “construct validity.” In every field that I’ve studied other than computing (I’ve looked at measurement theory in a lot of fields), when people develop measurements or “metrics” and when they teach metrics, they talk seriously about construct validity. In contrast, construct validity is almost never mentioned in computing. (Do a search for “construct validity” in ACM’s Guide to the Digital Literature! Filter out the papers in business-oriented or medicine-oriented journals. What do you have left?). We have textbooks with dozens (I taught from one text with hundreds) of “metrics” and no practically-applicable system for assessing what these “metrics” measure.

Capers Jones sometimes talks disparagingly about the (claimed) fact that 95% of American software companies have no metrics program. On the surface, this sounds terrible. But what I saw as a consultant was that most software companies have tried a measurement program, or have executives with lots of experience with metrics programs in other companies. The problem is that their experiences were bad. The measurement programs failed. Robert Austin wrote a terrific book, Measuring & Managing Performance in Organizations. When you start measuring something that people do, people will change their behavior to make their scores better. People will change what they do to get better scores on the measurements—that’s what they’re supposed to do. But they don’t necessarily change in ways that improve what you want to improve. Often, the changes make things worse instead of better (a problem commonly called “measurement dysfunction.”) This problem happens more often, and worse, if you use weak, unvalidated metrics. I keep meeting software consultants, especially software process consultants, who say that it’s better to use bad measurements than no measurement at all. I think that’s’ a prescription for disaster, and that it’s no wonder that so many software executives refuse to harm their businesses in this way.

By the way, what if one person repeatedly says X is sweeter than Y and another person says Y is sweeter than X? Are these people wrong about their experiences? Probably not. So is X sweeter than Y? Yes. Is Y sweeter than X? Yes. Have fun putting THAT on your measuring stick.

Jerry Weinberg defines quality as “value to some person.” It’s subjective, like sweetness. Different for different people. Quality is not quantity. But that doesn’t make it any less real.

In terms of the contributions of multiple fields to software, it’s no surprise that Weinberg’s doctoral work was in psychology or that he is so popular a thinker in the testing community. When we evaluate the impact of a new piece of technology on humans, we are doing applied social science research. To say that testing is part of a fundamentally different from the social science is to miss the point of much of testing AND to turn a blind eye to tools developed over centuries that can help us do our work.

Let me switch disciplines–I started working in Silicon Valley in 1983. I watched some excellent companies fail and weak competitors succeed. A striking commonality was that companies would release products with missing features or features that didn’t work, often lying about what they offered. They would gain market share so quickly that they could drive out more honest companies. Most bad companies eventually failed but they cost the marketplace its best suppliers time and again. As a tester, and sometimes as a testing consultant, I was encouraging topnotch staff to hammer products very hard, to expose bugs and argue effectively to get them fixed – to do the right thing by their customers. But the effect of this was that companies were coming to market late with a better product but a worse competitive position. Doing the “right” thing was being punished in the market.

This REALLY troubled me.

I wrote Testing Computer Software over a period of 4 years (1983 to 1986). In many ways, it was an excuse for me to ask people probing questions and think with them about answers.

In terms of markets, I learned that in mature industries, North American society places expectations on companies. We expect companies to advertise honestly, to make products that do what they are supposed to do (maybe not perfectly, but pretty well), to fix their defects for free or give customers their money back, and to allow their products to be analyzed by journalists  who help customers distinguish good from bad products. As a society, we write those expectations into law. When we enforce the laws, we weed out the worst companies and create a safer market for honest businesses and their customers. These rules are less rigid for companies that sell services (e.g. custom programming), but in mature industries, service expectations (and rights) are pretty demanding too.

I came to believe that the core quality problems in software were (and still are) to a very large degree the result of loose enforcement of laws governing fraud, breach of contract, breach of warranty, and service negligence. I learn by doing. To learn more, I became an Investigator (part time, volunteer) in the County of Santa Clara’s Consumer Affairs Department. (Most of Silicon Valley is in Santa Clara County.) After a couple of years, I decided to go to law school and to focus on the law of software quality.

From 1984 to 1987, I worked in a telephone company in a user interface group. A telephone is a user interface to a richly featured, complex network of computers. Our particular system was a really early provider of integrated voice and data. You plugged your computer into the phone to be on the network, rather than into a modem or an Ethernet cable. We also delivered the first phone system that was actually for sale (rather than demonstrated in research laboratories) that had an LCD display.  The menus on our 2 line x 40 character display gave access to 108 voice features and 110 data features.

In our group, we could all program, we all knew a lot about usability. We evolved our system’s user interface over years, in an evolutionary manner (add one or a few features and get them working before adding the next). We moved fluidly between customer focus and code focus, and we modified each other’s code all the time, trying out new ideas for design, which we (and the rest of our company and sometimes our external customers) assessed at the systems level in weekly deliveries of new system builds. It was a great experience.

I don’t want to downgrade the technological challenges of this work. We had to pack a ton of capability into small spaces and have it run very quickly and extremely reliability. The programming side of my work was very difficult. But the programming would have been meaningless without a focus on who the programming was for and it would have been ineffective without an ongoing stream of information about what was working (design and code) and what was not.

At another company (Power Up Software) I was a Software Development Manager. When I started, my task was to design products for small businesses, help plan the marketing of these products, and manage programmers (typically remotely located programmers) who were to write these products. I wanted to turn my products into best sellers but I didn’t know how, so I took a job at Egghead Software, then a successful software retailer, and sold my best competitors’ products to Egghead’s customers. These people didn’t just buy once and disappear. They came back to the store every week or two, buying new things or complaining about old ones. You learn a lot about what makes a product great, and not-so-great, by selling it to people and then dealing with them, face-to-face, over and over. What is quality? Quality is what made my customers happy. What’s a defect? A defect is what made my customers shout at me.

Some people would tell me that quality is conformance to a specification and a defect is a nonconformity. But really, what’s a specification? A specification is a piece of paper, often written by someone who knows nothing about quality.

About those specifications. I did a lot of school learning about software process. I came to Silicon Valley expecting to see lots of well written specifications. What I actually saw were a lot of scribbles on napkins (from the local pub or restaurant), scribbles on flipcharts, and scribbles on whiteboards. They were informal and they were updated as we came to understand our customers, our competitors, and our products better. Many of the most tedious-to-use, most poorly-designed products that I worked on or consulted about were coded to meet a detailed specification. Famous consultants would come visit our companies or teach courses that I sent staff to, and they would tell us that The Right Way to build software was to write everything down at the start and then to build to that specification. But the start is when you know the least about whatever you are creating. As you create it, you learn. Why make your decisions when you don’t know as much as you’ll know tomorrow? Why set up change control processes to make it expensive to fix all the mistakes you made when you were more ignorant than you are today?

Of course, there are reasons to set up change control processes. And there are reasons to develop a clear idea of the product’s scope early on (try designing an appropriate architecture without this).

But as the Extreme Programming advocates came to say much more eloquently than I knew how, knowing the big picture is one thing, but there is rarely a good reason to lock down details until you need to code them. And you cannot call something a “quality” process if it has you refusing to improve a product because it makes the cost of testing too high or the paperwork too expensive.

While I was dev manager (and later, director) at Power Up, I went to law school (you don’t learn much humility there, but it’s good for other things), went to University of California Extension for several courses on technical publications management (to help me work with a tech pubs group that adopted me as their manager), and wrote the second edition of Testing Computer Software.

Hung Nguyen joined me in writing TCS 2.0, but on the condition that I take courses in traditional quality control. He had a B.Sc. in Quality and wouldn’t write with me until I got ASQ-certified in quality engineering. His point was that I had become an expert in informal processes that worked on (relatively) small projects, but in his view, I needed to understand what effective control processes were and why they were important on larger projects and formally-negotiated projects (outsourced development, especially when your customer is the government). That blending helped us make TCS 2 a much better book than the original.

The diversity of my situations taught me powerful lessons about how little I know, how much the other people on the project know that I don’t know, and how complex are the tradeoffs faced by the average project manager. When I work as a tester, I understand that I am playing a supporting role. It is an important role. It calls for skilled and thoughtful work. But as James Bach puts it, I am “the headlights of the project”, not the driver, not the brakes, not the engine, and not the door to the customer. I can change roles. Maybe I can work two or three roles at the same time. But IN MY ROLE AS A TESTER, I don’t understand as much about the rest of the project as many other testers (and test consultants/trainers) seem to think they understand. There is a lot of arrogance in our field. We need to learn more humility.

I graduated law school in 1994, passed the bar exam and was sworn in as an attorney. I went back to the Santa Clara County government, this time to work as a full-time volunteer in the District Attorney’s office. My ideas about fraudulent and unfair trade practices had deepened into a passion. As a DA, I prosecuted about 130 cases, including 5 trials. None of this was white collar crime (e.g., economic crime by corporations) but I learned how prosecutions were run, how crimes were investigated, how cases were organized, negotiated, and occasionally, brought all the way to trial. I didn’t become an expert, but I gained more courtroom experience than most American lawyers gain in their professional lifetime. It was foundational knowledge.

Over the next six years, I spent a huge portion of my time as an advocate for a higher legal standard for software quality. I helped write the Uniform Electronic Transactions Act (UETA), which was adopted by almost all American state governments. The federal government renamed it the ESIGN act (electronic signatures) and applied it to all the states so that we had nationwide uniformity in some critical aspects of electronic commerce. I also tried to help write, and then led the opposition to, what became the Uniform Computer Information Transactions Act (UCITA), a bill that in essence, rewrote contract and copyright rules to benefit large software companies and large companies that embedded software in their traditional products (e.g., cars). UCITA ultimately failed despite the millions spent on it. And one of the most important legal organizations in the United States, the American Law Institute, elected me as a member. The ALI has up to (a maximum of) 3000 members. Most of them are judges or tenured law professors. Some are senior partners in large law firms, a few are famous consumer advocates. I think I was the least experienced lawyer ever elected to ALI and every time I go to one of their meetings, I know (or remember quickly) that every person in that room knows more about the law than I do. But I know a little more about computers-and-law than some of them, so sometimes, my background is useful.

ALI writes books that judges pay careful attention to. In the United States, and in all other countries whose systems evolved from British Common Law, laws passed by legislatures are more like detailed statements of principles. They do not cover every situation. They can’t. So judges have to figure out how a body of laws applies to each particular dispute before them. Across different states, judges might apply the same laws differently. ALI helps organize and make sense of the diversity of decisions. They publish the Restatements of the Law (e.g. the Restatement of Contracts, the Restatement of Torts, etc.) and Principles (e.g. Principles of the Law of Software Contracts). After UCITA failed (only two states adopted it, dozens of states rejected it), ALI let the dust settle for a couple of years and then started writing the Principles of the Law of Software Contracts. One of its most important rules is one that I advocated: a seller of software who knows about a defect of the software but does not disclose the defect to the customer will be held liable for damages caused to that customer by that defect. Note that this does not apply to free software (not sold). And if the seller discloses the defect, it becomes part of the product’s specification (it’s a feature). And if the seller doesn’t know about the defect there is no liability (once customers tell you about a defect, you have knowledge, so you cannot avoid knowledge for long by not testing). ALI adopted it unanimously last year. This is not law, but until the legislatures pass statutes, the Principles will be an important guide for judges. Even though I am a minor contributor to this work, I think the defect-disclosure requirement might be my career’s most important contribution to software quality.

The Association for Computing Machinery recently honored this aspect of my work with its “Making a Difference Award” which is “presented to an individual who is widely recognized for work related to the interaction of computers and society. The recipient is a leader in promoting awareness of ethical and social issues in computing.”

Editor’s note: Read part II of the interview.

In the second part of our Testing the Limits interview with James Whittaker, we tackle Google vs. Microsoft; dogs vs. cats; why SCRUM is just a name; his advice for college graduates; bad habits of exploratory testing and more. If you missed Part I, you can find it here.

If you want to read more of James’ work, bookmarking the Google Testing Blog would be a good place to start. You can also read his 2009 book Exploratory Software Testing or check out some of his uTest eBooks and webinars.

uTest: The Microsoft vs. Google battle continues to play out very publicly in the media. Just last week, Computerworld wrote this story: “Microsoft: No Matter What Google Says, Windows Is Secure.” Having been at both companies, we think you have a unique perspective on this one. Any thoughts?

JW: Let me say right away that I enjoyed my time at Microsoft and admire the company and the smart people who work there. As a resident of Seattle, it is in my best interest for Microsoft to prosper! But the two companies are vastly different regarding the way their talent is managed and their products are built. Google is an engineering-centric company where innovation comes from individuals who are empowered to do whatever they need to get ideas into production. Much has been made of Google’s game-theory approach to managing people where rewards are given quickly for impactful behavior. It works. Morale is high and people work very hard and take quality very seriously.

Does this mean we produce more secure or more reliable products? We try hard to do so; Microsoft tries equally hard. I think we have the advantage of less legacy and a more modern and reliable platform (the Web as opposed to client operating systems) to work from. But the secret sauce at both companies is the same: hard work and due diligence.

uTest: You shared with us (as the pioneer of Testing the Limits posts) that your first assignment at Google was “To raise the level of testing precision and diligence.” So, how did it go?

JW: It didn’t take long. Google was mostly already there so I can’t really take credit for it. Now I am busy raising the bar further.

uTest: Top tester Glory Leung is curious: What are your views on SCRUM testing in general? Are people doing it properly? What is the ideal situation?

JW: Scrum is just a name. I don’t like names, they feel too confining and people have their own ideas of what they mean. I took a lot of flak for using the name ‘exploratory testing’ for my book. People love to confine you to how they view a specific named idea or technique. Flexibility is required.

The ideal situation is that the best testers for the job be involved. For unit testing and TDD this means the individual devs who wrote the code. No one is more qualified to test a function in isolation than the person who wrote it. However the dev is often the worse person to test how his function interacts within a larger feature or with its operating environment. At Google we expect devs to do the former, all of it. Then and only then have they earned the right to have testers do the latter. Having broken or missing unit tests is a great way to ensure that testers get taken off your product.

The question I am struggling with now is product expertise. What is the right mixture of testers who are close enough to the product to become real experts with it and specialist testers who are expert breakers? Call it whatever you like but product expertise and testing expertise are the two ingredients that I find make for good testing, regardless of your ship cycles or how your team develops their code.

uTest: There are many varieties of “testing frameworks” out there that guide testers in choosing test cases and defines the way they approach test design. Out of the ones you’ve recently seen (e.g. Input Domain, Divide & Conquer, Fishbowl, Storybook, Pessimists), which of these works best? Is it a combination of two or three? Does it depend on the job?

JW: More names, less meaning and more chances of some consultant claiming ownership over them! Like I said in the previous answer to me the thing that I find most valuable are people who know the product and people who know how to test. Stir those together and you have a winning formula.

uTest: Doc JW rapid-fire (in no particular order):

1. Last book read? The Hobbit with my 12 year old son. It was his first time (how I envied him for that!)
2. Last movie watched? Avatar, 4 times.
3. Favorite band or album? Neil Young, Led Zeppelin, and Nirvana … in that order.
4. Favorite Kevin Bacon movie? Kevin who?
5. Browser of choice? Chrome!!!
6. Cats or dogs? Dogs, not even close.
7. Favorite video game? I played Monkey Ball 14 years ago, once.
8. Which mobile device are you packin’? Android Nexus One
9. Favorite mobile app? Google Sky Map.

uTest: What would you be doing with your career if you were not a Test Director (e.g. bullfighter, gourmet chef, cobbler, explorer)?

JW: Stand up comedian. I still intend to do it some day.

uTest: Can you give us a sneak peek into any of your upcoming guest lectures, books, articles or presentations on the horizon?

JW: Alan Page inspired me with his book on how Microsoft does testing, perhaps I will write one on how Google does it. There is almost no overlap. I have also doodled a memoir of sorts trying to figure out how a guy like me managed to succeed. I’ve only shared it with a single reader who thinks I should wait until I retire to publish it.

uTest: It’s graduation time…  what would you tell college students who want to enter the world of professional software testing?  What should they do (or avoid) to get their career started on the right track?

JW: Simple, learn computer science. Then, get a job at a company that takes testing seriously and be prepared to learn. A degree in CS is, for me, the first and most important background.

uTest: Since you literally wrote the book on the subject, what separates a bad exploratory tester from a good exploratory tester?  And what’s the difference between good and great in this area?

JW: Bad exploratory testers spend little time preparing and are only seeking bugs as their primary outcome. Good exploratory testers apply structure to their work and use tools to collect data so that their testing sessions are more meaningful than just bug finding exercises. Exploratory testing should be connected to the rest of the testing process and not just a late cycle bug finding activity.

Editor’s note: We hope you’ve enjoyed our latest interview with James Whittaker. We’re always open to suggestions for future guests, so if you have someone in mind, or want to submit a question of your own, be sure to let us know!

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

What we are working toward is a browser that is capable of testing applications that run inside it. The idea is to build the test framework into the browser, in our case Chrome, so that test artifacts don’t have to be moved around and installed in test labs and on test machines. Also since the browser and automation are one, problems such as Ajax and self-updating web pages won’t confuse automation. This is true cloud-based test automation. What I want is the ability to launch one of these WTF-equipped browsers and point it at the web site/app I want to test and have all the back-end test case storage, bug reporting, progress reporting etc managed for me, automatically, in the cloud. Tests that I wrote or others wrote can be consumed by the browser and applied en mass. These assets all exist in the cloud at Google, why not make them universally available?

I’m betting you want the same thing.

uTest: You’ve argued that the past two decades of software testing innovation have been “the disposable decades” – too tied to the app, the domain and the technology. What does the next decade look like to you? What will surprise us most?

JW: At GTAC 2008 and EuroStar the same year I gave a talk on the Future of Testing and I am returning to GTAC in October 2010 to give an update on my vision for the next decade. I don’t want to preempt that talk, but I will say that my vision is far more aggressive now than it was then. What I thought was 10-20 years out two years ago will be here in less than 5. Things are changing fast and I am anxious to push them faster. We are working unnecessarily hard to test our applications and it is limiting innovation and creating a market full of sub par software. This has got to stop.

The central theme of my vision of the future is getting rid of this disposable nature of our test artifacts. We are reinventing the wheel every time we test wrt planning, test development, automation, reporting and metrics. Let’s get together and stop this. One of the reasons I am so nice to you guys at uTest is that you are working with the idea of leveraging the cloud and the crowd to perform testing. There is much more collective intelligence involved in what you guys are doing and I like that. I am anxious to get our testing tools in the hands of your crowd.

uTest: In your most recent blog post, you describe a (very near) future where test data will be more like a cellophane wrapper around a web app UI, an overlay that would replace the tiresome querying of multiple databases. Just how close are we to achieving this innovative bug and test case overlay and what would be the significance of such a feat?

JW: You keep trying to pull my GTAC talk out of me prematurely! But let me give a bit of a preview. Test managers need data to make decisions about how to optimize their testing and their testing resources. Most technical test managers I know end up creating a set of SQL queries that they use to get information about features, bugs, test cases, test runs and so forth. All the data is there, it’s just hard to access. Video gamers have a similar problem, there is a lot of information in the game and it is in a lot of different places. Our solution in testing should be the solution gamers use: a heads up display – a single location to surface all the information a tester and test manager needs to make the right decisions during a testing campaign. We are building a heads up display that can consume all this information and use it to guide the hand of the tester and test manager. This is part of what I meant earlier when I said that manual testing needs automated assistance. People doing exploratory testing without a HUD are like gamers who play without one. In the latter case, the gamer’s character gets killed, in the former case the tester is less effective.

uTest: You are a busy man, James, having less and less time to write witty blog posts (which we miss)! What has been keeping you most busy and what have your greatest accomplishments been in the past 12 months?

JW: Busy or just having so much fun I can’t take the time to write a lot? My greatest accomplishment happens every day: people who work for me questioning every aspect of our business and pushing innovative solutions to other testers to improve the way they execute. I had a direct tell me two weeks ago that he couldn’t wait for Sunday to pass so he could come to work on Monday. Nothing else I do compares to that.

uTest: One of our top testers, Madhukar Jain, would like to know: Do you see cloud computing as the next evolving challenge in testing? We would take this one step further and also ask if the cloud represents or enables the fourth gen of testing, or testsourcing, you’ve often alluded to? What is the cloud’s potential to empower the crowd (third gen)?

JW: I see cloud computing as the way to solve the testing problem. And yet again you are forcing me to reveal my GTAC talk prematurely. So be it. Imagine for a moment every test case database, bug repository, build script, static analysis tool, metrics engine, and so forth in the cloud. Every tool and scrap of information a tester will ever need stored in the cloud and surfaced in his HUD when it is most relevant. It’s already stored on some server somewhere, putting it the cloud is a small step. Over time (hopefully a short time) we’ll be able to use this information seamlessly. Instead of the Internet, you have the Testernet capable of answering any question you have about testing and being there to guide you through whatever testing tasks you are performing. Other industries are already doing this. We’re behind the curve in testing.

Do you remember Star Trek The Next Generation? My friend and fellow Googler Dan Massey and I like to hold this series up as the type of technology we want to build. Geordie and Data were constantly “reconfiguring the subroutines” to solve some important task. They could reprogram the phasers, the photon torpedos, the holodeck, anything on the ship! Plug and play “subroutines!” They did this from any console on the ship. They did this without installing any back end repositories or test assets because it was all available in the cloud. They did this without running any all-night build scripts or writing new test cases. The build happened automatically, test assets were applied automatically. The reason Geordie and Data could do this programming so quickly is because they only had to solve the problem they were working on, they would ignore all the back-end build and test nonsense better suited for a computer.

I bet Geordie had one hell of a testing HUD and never had to wade through the complexity of partially automated build systems and SQL queries to find the tests and test data he needs, in fact I bet he didn’t even know anything about these backend systems. They were hidden in the cloud, exactly where they belong, allowing him to concentrate on his programming.

Editor’s note: Read Part II of the interview.

uTest: The Seattle area has spawned an inordinate number of top testers (Whittaker, Bach, Bach, Creamer, et al) – what’s the deal with that?  Is there something in the water or is just a result of the Microsoft ecosystem being nearby?
LC: If there was no James Bach there would be no interview with a crazy redheaded tester named Creamer, because I would have no testing blog. If James Bach wasn’t in Seattle, I may not have had the chance to see him speak so often. Cast 2007 was in Bellevue, WA, maybe partially because that is close to Microsoft, so I guess in a roundabout way, it could be the Microsoft ecosystem being nearby that made Bellevue the location for Cast at the right time.

I prefer to think of it as something special about Seattle that fosters a unique perspective and resilience. Maybe it’s all of the cloudy weather. The grunge movement started in Seattle, and much like Nirvana, Pearl Jam, and Soundgarden, there are some innovative contrarians who aren’t afraid to blaze new trails coming out of Seattle to this day – and flannel is in style again.

uTest: Numbers-wise, the software testing profession is clearly dominated by dudes. Why do you think that is? How do we change this trend? Does it matter, or is this topic completely overblown?
LC: When all of the women who have the talent, skills, and desire to be testing are appreciated for the value they offer, and the field is still dominated by dudes, then great! It is about having the opportunity, not about enforcing some gender ratio. Right now things are not equal and fair for female testers and I’d like to see that change in my lifetime. I don’t think male testers are the problem at all. After a few curious looks, once we start actually testing or talking about it, in my experience, most testers are supportive and eager to help each other learn regardless of gender. The problem is higher up in the companies where the value testers bring isn’t well understood and diversity isn’t valued for men or women. The top reason we should care about diversity in our testing teams is because the demographic of a computer user is more diverse than ever before.

I think we change this trend by educating hiring managers about the value of skilled and thoughtful testers. There is a misconception than any monkey can perform testing and that only coding requires skill. Designing maintainable test automation that is cost effective over time is also a rare skill that requires talent. Being able to code on a whiteboard does not mean that you have all of the skills needed to design great test automation. If you have a balance of skills on your team, you have options to blend skills, cross-train, and get a better results.

I am an advocate for a balanced approach to testing. To me, that means there is well-designed test automation that is maintainable and saves time and money for the company. Tools are used to support human test efforts. Testers are always learning and expanding their skills, both technically and to gain better user understanding and focus. The goal is always to provide the best experience to the user. I see the most potential for growth in the small to medium-sized software companies who are sincerely trying to adhere to the Agile manifesto. I don’t mean the companies trying SCRUM but keeping the stack ranking and individual competitive rewards, but companies who are really ready to try a team approach complete with team goals and rewards. Companies who are focused on intrinsic motivators and building a strong team can benefit the most from diverse testing teams including some females that fit well with their culture. I think that the large companies are going to continue to see a diminishing percentage of female testers at least until they make some changes.

uTest: What’s the greatest challenge that testers face in 2010?  What about testing managers?
LC: Lack of understanding about the value of testing is the most critical issue facing testers and test managers in 2010. We must find a way to communicate why testing is not optional.

uTest: In your experience, what skills/experience/traits represent the difference between a good tester and a great tester?
LC: Insatiable curiosity, creativity and persistence make a great tester. To be a decent tester, one must be smart, technical, honest and reliable – but that’s not enough to be a great tester.

uTest: How does the proliferation of Android, iPhone and iPad apps affect the mission that faces testers?
LC: I think we need to move beyond the emulators, away from the desk, and test in our cars while driving like the users do. On second thought, maybe we should park.

uTest: Redhead rapid-fire (in no particular order):

• Last book read?  Fun: Hammer of God by Karen Miller (Book 3 of the Godspeaker Trilogy) Testing: Agile Testing by Lisa Crispin & Janet Gregory, Business: Driving by Starlight
• Last movie watched?  How to Train Your Dragon
• Favorite band or album? Covenant
• Favorite Nicholas Cage movie?  Adaptation
• Browser of choice?  Firefox
• Cats or dogs? Cats! I have three. They show up in my slides often.
• Favorite video game? Columns, Tetris (puzzles). I used to play Tele Arena in the BBS days. I always played a healer, unsurprisingly.
• Which mobile device are you packin’? iPhone
• Favorite mobile app? Shazam for fun, but Flashlight I use constantly to find keys and lipgloss in my purse.
• Is it true what they say about redheads (that they make great testers)? It is totally true! We aren’t very sun tolerant, so I carry sunscreen so that I won’t burn.
• Favorite Bach brother? Johann Sebastian (He’s even more famous than James, so far!). I’d choose Jon if I had to pick who would be my manager. I admire the way he gives feedback and I know I’d flourish and grow as his employee. James is such an incredible teacher that his one day session changed my whole life. I’d have to pick him because I must go to another class he’s teaching! I like them both for different reasons. They have my vote as the best tag team, even if they don’t wear spandex and fight in a ring.

uTest: We’ve been fortunate enough to interview James Whittaker, Matt Heusser, James Bach, Michael Bolton, Jon Bach and Scott Barber for previous ‘Testing The Limits’ posts – and now you!  Who should we interview next to help educate and entertain our readers?
LC: Elisabeth Hendrickson should be next, no doubt! Adam Goucher, Chris McMahon, Dawn Cannan and Harry Robinson would be great additions to share their approaches to using testing tools in balance with thoughtful human testing. Lisa Crispin, Janet Gregory, Karen Johnson and Selena Delesie could provide some great insight into agile testing. Liz Marley is on the bleeding edge of testing mobile devices. Lee Copeland and Rob Sabourin bring great experience about finding and training talented testers. Cem Kaner, of course, who I referred to reverently above since he is one of the authors of the “software testing bible” as well as the person who should receive the lifetime achievement award in Software Testing if we had such a thing for all that he’s accomplished so far and continues to bring to testing.

uTest: Put on your prognosticator hat… what will Lanette be doing professionally in 2015?
LC: Making using software a better experience. I belong in quality. I want to contribute something of value to the profession of software testing.  Then again, perhaps I’ll be the one driving the backhoe as you pass your next construction site.

Editor’s note: We hope you enjoyed our Testing the Limits interview with Lanette Creamer. If you would like to suggest a candidate for our next Q&A, send your suggestions to marketing@utest.com.

In part II of our Testing the Limits interview with Lanette Creamer – aka “Testy Redhead” – we cover the need for Exploratory Testing; Matt Heusser and the “rebel alliance” of testing; how to create a popular testing blog; her stance on tester certifications and more from the wide world of QA. Catch up by reading part I, and when you’re done with this one, go check out part III.

uTest: In one of your recent blog posts, you mention Elisabeth Hendrickson’s STAREAST declaration that “Exploratory Testing Is Not Optional.” Why did this statement resonate so strongly for you? Do you think all testing managers should follow Hendrickson’s lead?

LC: As a frequent conference attendee and enthusiastic reader of testing blogs, I’ve seen many ideas about how to improve testing. I’ve been through countless industry trends, such as borrowing manufacturing ideas, extensive measuring schemes, and repeated attempts to automate all testing. The bottom line is exploratory testing works in practice. Not for a few months or years, but it works to find important bugs year after year no matter what other quality trends are happening. It works well side by side with automated checks, manufacturing ideas, and it can be used with session based test management to provide measures and metrics if needed. It is the one constant a tester can go back to and find bugs that impact the user experience. It is the meat in my testing sandwich. (My pun filled humor is the cheese.) To hear Elisabeth acknowledge the importance of exploratory testing in public shows me that agile testing is about more than just automation. Agile testing can be about a balanced approach to overall quality. It resonated with me strongly because it makes me hopeful for the future of testing on agile teams.

I think test managers should evangelize and defend what works well in practice on their teams. My experience has been that exploratory testing is generally undervalued considering how effective and practical it is.

uTest: What’s the deal with this “rebel alliance” thing we’ve been hearing so much about? It sounds subversive – we want in! Seriously though, what’s the mission of this group? Please explain it to our un-initiated readers.

LC: Our Han Solo, and Rebel Alliance leader is Matt Heusser. You may have heard of him? (Editor’s note: Oh yeah, that guy!) I’m a rebellious redhead, and was lucky enough to be invited to join, so why not? A few of us paid for our own travel and lodging so that we could swing StarEast in this economy despite diminished budgets. We pooled our resources to get some discounts, and Matt put together a dinner. He did a tremendous amount of work! Dinner plans grew. Then the force was with us, and a few Jedi mind tricks later, it turned into something incredible.

We ended up with some new lightening talks presented to a small group and having a party in a 68 person conference room that the hotel happened to assign a conference attendee as they were out of normal hotel rooms. Sitting at a table pairing on the dice game led by Michael Bolton was enlightening. Hearing Matt Heusser and Jon Bach share brand new ideas and seeing Shmuel Gerson’s session based test management test tool in action, followed by an in depth description of where combinatorics have the most impact with Justin Hunter from Hexawise was a great experience! Adam Goucher even talked about pirates! Selena recorded many of these 5 minute talks. They are now up on YouTube and you can check them out, including my talk about Cat Collaboration.

The Rebel Alliance isn’t just something you join, it is something you create at your conference. The founding principle is “no one wants to eat a bagel alone”. In practice it is how we joined together and saved hundreds of dollars so that testers without corporate backing were able to see some of the best speakers in the industry. Don’t wait for your company to send you. Send yourself! That’s what I consider the real founding principle of the rebel alliance, but hey, it’s Matt’s invention, so he gets to say it is about having company while eating bagels.

uTest: In our interview with James Bach a few months back, he called you a “great unknown” and “someone to watch.” Blogging seems to have really opened up a few doors for you, but can it do the same for other testers? What’s the secret to getting “discovered” as a testing blogger?
LC: The idea that you can “get discovered” or that I know some secret made me giggle. I’ve been blogging on software testing regularly since 2007. I go to every conference I can possibly go to. My secret to getting discovered as a blogger is to sincerely care about software testing and do good work in public. Interact. Follow up. Be on twitter. Read the books that testers you admire suggest. Write a technical paper and submit entries at some conferences. Are you willing to test in public? Are you proud to talk about your ideas to anyone? I used to send links to my own team at Adobe if I posted a blog, and exactly that many people would look at the post. Blog even if you get 30 visitors a year for a few years. Blog like no one is watching, because at first no one will be, and later because that is what the blogging format is for. You have to care more about software testing than you care about what happens to you personally. That is the one thing the testers I admire have in common.

In up and coming testers, my focus has been mostly on encouraging other female testers who I admire to begin writing and presenting. Some of the women to watch in testing are Marlena Compton, who gave her first conference presentation on visualizing test data last year, and Liz Marley, who presented on Testing Touch Devices at Adobe a few months back. There are some really intelligent Agile testers you should check out if you get the chance. Lisa Crispin, Janet Gregory, Dawn Cannan, and Selena Delesie are teaching me new things about agile testing.

A few guys to watch for are Eric Jacobson I met at StarEast a few weeks back. He’s really articulate, generous, and has a clear point of view on testing. A programmer in testing who interests me is Shmuel Gershon from Intel. His enthusiasm and good nature are infectious!

uTest: What’s your stance on tester certifications? Pro-cert, anti-cert or meh?
LC: Meh. I’d like it if as a community we could agree to a loose guideline that unless you have something groundbreaking and new to say on the topic no one else blog about this. It’s been done (to death, or at least to me wishing for death).

I don’t enjoy conversations about certification, which is why my only blog post about it is a joke. I’d prefer to work for a company that is focused on finding testing talent, not relying on someone else to determine what that is.

Editor’s note: Read part III of the interview now.