In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

• http://kaner.com/pdfs/FITcolloquiumBuildingFreeCoursewareCommunity.pdf

I describe the challenges in a little more graphic detail here:

• http://kaner.com/pdfs/nsfKanerFiedlerBarber.pdf

And I walk through our instructional model in a lot more detail here:

• http://kaner.com/pdfs/DevelopingInstructor-CoachedActivitiesNSF2008.pdf

At university and in practitioner training, my preferred teaching style is to leave the lectures to the videos, to use quizzes to drive students to watch the videos (and read the assigned readings) carefully,  so that they get the basic knowledge that I’m trying to convey, and then to use classroom time (in a live class or most of the course time in an online class) to do tasks based on the materials covered in the videos and readings. The students can apply and argue with the videos and readings in these tasks. I believe we foster a deeper level of learning this way.

As to “curriculum,” I think there are many different, valid, instructional objectives.

In an undergraduate university curriculum, the goal is not to train people for a specific type of future job. It’s to prepare people for the next 10 years of their career, with the expectation that they’ll want to switch jobs, and probably careers, a few times in that period. Undergraduate education has to be about teaching people learning skills as much as technical skills, and enough fundamental education that they are not blocked when they approach materials outside of their core domain. This drives us to a hard-to-optimize conflict between “general education” and technically-focused education in the field of the student’s choice. The broad curricular tradeoffs are too complex for me to understand how to resolve.

In terms of software testing, most universities offer little education directly focused on testing, and much of what is taught at most schools (that teach testing) is thoughtful code verification blended with applied mathematics, rather than skeptical assessment of product quality. Much of this contributes valuably to general education and to programming skill. But not so much to the craft of testing.

Mainly, our field relies on on-the-job experience and commercial training. The training courses rarely get much beyond the basics because the average student hasn’t gotten much past the basics. In addition, there is a critical difference between university education (done well) and commercial training: projects and homework. Students in my Testing 2 class (programmer testing) write take-home final exams of up to 5000 lines of code. Other courses at Florida Tech demand more than this. Most projects are much smaller, but each of them provides experience, not just vocabulary. Active engagement with course material is what fosters deeper learning, and we don’t have time for that in commercial training.

As a result, instead of starting from a well-built-up foundation, testing practitioners learn most of what they learn about testing on the job and in relatively simplistic training courses. They reinvent the field, each for herself or himself, learning the same lessons that we have learned 10, 20 or 30 years before. This is not a path to significant progress in the field.

This low level of common knowledge in the field is convenient for certification courses. They can sell a few days of training (at a high price) that is enough for students to pass a relatively trivial exam. Think of the absurdity of this. The current certifications are essentially advertisements that you (the certificate holder) know as much as someone with no experience and three days of training. Would you hire someone who was proud of that low a level of knowledge?

I keep meeting people who tell me that they send their staff to these kinds of courses because the course teaches everyone a common vocabulary, and our field needs a common vocabulary. I don’t personally think that a common vocabulary has much value. What is more valuable is learning to ask someone else what they mean when they say something.

Thirty years ago, a “big” program was 10,000 lines of code. Today, even cell phones have millions of lines of code. This enormous change in productivity comes from revolutions in the practice of programming and software design. But we aren’t seeing revolutions in testing, not like the change into structured programming or out of structured programming and into object-oriented. No one would think of teaching the programming methods of the 1970’s or 1980’s as state of the art today. But the current certifications in testing (and many of the “applied” university and college courses) seem to me to be based on “best practices” and attitudes that aren’t much different from what I started writing Testing Computer Software to rebel against back in 1983.

We need to make progress on the education problem, or we will dig deeper and deeper into the same old ruts.

uTest: How well are companies adhering to The Software Consumer Bill of Rights (which you wrote in 2007)? Have conditions improved for the average software consumer?

Kaner: The Principles of the Law of Software Contracts have finally been adopted by the American Law Institute. That will probably reflect itself in judicial opinions over the next few years.

In terms of legislation, the United States is in a dysfunctionally polarized period. Very little legislation of any kind is being passed. We might not see useful statutes (laws passed by legislatures) on computer law for several more years. Until then, companies will start acting more responsibly after some successful lawsuits or enforcement actions.

uTest: James Whittaker has made the jump from academia to corporate America.  Will we see you take your considerable talents “across the aisle” at some point (assuming that you aren’t doing so already)?

Kaner: For me, this draws a false dichotomy.

As I said in my last note, programming has been evolving in powerful ways. Testing: not so much. Developing better education/training for testers is one part of the way out of this hole. That’s what I’m currently working on. I’m working at a university because universities house the cutting edge for educational practice.

On an ongoing basis, I ask myself whether I am working on the projects that have the potential to make the greatest positive impact and whether I still think I can realistically make the kind of contribution that is needed. For now, I’m in the right place doing what I think are the right things for me to be doing. Later, that will certainly change. When? How? I don’t know.

Editor’s note: Check back tomorrow for part III of the interview.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

My dissertation focused on measurement of perceptual experiences (a field called psychophysics). We asked questions like “How loud is this sound?” Hearing is an experience. The physics of the sound can help us predict some aspects of that experience, but loudness is about what it sounds like to you, not the physics.

It’s pretty easy to think about how to measure the length of a table. Use a ruler. But what ruler do we use for sweetness? In order to think clearly about this, we need to clarify for ourselves the construct (the idea of) “sweetness.” What does someone mean when they say that X is sweeter (to them) than Y? Then we have to imagine a measuring instrument (like a ruler) that would give us low numbers for not-sweet, high numbers for very-sweet, and reasonable distances between different amounts of sweetness.

“How do we know that this (ruler) measures that (construct)?” is one of the fundamental questions of measurement. It even has a name, “construct validity.” In every field that I’ve studied other than computing (I’ve looked at measurement theory in a lot of fields), when people develop measurements or “metrics” and when they teach metrics, they talk seriously about construct validity. In contrast, construct validity is almost never mentioned in computing. (Do a search for “construct validity” in ACM’s Guide to the Digital Literature! Filter out the papers in business-oriented or medicine-oriented journals. What do you have left?). We have textbooks with dozens (I taught from one text with hundreds) of “metrics” and no practically-applicable system for assessing what these “metrics” measure.

Capers Jones sometimes talks disparagingly about the (claimed) fact that 95% of American software companies have no metrics program. On the surface, this sounds terrible. But what I saw as a consultant was that most software companies have tried a measurement program, or have executives with lots of experience with metrics programs in other companies. The problem is that their experiences were bad. The measurement programs failed. Robert Austin wrote a terrific book, Measuring & Managing Performance in Organizations. When you start measuring something that people do, people will change their behavior to make their scores better. People will change what they do to get better scores on the measurements—that’s what they’re supposed to do. But they don’t necessarily change in ways that improve what you want to improve. Often, the changes make things worse instead of better (a problem commonly called “measurement dysfunction.”) This problem happens more often, and worse, if you use weak, unvalidated metrics. I keep meeting software consultants, especially software process consultants, who say that it’s better to use bad measurements than no measurement at all. I think that’s’ a prescription for disaster, and that it’s no wonder that so many software executives refuse to harm their businesses in this way.

By the way, what if one person repeatedly says X is sweeter than Y and another person says Y is sweeter than X? Are these people wrong about their experiences? Probably not. So is X sweeter than Y? Yes. Is Y sweeter than X? Yes. Have fun putting THAT on your measuring stick.

Jerry Weinberg defines quality as “value to some person.” It’s subjective, like sweetness. Different for different people. Quality is not quantity. But that doesn’t make it any less real.

In terms of the contributions of multiple fields to software, it’s no surprise that Weinberg’s doctoral work was in psychology or that he is so popular a thinker in the testing community. When we evaluate the impact of a new piece of technology on humans, we are doing applied social science research. To say that testing is part of a fundamentally different from the social science is to miss the point of much of testing AND to turn a blind eye to tools developed over centuries that can help us do our work.

Let me switch disciplines–I started working in Silicon Valley in 1983. I watched some excellent companies fail and weak competitors succeed. A striking commonality was that companies would release products with missing features or features that didn’t work, often lying about what they offered. They would gain market share so quickly that they could drive out more honest companies. Most bad companies eventually failed but they cost the marketplace its best suppliers time and again. As a tester, and sometimes as a testing consultant, I was encouraging topnotch staff to hammer products very hard, to expose bugs and argue effectively to get them fixed – to do the right thing by their customers. But the effect of this was that companies were coming to market late with a better product but a worse competitive position. Doing the “right” thing was being punished in the market.

This REALLY troubled me.

I wrote Testing Computer Software over a period of 4 years (1983 to 1986). In many ways, it was an excuse for me to ask people probing questions and think with them about answers.

In terms of markets, I learned that in mature industries, North American society places expectations on companies. We expect companies to advertise honestly, to make products that do what they are supposed to do (maybe not perfectly, but pretty well), to fix their defects for free or give customers their money back, and to allow their products to be analyzed by journalists  who help customers distinguish good from bad products. As a society, we write those expectations into law. When we enforce the laws, we weed out the worst companies and create a safer market for honest businesses and their customers. These rules are less rigid for companies that sell services (e.g. custom programming), but in mature industries, service expectations (and rights) are pretty demanding too.

I came to believe that the core quality problems in software were (and still are) to a very large degree the result of loose enforcement of laws governing fraud, breach of contract, breach of warranty, and service negligence. I learn by doing. To learn more, I became an Investigator (part time, volunteer) in the County of Santa Clara’s Consumer Affairs Department. (Most of Silicon Valley is in Santa Clara County.) After a couple of years, I decided to go to law school and to focus on the law of software quality.

From 1984 to 1987, I worked in a telephone company in a user interface group. A telephone is a user interface to a richly featured, complex network of computers. Our particular system was a really early provider of integrated voice and data. You plugged your computer into the phone to be on the network, rather than into a modem or an Ethernet cable. We also delivered the first phone system that was actually for sale (rather than demonstrated in research laboratories) that had an LCD display.  The menus on our 2 line x 40 character display gave access to 108 voice features and 110 data features.

In our group, we could all program, we all knew a lot about usability. We evolved our system’s user interface over years, in an evolutionary manner (add one or a few features and get them working before adding the next). We moved fluidly between customer focus and code focus, and we modified each other’s code all the time, trying out new ideas for design, which we (and the rest of our company and sometimes our external customers) assessed at the systems level in weekly deliveries of new system builds. It was a great experience.

I don’t want to downgrade the technological challenges of this work. We had to pack a ton of capability into small spaces and have it run very quickly and extremely reliability. The programming side of my work was very difficult. But the programming would have been meaningless without a focus on who the programming was for and it would have been ineffective without an ongoing stream of information about what was working (design and code) and what was not.

At another company (Power Up Software) I was a Software Development Manager. When I started, my task was to design products for small businesses, help plan the marketing of these products, and manage programmers (typically remotely located programmers) who were to write these products. I wanted to turn my products into best sellers but I didn’t know how, so I took a job at Egghead Software, then a successful software retailer, and sold my best competitors’ products to Egghead’s customers. These people didn’t just buy once and disappear. They came back to the store every week or two, buying new things or complaining about old ones. You learn a lot about what makes a product great, and not-so-great, by selling it to people and then dealing with them, face-to-face, over and over. What is quality? Quality is what made my customers happy. What’s a defect? A defect is what made my customers shout at me.

Some people would tell me that quality is conformance to a specification and a defect is a nonconformity. But really, what’s a specification? A specification is a piece of paper, often written by someone who knows nothing about quality.

About those specifications. I did a lot of school learning about software process. I came to Silicon Valley expecting to see lots of well written specifications. What I actually saw were a lot of scribbles on napkins (from the local pub or restaurant), scribbles on flipcharts, and scribbles on whiteboards. They were informal and they were updated as we came to understand our customers, our competitors, and our products better. Many of the most tedious-to-use, most poorly-designed products that I worked on or consulted about were coded to meet a detailed specification. Famous consultants would come visit our companies or teach courses that I sent staff to, and they would tell us that The Right Way to build software was to write everything down at the start and then to build to that specification. But the start is when you know the least about whatever you are creating. As you create it, you learn. Why make your decisions when you don’t know as much as you’ll know tomorrow? Why set up change control processes to make it expensive to fix all the mistakes you made when you were more ignorant than you are today?

Of course, there are reasons to set up change control processes. And there are reasons to develop a clear idea of the product’s scope early on (try designing an appropriate architecture without this).

But as the Extreme Programming advocates came to say much more eloquently than I knew how, knowing the big picture is one thing, but there is rarely a good reason to lock down details until you need to code them. And you cannot call something a “quality” process if it has you refusing to improve a product because it makes the cost of testing too high or the paperwork too expensive.

While I was dev manager (and later, director) at Power Up, I went to law school (you don’t learn much humility there, but it’s good for other things), went to University of California Extension for several courses on technical publications management (to help me work with a tech pubs group that adopted me as their manager), and wrote the second edition of Testing Computer Software.

Hung Nguyen joined me in writing TCS 2.0, but on the condition that I take courses in traditional quality control. He had a B.Sc. in Quality and wouldn’t write with me until I got ASQ-certified in quality engineering. His point was that I had become an expert in informal processes that worked on (relatively) small projects, but in his view, I needed to understand what effective control processes were and why they were important on larger projects and formally-negotiated projects (outsourced development, especially when your customer is the government). That blending helped us make TCS 2 a much better book than the original.

The diversity of my situations taught me powerful lessons about how little I know, how much the other people on the project know that I don’t know, and how complex are the tradeoffs faced by the average project manager. When I work as a tester, I understand that I am playing a supporting role. It is an important role. It calls for skilled and thoughtful work. But as James Bach puts it, I am “the headlights of the project”, not the driver, not the brakes, not the engine, and not the door to the customer. I can change roles. Maybe I can work two or three roles at the same time. But IN MY ROLE AS A TESTER, I don’t understand as much about the rest of the project as many other testers (and test consultants/trainers) seem to think they understand. There is a lot of arrogance in our field. We need to learn more humility.

I graduated law school in 1994, passed the bar exam and was sworn in as an attorney. I went back to the Santa Clara County government, this time to work as a full-time volunteer in the District Attorney’s office. My ideas about fraudulent and unfair trade practices had deepened into a passion. As a DA, I prosecuted about 130 cases, including 5 trials. None of this was white collar crime (e.g., economic crime by corporations) but I learned how prosecutions were run, how crimes were investigated, how cases were organized, negotiated, and occasionally, brought all the way to trial. I didn’t become an expert, but I gained more courtroom experience than most American lawyers gain in their professional lifetime. It was foundational knowledge.

Over the next six years, I spent a huge portion of my time as an advocate for a higher legal standard for software quality. I helped write the Uniform Electronic Transactions Act (UETA), which was adopted by almost all American state governments. The federal government renamed it the ESIGN act (electronic signatures) and applied it to all the states so that we had nationwide uniformity in some critical aspects of electronic commerce. I also tried to help write, and then led the opposition to, what became the Uniform Computer Information Transactions Act (UCITA), a bill that in essence, rewrote contract and copyright rules to benefit large software companies and large companies that embedded software in their traditional products (e.g., cars). UCITA ultimately failed despite the millions spent on it. And one of the most important legal organizations in the United States, the American Law Institute, elected me as a member. The ALI has up to (a maximum of) 3000 members. Most of them are judges or tenured law professors. Some are senior partners in large law firms, a few are famous consumer advocates. I think I was the least experienced lawyer ever elected to ALI and every time I go to one of their meetings, I know (or remember quickly) that every person in that room knows more about the law than I do. But I know a little more about computers-and-law than some of them, so sometimes, my background is useful.

ALI writes books that judges pay careful attention to. In the United States, and in all other countries whose systems evolved from British Common Law, laws passed by legislatures are more like detailed statements of principles. They do not cover every situation. They can’t. So judges have to figure out how a body of laws applies to each particular dispute before them. Across different states, judges might apply the same laws differently. ALI helps organize and make sense of the diversity of decisions. They publish the Restatements of the Law (e.g. the Restatement of Contracts, the Restatement of Torts, etc.) and Principles (e.g. Principles of the Law of Software Contracts). After UCITA failed (only two states adopted it, dozens of states rejected it), ALI let the dust settle for a couple of years and then started writing the Principles of the Law of Software Contracts. One of its most important rules is one that I advocated: a seller of software who knows about a defect of the software but does not disclose the defect to the customer will be held liable for damages caused to that customer by that defect. Note that this does not apply to free software (not sold). And if the seller discloses the defect, it becomes part of the product’s specification (it’s a feature). And if the seller doesn’t know about the defect there is no liability (once customers tell you about a defect, you have knowledge, so you cannot avoid knowledge for long by not testing). ALI adopted it unanimously last year. This is not law, but until the legislatures pass statutes, the Principles will be an important guide for judges. Even though I am a minor contributor to this work, I think the defect-disclosure requirement might be my career’s most important contribution to software quality.

The Association for Computing Machinery recently honored this aspect of my work with its “Making a Difference Award” which is “presented to an individual who is widely recognized for work related to the interaction of computers and society. The recipient is a leader in promoting awareness of ethical and social issues in computing.”

Editor’s note: Read part II of the interview.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

What we are working toward is a browser that is capable of testing applications that run inside it. The idea is to build the test framework into the browser, in our case Chrome, so that test artifacts don’t have to be moved around and installed in test labs and on test machines. Also since the browser and automation are one, problems such as Ajax and self-updating web pages won’t confuse automation. This is true cloud-based test automation. What I want is the ability to launch one of these WTF-equipped browsers and point it at the web site/app I want to test and have all the back-end test case storage, bug reporting, progress reporting etc managed for me, automatically, in the cloud. Tests that I wrote or others wrote can be consumed by the browser and applied en mass. These assets all exist in the cloud at Google, why not make them universally available?

I’m betting you want the same thing.

uTest: You’ve argued that the past two decades of software testing innovation have been “the disposable decades” – too tied to the app, the domain and the technology. What does the next decade look like to you? What will surprise us most?

JW: At GTAC 2008 and EuroStar the same year I gave a talk on the Future of Testing and I am returning to GTAC in October 2010 to give an update on my vision for the next decade. I don’t want to preempt that talk, but I will say that my vision is far more aggressive now than it was then. What I thought was 10-20 years out two years ago will be here in less than 5. Things are changing fast and I am anxious to push them faster. We are working unnecessarily hard to test our applications and it is limiting innovation and creating a market full of sub par software. This has got to stop.

The central theme of my vision of the future is getting rid of this disposable nature of our test artifacts. We are reinventing the wheel every time we test wrt planning, test development, automation, reporting and metrics. Let’s get together and stop this. One of the reasons I am so nice to you guys at uTest is that you are working with the idea of leveraging the cloud and the crowd to perform testing. There is much more collective intelligence involved in what you guys are doing and I like that. I am anxious to get our testing tools in the hands of your crowd.

uTest: In your most recent blog post, you describe a (very near) future where test data will be more like a cellophane wrapper around a web app UI, an overlay that would replace the tiresome querying of multiple databases. Just how close are we to achieving this innovative bug and test case overlay and what would be the significance of such a feat?

JW: You keep trying to pull my GTAC talk out of me prematurely! But let me give a bit of a preview. Test managers need data to make decisions about how to optimize their testing and their testing resources. Most technical test managers I know end up creating a set of SQL queries that they use to get information about features, bugs, test cases, test runs and so forth. All the data is there, it’s just hard to access. Video gamers have a similar problem, there is a lot of information in the game and it is in a lot of different places. Our solution in testing should be the solution gamers use: a heads up display – a single location to surface all the information a tester and test manager needs to make the right decisions during a testing campaign. We are building a heads up display that can consume all this information and use it to guide the hand of the tester and test manager. This is part of what I meant earlier when I said that manual testing needs automated assistance. People doing exploratory testing without a HUD are like gamers who play without one. In the latter case, the gamer’s character gets killed, in the former case the tester is less effective.

uTest: You are a busy man, James, having less and less time to write witty blog posts (which we miss)! What has been keeping you most busy and what have your greatest accomplishments been in the past 12 months?

JW: Busy or just having so much fun I can’t take the time to write a lot? My greatest accomplishment happens every day: people who work for me questioning every aspect of our business and pushing innovative solutions to other testers to improve the way they execute. I had a direct tell me two weeks ago that he couldn’t wait for Sunday to pass so he could come to work on Monday. Nothing else I do compares to that.

uTest: One of our top testers, Madhukar Jain, would like to know: Do you see cloud computing as the next evolving challenge in testing? We would take this one step further and also ask if the cloud represents or enables the fourth gen of testing, or testsourcing, you’ve often alluded to? What is the cloud’s potential to empower the crowd (third gen)?

JW: I see cloud computing as the way to solve the testing problem. And yet again you are forcing me to reveal my GTAC talk prematurely. So be it. Imagine for a moment every test case database, bug repository, build script, static analysis tool, metrics engine, and so forth in the cloud. Every tool and scrap of information a tester will ever need stored in the cloud and surfaced in his HUD when it is most relevant. It’s already stored on some server somewhere, putting it the cloud is a small step. Over time (hopefully a short time) we’ll be able to use this information seamlessly. Instead of the Internet, you have the Testernet capable of answering any question you have about testing and being there to guide you through whatever testing tasks you are performing. Other industries are already doing this. We’re behind the curve in testing.

Do you remember Star Trek The Next Generation? My friend and fellow Googler Dan Massey and I like to hold this series up as the type of technology we want to build. Geordie and Data were constantly “reconfiguring the subroutines” to solve some important task. They could reprogram the phasers, the photon torpedos, the holodeck, anything on the ship! Plug and play “subroutines!” They did this from any console on the ship. They did this without installing any back end repositories or test assets because it was all available in the cloud. They did this without running any all-night build scripts or writing new test cases. The build happened automatically, test assets were applied automatically. The reason Geordie and Data could do this programming so quickly is because they only had to solve the problem they were working on, they would ignore all the back-end build and test nonsense better suited for a computer.

I bet Geordie had one hell of a testing HUD and never had to wade through the complexity of partially automated build systems and SQL queries to find the tests and test data he needs, in fact I bet he didn’t even know anything about these backend systems. They were hidden in the cloud, exactly where they belong, allowing him to concentrate on his programming.

Editor’s note: Read Part II of the interview.

About a month ago, James was again interviewed on the subject of “buccaneer testing”, except this time it was caught on camera by Yvette Francino of TechTarget (while covering StarEast 2010).

Check out this inspiring, two-minute explanation of buccaneer testing:

Happy buccaneer testing, mateys!

In part II of our Testing the Limits interview with Lanette Creamer – aka “Testy Redhead” – we cover the need for Exploratory Testing; Matt Heusser and the “rebel alliance” of testing; how to create a popular testing blog; her stance on tester certifications and more from the wide world of QA. Catch up by reading part I, and when you’re done with this one, go check out part III.

uTest: In one of your recent blog posts, you mention Elisabeth Hendrickson’s STAREAST declaration that “Exploratory Testing Is Not Optional.” Why did this statement resonate so strongly for you? Do you think all testing managers should follow Hendrickson’s lead?

LC: As a frequent conference attendee and enthusiastic reader of testing blogs, I’ve seen many ideas about how to improve testing. I’ve been through countless industry trends, such as borrowing manufacturing ideas, extensive measuring schemes, and repeated attempts to automate all testing. The bottom line is exploratory testing works in practice. Not for a few months or years, but it works to find important bugs year after year no matter what other quality trends are happening. It works well side by side with automated checks, manufacturing ideas, and it can be used with session based test management to provide measures and metrics if needed. It is the one constant a tester can go back to and find bugs that impact the user experience. It is the meat in my testing sandwich. (My pun filled humor is the cheese.) To hear Elisabeth acknowledge the importance of exploratory testing in public shows me that agile testing is about more than just automation. Agile testing can be about a balanced approach to overall quality. It resonated with me strongly because it makes me hopeful for the future of testing on agile teams.

I think test managers should evangelize and defend what works well in practice on their teams. My experience has been that exploratory testing is generally undervalued considering how effective and practical it is.

uTest: What’s the deal with this “rebel alliance” thing we’ve been hearing so much about? It sounds subversive – we want in! Seriously though, what’s the mission of this group? Please explain it to our un-initiated readers.

LC: Our Han Solo, and Rebel Alliance leader is Matt Heusser. You may have heard of him? (Editor’s note: Oh yeah, that guy!) I’m a rebellious redhead, and was lucky enough to be invited to join, so why not? A few of us paid for our own travel and lodging so that we could swing StarEast in this economy despite diminished budgets. We pooled our resources to get some discounts, and Matt put together a dinner. He did a tremendous amount of work! Dinner plans grew. Then the force was with us, and a few Jedi mind tricks later, it turned into something incredible.

We ended up with some new lightening talks presented to a small group and having a party in a 68 person conference room that the hotel happened to assign a conference attendee as they were out of normal hotel rooms. Sitting at a table pairing on the dice game led by Michael Bolton was enlightening. Hearing Matt Heusser and Jon Bach share brand new ideas and seeing Shmuel Gerson’s session based test management test tool in action, followed by an in depth description of where combinatorics have the most impact with Justin Hunter from Hexawise was a great experience! Adam Goucher even talked about pirates! Selena recorded many of these 5 minute talks. They are now up on YouTube and you can check them out, including my talk about Cat Collaboration.

The Rebel Alliance isn’t just something you join, it is something you create at your conference. The founding principle is “no one wants to eat a bagel alone”. In practice it is how we joined together and saved hundreds of dollars so that testers without corporate backing were able to see some of the best speakers in the industry. Don’t wait for your company to send you. Send yourself! That’s what I consider the real founding principle of the rebel alliance, but hey, it’s Matt’s invention, so he gets to say it is about having company while eating bagels.

uTest: In our interview with James Bach a few months back, he called you a “great unknown” and “someone to watch.” Blogging seems to have really opened up a few doors for you, but can it do the same for other testers? What’s the secret to getting “discovered” as a testing blogger?
LC: The idea that you can “get discovered” or that I know some secret made me giggle. I’ve been blogging on software testing regularly since 2007. I go to every conference I can possibly go to. My secret to getting discovered as a blogger is to sincerely care about software testing and do good work in public. Interact. Follow up. Be on twitter. Read the books that testers you admire suggest. Write a technical paper and submit entries at some conferences. Are you willing to test in public? Are you proud to talk about your ideas to anyone? I used to send links to my own team at Adobe if I posted a blog, and exactly that many people would look at the post. Blog even if you get 30 visitors a year for a few years. Blog like no one is watching, because at first no one will be, and later because that is what the blogging format is for. You have to care more about software testing than you care about what happens to you personally. That is the one thing the testers I admire have in common.

In up and coming testers, my focus has been mostly on encouraging other female testers who I admire to begin writing and presenting. Some of the women to watch in testing are Marlena Compton, who gave her first conference presentation on visualizing test data last year, and Liz Marley, who presented on Testing Touch Devices at Adobe a few months back. There are some really intelligent Agile testers you should check out if you get the chance. Lisa Crispin, Janet Gregory, Dawn Cannan, and Selena Delesie are teaching me new things about agile testing.

A few guys to watch for are Eric Jacobson I met at StarEast a few weeks back. He’s really articulate, generous, and has a clear point of view on testing. A programmer in testing who interests me is Shmuel Gershon from Intel. His enthusiasm and good nature are infectious!

uTest: What’s your stance on tester certifications? Pro-cert, anti-cert or meh?
LC: Meh. I’d like it if as a community we could agree to a loose guideline that unless you have something groundbreaking and new to say on the topic no one else blog about this. It’s been done (to death, or at least to me wishing for death).

I don’t enjoy conversations about certification, which is why my only blog post about it is a joke. I’d prefer to work for a company that is focused on finding testing talent, not relying on someone else to determine what that is.

Editor’s note: Read part III of the interview now.

In part I of our interview, we get her thoughts on testers vs. hardware; the idea of “quality advocacy”; why unemployed testers should study The Price is Right;  life as a shift manager at a charity bingo parlor; and much more. When you’re done with one, be sure to check out part II.

uTest: What’s the biggest trend/challenge in testing that no one’s talking about yet?
LC: Testers are breaking out of the office like William Wallace, but with laptops, not swords. How much more affordable is it for a company to buy a great laptop every few years than all sorts of different hardware? Let someone else manage the machines so we can focus on the testing. Of course, this isn’t appropriate for every context, but I’m interested in going beyond multi-boot systems, local images, and to truly getting out of the business of managing hardware. I’m interested in cloud-based imaging. Part of my personal strategy of investing in one laptop that can run multiple operating systems is the temping ability to verify the scope of a bug on one machine. To do that without even rebooting with more built-in logging and debugging tools is really the next step to freedom from hardware and location-reliant testing.

uTest: In the last year, we’ve noticed you blog about the prospect of unemployment. What advice do you have for other testers who find themselves in this situation? Should they just wake up at noon, watch The Price is Right and eat nachos until a hiring manager comes knocking on the door? Or should they try to keep their skills sharp? If it’s the latter, then please elaborate on how to go about this.
LC: The Price is Right can teach you something amazing about interviewing. Have you ever noticed who they pick? It is the most enthusiastic people with the best stories. Come on down, job candidate! You’re the next contestant on The Job is Right. Can you imagine what The Price is Right would be like if they picked a contestant who was just above it all? Laughed at the fabulous prizes? Ignored the host? Win or lose, play the job interview game with style and be memorable. Also, I do like nachos. Layer the cheese and make them in the oven.

Well, rather than bouts of unemployment, I’ve been facing one very long pending layoff. I’ve not yet experienced the unemployment part, so maybe your readers can help me out with their advice when that happens. As a part of the CS5 team, my layoff isn’t effective until June, and it impacted every tester on my current team. The day I first became aware of my pending layoff, I felt a bit powerless. I realized that it was really up to me to decide what to do next. I decided that I wanted to end well and finish the project, and I really wanted to complete my 10 years at Adobe. I am proud of my work for my entire career at Adobe, and that hasn’t changed with my layoff notice.

Here’s what I recommend for those of you working in a job that you know is ending:

• Choose your exit strategy and end well, whatever that means for you.  Software testing is a small world.
• Know that you aren’t alone and continue to reach out to the quality community.
• Decide what to learn. It is your life, your brain, and your chance to choose what you want to become! I’ve had a personal learning syllabus since 2007 when I attended the Self Education for Testers tutorial by James Bach. Create your learning syllabus and hold yourself accountable.
• Be yourself. Be sincere, honest, and upfront about your skills and experience. Finding a job that will grow your skills and be a good fit may take longer, but it will also be mutually beneficial to you and your employer.
• Volunteer. Keep testing. Consider speaking at a conference near you for admission. Want to test for a certain product? Sign up for the beta and go to a user group.

Getting laid off was a real “Come to Kaner” moment for me. Who am I if I’m not a tester? Are testing skills relevant in the marketplace? Is talent at finding bugs totally obsolete? You’ll see in my blog that I started meeting with the people I admire most in testing. Mostly I asked them their story. How did they get past the doubt? Does it mean that I’m not a good tester or that I don’t belong in quality? Every person I spoke with has been through more adversity than I’m going through and it made them stronger in their career eventually. It helped them define boundaries when working with and for companies. Many of them are now independent consultants who decide which jobs they work on. All of them realize that they are not their job. They have an identity and a reputation as a tester that goes well beyond one company. When I hear Jon Bach, does it cross my mind that he’s been laid off before? No. Because every person you know of substance has been laid off. Most of them have been fired from somewhere, quit under non-ideal circumstances, or their startup went under.

Being laid off is an opportunity to transform. At any given job in software you are more likely to be laid off than you are to retire from the same company. Career growth has to be more about growing as a person and less about climbing the corporate ladder at the same company. My upcoming layoff is both terrifying and exhilarating. I’m about to learn something, and it may not be pretty, but I trust it needed to happen and everything is exactly as it should be.

uTest: Jon Bach recently told us that “QA” should stand for “quality assistance” rather than “quality assurance” – do you agree with that sentiment or is he sand-bagging?
LC: I think Jon has a great point. If he could talk to each business person about the value of testing it would help us so much on the biggest challenge we face in 2010. Ok Jon, it’s all you! Hope you have those frequent flyer miles saved up. It’s going to be a long year.

I like to think of myself as providing a bit more than just “quality information”, but also providing “quality advocacy” as the cherry on top of the best information I can possibly provide. My job as a tester includes understanding and advocating for a great customer experience. This means feedback beyond “does this meet requirements” and evaluating “does this meet the customer needs overall based on all that I know and continue to learn about the customer?” If not, there is an issue the team needs to discuss and make a decision on. Delivering a great product is something we can’t assure and can’t do ourselves, but we are a part of the team to make it happen, and we certainly can contribute our ideas and effort to make it a reality.

I’ve been working on a session for Better Software/Agile Development Practices 2010 to share some of my ideas with other testers. I’m speaking on Quality Beyond the Code and Requirements on June 10^th in Las Vegas. It isn’t just a presentation, but also a chance to practice hands on with real examples. It will be my first ever SQE conference as a presenter and by far the biggest conference I’ve presented at so far.

uTest: If Al Gore hadn’t invented the Internet, what would you be doing with your career (bullfighter, gourmet chef, cobbler, explorer)?
LC: My dream job is driving a backhoe, so heavy equipment operator. Before I switched my major to graphic design, I did take a quarter of auto mechanic courses along with beginning arc welding. If needed for survival, I could fix brakes, although I’m a bit out of practice. Prior to software testing I was a shift manager at a charity bingo parlor. I did call the bingo numbers as well as deal pull tabs. If you are in need of a bingo caller/backhoe operator for your next charity game of backhoe bingo, please email me.

Editor’s note: We hope you enjoyed part I of our interview with Lanette Creamer. Go check out part II.

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

uTest: When there are delays in development process, why does testing always seem to take the blame? Is their role just misunderstood? Or is it really the testing team’s fault?  How can companies avoid this seemingly never-ending spiral?

JB: Often, we’re saved until last.  It’s like the novel is written, the movie is produced, but the reviews aren’t in yet, leaving the creators in a heightened state of worry and concern. It’s a lot of stress to be in that position. As testers shine the light on the product, looking for risks and vulnerabilities, the light is really shining on *us*.  We’re being tested just as much as that software, so we tend to be sensitive and aware of being in a critical position as others wait for our findings.

Not being an Agile guy, I’m finding that the values and principles of Scrum, Agile, Lean, and XP may be compelling enough to try.  I’m getting more into that domain and I see merit in the values and principles it is trying to instill.  One of my favorite colleagues (Elisabeth Hendrickson) is coaching me in a gentle way and her stories are enlightening because she lives this stuff. Like me, she used to focus on exploratory testing, but catching up with her recently, she seems to be the whole package — developer, tester, manager, coach — and seems to be immune from that never-ending blame spiral you asked about.  That’s compelling to me.  Some may call her one of those ardent “Agilista” types, but to me, she hasn’t lost her testing spirit or soul.  Though she may agree there’s no silver bullet, she has some great experiences that convince me that with those approaches, the blame spiral is being made irrelevant because of the way testers are more involved, earlier.

uTest: In Half-Baked Ideas for Rapid Test Management (PDF) you used a term called “Agile-fall” (a combination of Agile and waterfall). This seems to be the methodology that most companies follow, yet they always call themselves agile. Is there any shame in being Agile-Fall? And did you coin this term? We couldn’t find it anywhere else.

JB: “Agile-fall” was something I heard at LexisNexis from an awesome PM named Lance Thomas, but in a Google talk in 2005, Elisabeth Hendrickson called it “Scrumfall”, so search on that term and you’ll see that it refers to having the principles associated with Agile (daily stand-ups, sprints, burndown charts, etc.), done in a waterfall-y series of development steps.  Example: Sprint 1: Gather requirements, Sprint 2: Design your tests, Sprint 3, Run those tests, Sprint 4, Fix bugs, Sprint 5 regress those bugs.  There’s no shame in that if that’s what works, and when you’re going through a transition from Waterfall to Agile, that may be the best thing as opposed to a sudden lever-pull one day where you show up and your desk is next to someone else with no walls and there’s a stack of sticky notes and markers on your chair with an email to report to your first standup in 30-minutes.

uTest: It looks like software bugs are partly to blame for the recent Toyota recall debacle. Is this the worst nightmare for testing managers? What else keeps them up at night?

JB: I almost got a Toyota last month, and wish I had, just so I could find a cool defect.  But my nightmares in testing are: “Why didn’t you catch that bug?” Though I know several answers to that, and my favorite colleague Michael Bolton has a rich list of answers to that question, they don’t often overcome the strong emotional attachment that stakeholders have.  And I can’t blame them, really.  The more we call ourselves “Quality Assurance” (like we can guarantee quality), the more they will lean on us to assure them.  I can’t.  But I can certainly *assist* with the notion of  whether something has value to its intended customer.

What keeps me awake is how to know if I’m bringing the right value to the project.  There are a million things I could do, but which ones have the most value right now?  I’m worried I would stumble into an idea too late, or not at all.  That’s why I like heuristics and mnemonics and checklists to kick my mind into gear when I feel stuck or worried.

uTest: What’s the biggest difference between working for a large, mature enterprise and a small young startup (from a testing point of view, that is)?

JB: I’ve worked at both.  At Quardev, which is neither startup or enterprise, we’ve got a mature-but-startup mentality.  Since the word “Quardev” comes from an amalgam of QUality Assistance, Research, and DEVelopment, the balance of those gives us enough diverse opportunities to make it feel like a big company.

From a testing point-of view, test ideas are king no matter what size your company. Some may say the toolset is king, or dev skills are king, but when I’ve worked for start-ups, you don’t need as many signatures on things, and that helps creativity. You might fail faster with those test ideas because you have to do it cheaply, and that’s good.  It’s like projects I’ve been on that use notions of Agile compared to those that don’t.  I see those kinds of projects revealing problems faster because of the way the culture organizes to deliver working software.

At a small startup, it’s often expected that you’ll be brilliant.  That pressure can make work not-so-fun.  Maybe that’s why some of my best ideas came from working in enterprise environments (Microsoft, HP, LexisNexis). In those places, there was a perception that every test idea, method, tactic, and strategy had been tried before.  That carried an unspoken dare (in my view) that begged to be challenged, which few people took on.  For me, brilliance and innovation come when someone counts me out or doesn’t expect it.  I like that.  I like exceeding expectations no matter how big or small the crew.

What sums it up is a TED talk by Ken Robinson, who said: “If you’re not prepared to be wrong, you’ll never come up with anything original.”

uTest: Who plays Jon Bach in the made-for-TV-movie about your career in testing? And what’s the title?

JB: John Schneider, only because it has to be called “The Duke of Hazards.” It also fits because I told my friends in school that I was related to Cousin Daisy (played by Catherine Bach). Not true, but it helped keep the bullies away.

Editor’s note: Here’s part II of the interview.

In Part II of our interview with Google’s Patrick Copeland, we discuss the challenges of managing a global engineering team; rewarding developers with food pellets; the difference between a good tester and a great tester; and why some companies will never launch a high-quality app. By the way, did you miss Part I of our interview?

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

PC: We have a pretty open culture where engineers have a good degree of freedom to invest in areas that interest them. We have an idea called “20% time” where engineers are encourage to invest a portion of their time in projects outside of their primary area.

Does culture help to make better software? I think so, but it’s a tough thing to quantify. On occasion the idea of collecting and measuring “productivity metrics” has come up (ex. lines of code, # of check-ins). We uniformly resist this because similar metrics can result in undesirable outcomes as people try to “succeed” in the system. Our biggest measure of performance, besides the velocity of innovation being delivered to customers, is quarterly peer reviews. This type of peer system reinforces ideas like teamwork, making sure you are having an impact, and building respect from peers. It’s more subjective, but harder to “game” and, ultimately, much more valuable in reflecting the real value and contribution of an individual.

BTW, “Food-pellets” wouldn’t work because our engineers already have free access to gourmet food.

uTest: What’s the difference between a good tester and a great tester?

PC: Good testers can be trained. Among many others, the basics you need to be good are: depth in computer fundamentals, an understanding of the application domain, an strong understanding of the use case, empathy of the user, mastery of metrics, and a focus on the development process.

Great testers, meaning the 90th percentile, are rare and special. Not everyone can be truly great. In my experience great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. From the 100s of interviews I’ve done, “great” boils down to: 1) a special predisposition to finding problems and 2) a passion for testing to go along with that predisposition. In other words, they love testing and they are good at it. They also appreciate that the challenges of testing are, more often than not, equal or greater than the challenges of programming. A great “career” tester with the testing gene and the right attitude will always be able to find a job. They are gold.

uTest: What’s the greatest barrier to designing, developing and launching high-quality products on a consistent basis (deadlines, prioritizing, competing visions, etc)? Said different, why can’t every company launch world-class apps every time?

PC: Every product I’ve worked on required a unique approach. There are so many variables in play and most of them are situational. Some of them you can control almost completely (e.g. technology choices). Many others you can control somewhat. Others still are completely outside of your control (ex. what the competition decides to do).

Some companies try to normalize their development process. One thread common to formal development models are that they focus on a few of the many variables: improving efficiency, predictable process, estimation of quality, or others. Process-heavy development models may work well for manufacturing airplanes and have been successfully applied by some companies, they have been viewed by many developers as burdensome and contrary to the creative nature of writing innovative software. Conversely, “process-less process”, can lead to a heroic culture that’s unable to repeatedly deliver. There needs to be balance.

Consider the physics of flight as an analogy to software process. In addition to reasonable flying conditions and an experienced pilot, the key to getting airborne is having an appropriate balance of factors that match the situation: too much weight or too little thrust can be disastrous depending on the situation. Similarly, teams, products and process all have virtual physics. For instance, adding engineers late in a product cycle doesn’t necessarily provide more lift. Adopting a new process may give a team more thrust momentarily, but may also incur a longer term drag that makes them incapable of innovation.

The popularity of Agile, while not a wholesale rejection of more rigid processes, indicates that developers desire more balance and creativity. Whatever we do to make software higher quality and more predictable to build, we must maintain a balance that encourages the innovative aspects of the art form. We need to motivate smart minds to solve hard problems and deliver rich features to customers. In other words, we need to be adaptable to stay airborne for the long term.

Editor’s note: Part III of our interview with Patrick will be posted tomorrow, so be sure to check back in.

In this month’s

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

• Strategic: Maybe a new feature, but in a market with existing competition (like Android).
• Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
• Customer usage: For example, popular high-traffic applications like GMail.
• Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
• Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

By the way, what I meant is that the surface area of the challenges we face in software has grown exponentially over time. Just in the past few years we’ve witnessed a paradigm shift to cloud computing that stretches our imagination and challenges the limits of software. Our process for developing that software is going through an equally dramatic revolution. We are reconsidering the appropriateness of the lessons we’ve taken from traditional software development companies. Testing teams need to continue to evolve quickly and innovate just keep up with what’s happening in the industry.

uTest: Not too long ago, you added testing guru and author, James Whittaker to the Google team, who is well-known to the uTest community. Why was he such an important addition, and has there been anything that’s surprised you since he came on board?

PC: James has been a great addition to the team. His passion for testing is incredible. The biggest surprise is that he’s having fun being a manager. James accepted our offer without knowing exactly what he was going to do. This is a pretty common situation for new senior people at Google. I remember giving him our offer and he said repeatedly, “I’m not interested in being a manager.” I kept saying, “ok, ok, no manager position for you.” But he’s decided to run the Kirkland and Seattle Test Teams, in addition to several groups in Mountain View. He loves it. I think that might even be a surprise even to him.

uTest: Google seems to break into another market every week, with the new Nexus One phone being the most recent example. How much of your time is spent looking forward to new apps and categories, as opposed to concentrating on existing products?

PC: At Google we have a 70-20-10 rule for investing in technology. While the numbers are not carved in stone, the basic concept is that we invest ~70% on our existing core products, ~20% in related but new categories and ~10% on riskier bets or areas outside our existing core competencies.

Innovation is a high order bit of course. We’re having to do a lot of innovation on supportive infrastructure to keep up with the pace. For instance, we’ve created an infrastructure that allows us to innovate very rapidly while maintaining high quality despite a huge and complex code base. On any given day, there are thousands of projects under development with over 5000 CPU cores producing 65K compilations of build targets, and over 100M executed test suites. The build and test system accommodates the load, and an annual growth rate of 75%, using distributed execution, caching, and work avoidance techniques. The average build and test cycle take just 4 minutes and we are quickly moving toward a model where developers can receive results before they formally check-in code changes (‘pre-submit checking’).

In our model, product teams maintain a build that’s always green (compiling and testing without error). We built the supporting system based on three priorities:

• Speed: All test and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data.
• Feedback: The focus of test systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not fixing code that is broken.
• Simplicity: Engineers should not have to understand how the underlying systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workflow that allows them to take appropriate action.

Thanks for reading Part I of our sit-down with Google’s Patrick Copeland. Check back Tuesday for Part II of our interview, where we’ll cover the challenges of managing a global engineering team; rewarding developers with food pellets; the difference between a good tester and a great tester, and more.

That said, here are a few upcoming testing events:

• “Why is Testing Taking So Long?” – Michael will be giving an interactive talk on this topic at the Markham meeting for TASSQ, the Toronto Association of System and Software Quality, on Wednesday, February 10, 2010.
• The Conference on Free Testing Tools: Michael will be giving a three-day public offering of his Rapid Software Testing course, and will deliver the keynote talk. The event will be sponsored by the Norwegian Computer Society, and will be held in Trondheim, Norway, March 22-26, 2010.

• Testing Experience will host Michael for a three-day public offering of Rapid Software Testing in Berlin, Germany, on March 29-31, 2010.
• Michael will present at the third annual Kitchener-Waterloo Software Quality Association’s conference, on April 21, 2010 in Waterloo, Ontario.
• “I Wouldn’t Have Seen It If I Hadn’t Believed It: Confirmation Bias in Testing”: Michael will host a track session on Thursday, April 29 at the STAR East Conference, Orlando, Florida. The conference runs from Monday through Friday, April 26-30, 2010. (NOTE: uTest will be there, too)
• uTest CEO and co-founder Doron Rueveni will be the keynote speaker at QUEST on Friday, April 23, 2010 in Dallas, Texas. His presentation is titled “Crowdsourced Testing for Mobile Apps.”
• Michael will also be attending Agile Testing Days in Berlin, on October 4-7, 2010.

Hope to see you there!