uTest: You have some great tips on how to handle bloated testing numbers and statistics: “Any number, any statistic is like software. It can be tested.” What other tips can you give testers when it comes to having the courage, diplomacy and patience to slow things down and get to the truth?

JB:  For me, the magic words that often make me feel more courageous, diplomatic and patient are: “I have been fooled before.”

No one will argue with that because it’s true.  Scammers often confess that the hardest person to fool is somebody who says “I can be fooled.”  So many times I’ve been so sure I was right just to meet someone who convinced me differently, sometimes in a matter of seconds.  So now say “I could be wrong”, and use other safety language like: “it could be”, “it seems like”, “it looks as if” and “maybe…”  That way, I don’t feel stupid when I’m shown refuting evidence to my claim.  If you practice that, chances are good that you will appear to be a kung fu master who, after having floored your 50th assailant with your skills, slowly backs out of the room on guard for the 51st.

Remember that testing is a craft.  It involves thinking about how things might be different.  Remembering to say “I have been fooled before” is consistent with that spirit.

uTest:  Testing certs: worthwhile or window dressing?

JB:  The only thing worthwhile about them is the debate they provoke.  Window dressing is an apt metaphor because it’s only meant to enhance a window’s *appearance*.  When there’s a flood or a storm or some other strong test of the window, the dressing often gets destroyed. Outside of the flood, people may prefer the look of the dressing; I just want to be a stronger window.  Passing multiple choice tests about so-called “best practices” don’t do that for me.

uTest: What are some of the blogs/sites that you read to stay on top of the latest QA trends and news?

JB: I’m mostly on Twitter because it leads to links to testing videos, newsletters, news stories, trivia, thoughts, questions, blogs and tools that I haven’t seen before.  For example, yesterday I learned about Scott Berkun and his excellent technology blog, Ken Robinson’s fantastic TED talk on rethinking education, the interesting notion of a testing “playbook”, and the Ignite Conferences (a series of 5-minute talks).  Thank you, tweeps!

The top 10 I follow are: James, Jim Benson, Lanette Creamer, Michael Bolton, Scott Hanselman, Elisabeth Hendrickson, Adam Goucher, Paul Boal, Steve Smith, and yes, I have to say, uTest.   All of these sources turned me on to such a wide variety of sources to edify myself that it’s no longer just a handful of blogs to read now and then.  Blogwise, I have to say James (of course), Scott Berkun, Jim Benson, Joel Spolsky, and Lanette Creamer are most worth my time.

uTest: Testers come from a wide range of backgrounds (and have a wide range of skills) yet they are often talked about as if they were all the same. In your experience, what’s the biggest misconception or stereotype of testers?

JB: That we like to “break stuff.”

Nah, we’re detectives… we FIND stuff.  It’s a treasure hunt, not a smash factory.  In fact, our reports are “findings”, like a scientist or an investigator would call them.

One of the first lessons in my career was my favorite — that software comes to testers *already broken*.  I like that.  I didn’t write the code that broke it (unless I did, like a unit test or a keyword-driven script).

Humans are tested the same way.  When we go on a job interview, the interviewer doesn’t “break” us, they find weaknesses and vulnerabilities that are already there in our programming.  The good news is that we can test ourselves before going into the interview — to find and address issues beforehand so we can show them our best value.

uTest: What qualities, characteristics and experiences do you look for when hiring testers?

JB: At Quardev as a hiring test manager, it is these: cautious, critical, curious, friendly, diplomatic, honest, insightful, thoughtful.  I want candidates to tell me about a cool bug they found, or give me their best test idea.  I want them to make me think. I want them to inspire me and make *me* curious*, even though the thing I give them to test when they come in has been tested by over 500 different testers over the last 10 years.  I’m not naive enough to think I’ve seen it all, because one in 20 testers will find something new or will put ideas together in ways I hadn’t thought of.

uTest: What advice would you give to new testers who want to stand out from the crowd and become tomorrow’s testing leaders/gurus/pundits?

JB: Give a talk. Practice in front of a high school class. Take an idea or a strong opinion you have and actually learn more about it, then present it.  Do a lightning talk at the closest Ignite Conference to you.  Read a testing book and email the author your comments.  Comment on blogs and sign your real name.  Question the gurus and the pundits. Call them out if you think their work is crap. Talk about your failures.  Tell us your *experience*, no matter how amateur you think it may be.  Could be you just invented the next cool method to try. It was like that for me with Open-Book Testing when I was on Microsoft Flight Sim.  It was a half-baked idea I had and now it’s blossomed into something that I keep in my intellectual briefcase to show prospective clients one way to teach, guide, and evaluate test teams.

uTest: One of the hottest areas of growth that we’ve seen at uTest is in the area of mobile app testing.  What unique challenges does mobile present to a testing manager?

JB: Finding the right minutes/text plan?  Those really add up when you’re testing!

Actually, I think uTest is in a great position to address the biggest challenge I would face if I was managing a mobile project right now – configuration testing.  I would love to have 1,000 testers at my disposal doing a variety of risk-based exploratory charters on their devices with all kinds of third-party apps installed, making calls on different service plans, texting, taking pictures, playing video, Tweeting, browsing, and maybe doing all of those actions at once from places around the world.

By the way, I read Patrick Copeland’s answer when you asked him the same question and really think he summed up the challenges – diversity of devices, input-output simulations, carrier networks — things that you’d have to be very technical, very savvy, and very organized to test.  But on top of all that, there’s the challenge of if you actually found a bug.  With all of those variables I mentioned, where is the real bug?  You may be just reporting the failure, so you’d have to have some good debugging and diagnostic tools to help you be confident that your concern was represented to those who could (or would) take action on it.

uTest:  If Al Gore hadn’t invented the interwebs, what would you be doing today (answers related to software or technology not allowed!)?

JB: I’d likely be a best-selling author, have a full head of hair, and would have retired at 18 because I would have invented Twitter in 1986 instead of messing around on the Apple II for hours on end.  Before Twitter, before Windows, before the Apple II, our family had the “chain letter” – a packet of letters sent from one member of my family to another, adding new content with each mailing until all 6 of us had caught up with each other’s news in the circle.  When the oldest person got it, they would add new content and mail it to the youngest, and round it would go again, replacing old content with new.  It was a great system and I would have turned that into Twitter as a system of snail-mailed 3×5 cards to everyone on the planet.

—————————————–

uTest: Rapid-fire Jon Bach pop trivia – Last book read?  Last movie read?  Last concert attended?  Favorite sport and team?  Favorite band or album?  Browser of choice?  What kind of cell phone are you carrying?  Blu-ray or DVD?  Paper or plastic?

JB: Last book read? You mean cover-to-cover, not skimming?  Ugh.  Oh!  Last night to my three year old: “How do Dinosaurs Say Good Night?”, after which, I dove back into “Extreme Programming Explored.” Actually, I think I was over-tired last night and got it backwards, which is why I suspect she woke up this morning talking about refactoring, code smells, and collective code ownership and why I dreamt about being a Gallimimus.

Last movie read? The screenplay for “Butch Cassidy and the Sundance Kid” by Goldman.

Last movie seen: the unfortunate and annoying “Where the Wild Things Are”

Concert: Sting, 1989 in Portland, Maine;

Sport: Football (Seattle Seahawks);

Album: Rush, Subdivisions;

Browser: Firefox;

cell: iPhone 3G;

DVD;

Plastic, because at least they spice up the boring look of those landfills!

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

uTest: When there are delays in development process, why does testing always seem to take the blame? Is their role just misunderstood? Or is it really the testing team’s fault?  How can companies avoid this seemingly never-ending spiral?

JB: Often, we’re saved until last.  It’s like the novel is written, the movie is produced, but the reviews aren’t in yet, leaving the creators in a heightened state of worry and concern. It’s a lot of stress to be in that position. As testers shine the light on the product, looking for risks and vulnerabilities, the light is really shining on *us*.  We’re being tested just as much as that software, so we tend to be sensitive and aware of being in a critical position as others wait for our findings.

Not being an Agile guy, I’m finding that the values and principles of Scrum, Agile, Lean, and XP may be compelling enough to try.  I’m getting more into that domain and I see merit in the values and principles it is trying to instill.  One of my favorite colleagues (Elisabeth Hendrickson) is coaching me in a gentle way and her stories are enlightening because she lives this stuff. Like me, she used to focus on exploratory testing, but catching up with her recently, she seems to be the whole package — developer, tester, manager, coach — and seems to be immune from that never-ending blame spiral you asked about.  That’s compelling to me.  Some may call her one of those ardent “Agilista” types, but to me, she hasn’t lost her testing spirit or soul.  Though she may agree there’s no silver bullet, she has some great experiences that convince me that with those approaches, the blame spiral is being made irrelevant because of the way testers are more involved, earlier.

uTest: In Half-Baked Ideas for Rapid Test Management (PDF) you used a term called “Agile-fall” (a combination of Agile and waterfall). This seems to be the methodology that most companies follow, yet they always call themselves agile. Is there any shame in being Agile-Fall? And did you coin this term? We couldn’t find it anywhere else.

JB: “Agile-fall” was something I heard at LexisNexis from an awesome PM named Lance Thomas, but in a Google talk in 2005, Elisabeth Hendrickson called it “Scrumfall”, so search on that term and you’ll see that it refers to having the principles associated with Agile (daily stand-ups, sprints, burndown charts, etc.), done in a waterfall-y series of development steps.  Example: Sprint 1: Gather requirements, Sprint 2: Design your tests, Sprint 3, Run those tests, Sprint 4, Fix bugs, Sprint 5 regress those bugs.  There’s no shame in that if that’s what works, and when you’re going through a transition from Waterfall to Agile, that may be the best thing as opposed to a sudden lever-pull one day where you show up and your desk is next to someone else with no walls and there’s a stack of sticky notes and markers on your chair with an email to report to your first standup in 30-minutes.

uTest: It looks like software bugs are partly to blame for the recent Toyota recall debacle. Is this the worst nightmare for testing managers? What else keeps them up at night?

JB: I almost got a Toyota last month, and wish I had, just so I could find a cool defect.  But my nightmares in testing are: “Why didn’t you catch that bug?” Though I know several answers to that, and my favorite colleague Michael Bolton has a rich list of answers to that question, they don’t often overcome the strong emotional attachment that stakeholders have.  And I can’t blame them, really.  The more we call ourselves “Quality Assurance” (like we can guarantee quality), the more they will lean on us to assure them.  I can’t.  But I can certainly *assist* with the notion of  whether something has value to its intended customer.

What keeps me awake is how to know if I’m bringing the right value to the project.  There are a million things I could do, but which ones have the most value right now?  I’m worried I would stumble into an idea too late, or not at all.  That’s why I like heuristics and mnemonics and checklists to kick my mind into gear when I feel stuck or worried.

uTest: What’s the biggest difference between working for a large, mature enterprise and a small young startup (from a testing point of view, that is)?

JB: I’ve worked at both.  At Quardev, which is neither startup or enterprise, we’ve got a mature-but-startup mentality.  Since the word “Quardev” comes from an amalgam of QUality Assistance, Research, and DEVelopment, the balance of those gives us enough diverse opportunities to make it feel like a big company.

From a testing point-of view, test ideas are king no matter what size your company. Some may say the toolset is king, or dev skills are king, but when I’ve worked for start-ups, you don’t need as many signatures on things, and that helps creativity. You might fail faster with those test ideas because you have to do it cheaply, and that’s good.  It’s like projects I’ve been on that use notions of Agile compared to those that don’t.  I see those kinds of projects revealing problems faster because of the way the culture organizes to deliver working software.

At a small startup, it’s often expected that you’ll be brilliant.  That pressure can make work not-so-fun.  Maybe that’s why some of my best ideas came from working in enterprise environments (Microsoft, HP, LexisNexis). In those places, there was a perception that every test idea, method, tactic, and strategy had been tried before.  That carried an unspoken dare (in my view) that begged to be challenged, which few people took on.  For me, brilliance and innovation come when someone counts me out or doesn’t expect it.  I like that.  I like exceeding expectations no matter how big or small the crew.

What sums it up is a TED talk by Ken Robinson, who said: “If you’re not prepared to be wrong, you’ll never come up with anything original.”

uTest: Who plays Jon Bach in the made-for-TV-movie about your career in testing? And what’s the title?

JB: John Schneider, only because it has to be called “The Duke of Hazards.” It also fits because I told my friends in school that I was related to Cousin Daisy (played by Catherine Bach). Not true, but it helped keep the bullies away.

Editor’s note: Here’s part II of the interview.

In the third and final installment of our Testing the Limits interview with Google’s Patrick Copeland, we get him to share his advice for youngsters who want to reach tech executive status; his programming language of choice; mobile testing challenges and his New Year’s resolutions (for testing of course). In case you missed our previous interviews, here’s Part I and Part II.

uTest: What advice would you give to young software turks who aspire to become a senior tech exec at a global powerhouse?

PC: Here’s my advice:

1. Be technical: At Google all of our engineering execs have very strong technical backgrounds. They were programmers and many of them can – and do – still program today. Advice: remember your computer science and stay sharp.
2. Be a connector and innovator: One exec commented that his job could be described as being “a very expensive email router.” Get connecting within your company and establish a network of strong peers.
3. Drive your career: Don’t wait for someone to recognize you. Inform people about your plans and successes. You are your own best billboard. People will create a myth about you, like it or not. If you are smart you’ll help them create the myth you want them to have.
4. Play to your strengths: Pick projects that you are passionate about and where you can rock worlds.
5. Bite the hand that feeds you (a little): Don’t be a jerk, but make sure you don’t go passively along with bad ideas. A weak willed tester is a bad thing. Make your concerns known…and know when to fold.
6. Go where there’s opportunity for growth: Not all projects or companies can offer you an ability to grow. Move when you think you are topping out. The biggest mistake I’ve seen is staying too long in a place that hinders growth.

uTest: Google is booming in the mobile apps space; what QA testing challenges do mobile apps present that web or desktop apps do not?

PC: Where do I begin?

A good starting place is the huge diversity of devices. For instance, significant variations in HW and SW: OS, memory, processors, screen size, and input methods. Just having access to a fraction of the devices on the market for testing purposes is a challenge.

On top of that, if you want any automation, you have the problem of simulating input and capturing the actual output on the device. We needed to develop tools that allow up to test and build against multiple platforms. Normalizing the testing is tough. We debated record and playback vs writing abstraction layers vs other methods. All can be brittle in the extreme. We also needed to think about the long term cost of maintaining tests suites on fast moving SUTs.

A couple other issues are the carrier networks and the numerous languages. As I said, I could go on for a long time, but I believe I hit some of the major pain points. Even with all of the automation in the world, a community based test approach can be complimentary.

uTest: Is there a particular programming language to which you are loyal (PHP, Java, Ruby, etc)?

PC: Nope. Not really. At Google the most common ones are Java, JavaScript, Python, Perl, and C++. Perhaps a personal failing, but I still think in C and my code ends up looking kind of C-ish even if it’s in another language. My preference also depends on the thing I’m trying to do, obviously each has strengths and weaknesses.

On a similar topic, I recently met famed computer language innovator, Niklaus Wirth at the last GTAC conference in Zurich. He had in interesting comment on testing and quality, “The experience, judgment, and intuition of programmers who have survived the rigors of testing are what make programs of the present day useful, efficient, and correct.” In other words, you can’t test quality in and quality comes from experience.

uTest: Any new year’s resolutions for Google’s testing & QA efforts?

PC: Yes. Every year I write up a “one pager” describing what I think we need to do. I try to stay focused on general productivity and not specifically testing. Testing is one tool among many we use to make software high quality and delivery faster. We are doing well on many levels, but we aspire to do even better. Here are a few of the high level bullets:

• Greater release velocity. We apply appropriate use of automation, tools, process, and information that solve Google challenges. We introduce and use metrics that fit naturally into the product cycle. We are introducing key metrics that are actionable and representative of the health of the project that allow team members to understand issues and take action quickly.
• Cohesive tools. We’ve built an impressive set of tools that have made significant impact. In our next stage, we need to better integrate some of the data and tools and focus on accelerating the end to end developer workflow. This includes identifying bottle necks that slow people down and keeping an eye on innovative ideas coming from product teams. And of course, continual improvement of build and test speed.
• Driving improvement upstream. We will take advantage of our ability to make good ideas useful across the company. We use our broad view of projects to push adoption of ideas and methodologies that work. We look for ways to compliment existing product strategy by identifying changes where the adoption is in a project’s self interest. We constantly evaluate our own practices by looking at their effectiveness in relation to customer and business needs.

uTest: You went to school at University of Arizona, and USC; you’ve worked at Microsoft and Google; are you a lifetime west coaster or has it just worked out that way? Also, Dodgers or Giants (or D-backs)?

PC: Are there any good East Coast teams? I heard it was so foggy the other day that the Washington Nationals couldn’t even see who was beating them.

They may not be sports teams, but I’m a fan of the testing teams we have on the east coast: NYC “Bubble Sorts”, Boston “Red SOX”, and Pittsburgh “Open Sourcers”. BTW, we just hired a Test Director on the East Coast named John Turek who’s looking for a few good players.

BTW, I actually have lived in other places. My family lived in Maryland and Tallahassee for a bit. I also lived in Madagascar, which is definably on the East Coast (of Africa).

uTest: What sites, blogs or magazines do you read to stay ahead of the curve in the areas of business, technology, etc?

PC: Track and Field, Cosmo and, occasionally, Monster Truck Weekly . Seriously, I have a bunch of custom news feeds on certain topics, like Google, our competitors, and testing. I also read wsj on line and at least one interesting industry technical paper per quarter.

Editor’s note: We hoped you’ve enjoyed our interview with Patrick Copeland. If you have suggestions for our future Testing the Limits guests, we’d love to hear them.

In Part II of our interview with Google’s Patrick Copeland, we discuss the challenges of managing a global engineering team; rewarding developers with food pellets; the difference between a good tester and a great tester; and why some companies will never launch a high-quality app. By the way, did you miss Part I of our interview?

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

PC: We have a pretty open culture where engineers have a good degree of freedom to invest in areas that interest them. We have an idea called “20% time” where engineers are encourage to invest a portion of their time in projects outside of their primary area.

Does culture help to make better software? I think so, but it’s a tough thing to quantify. On occasion the idea of collecting and measuring “productivity metrics” has come up (ex. lines of code, # of check-ins). We uniformly resist this because similar metrics can result in undesirable outcomes as people try to “succeed” in the system. Our biggest measure of performance, besides the velocity of innovation being delivered to customers, is quarterly peer reviews. This type of peer system reinforces ideas like teamwork, making sure you are having an impact, and building respect from peers. It’s more subjective, but harder to “game” and, ultimately, much more valuable in reflecting the real value and contribution of an individual.

BTW, “Food-pellets” wouldn’t work because our engineers already have free access to gourmet food.

uTest: What’s the difference between a good tester and a great tester?

PC: Good testers can be trained. Among many others, the basics you need to be good are: depth in computer fundamentals, an understanding of the application domain, an strong understanding of the use case, empathy of the user, mastery of metrics, and a focus on the development process.

Great testers, meaning the 90th percentile, are rare and special. Not everyone can be truly great. In my experience great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. From the 100s of interviews I’ve done, “great” boils down to: 1) a special predisposition to finding problems and 2) a passion for testing to go along with that predisposition. In other words, they love testing and they are good at it. They also appreciate that the challenges of testing are, more often than not, equal or greater than the challenges of programming. A great “career” tester with the testing gene and the right attitude will always be able to find a job. They are gold.

uTest: What’s the greatest barrier to designing, developing and launching high-quality products on a consistent basis (deadlines, prioritizing, competing visions, etc)? Said different, why can’t every company launch world-class apps every time?

PC: Every product I’ve worked on required a unique approach. There are so many variables in play and most of them are situational. Some of them you can control almost completely (e.g. technology choices). Many others you can control somewhat. Others still are completely outside of your control (ex. what the competition decides to do).

Some companies try to normalize their development process. One thread common to formal development models are that they focus on a few of the many variables: improving efficiency, predictable process, estimation of quality, or others. Process-heavy development models may work well for manufacturing airplanes and have been successfully applied by some companies, they have been viewed by many developers as burdensome and contrary to the creative nature of writing innovative software. Conversely, “process-less process”, can lead to a heroic culture that’s unable to repeatedly deliver. There needs to be balance.

Consider the physics of flight as an analogy to software process. In addition to reasonable flying conditions and an experienced pilot, the key to getting airborne is having an appropriate balance of factors that match the situation: too much weight or too little thrust can be disastrous depending on the situation. Similarly, teams, products and process all have virtual physics. For instance, adding engineers late in a product cycle doesn’t necessarily provide more lift. Adopting a new process may give a team more thrust momentarily, but may also incur a longer term drag that makes them incapable of innovation.

The popularity of Agile, while not a wholesale rejection of more rigid processes, indicates that developers desire more balance and creativity. Whatever we do to make software higher quality and more predictable to build, we must maintain a balance that encourages the innovative aspects of the art form. We need to motivate smart minds to solve hard problems and deliver rich features to customers. In other words, we need to be adaptable to stay airborne for the long term.

Editor’s note: Part III of our interview with Patrick will be posted tomorrow, so be sure to check back in.

In this month’s

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

• Strategic: Maybe a new feature, but in a market with existing competition (like Android).
• Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
• Customer usage: For example, popular high-traffic applications like GMail.
• Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
• Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

By the way, what I meant is that the surface area of the challenges we face in software has grown exponentially over time. Just in the past few years we’ve witnessed a paradigm shift to cloud computing that stretches our imagination and challenges the limits of software. Our process for developing that software is going through an equally dramatic revolution. We are reconsidering the appropriateness of the lessons we’ve taken from traditional software development companies. Testing teams need to continue to evolve quickly and innovate just keep up with what’s happening in the industry.

uTest: Not too long ago, you added testing guru and author, James Whittaker to the Google team, who is well-known to the uTest community. Why was he such an important addition, and has there been anything that’s surprised you since he came on board?

PC: James has been a great addition to the team. His passion for testing is incredible. The biggest surprise is that he’s having fun being a manager. James accepted our offer without knowing exactly what he was going to do. This is a pretty common situation for new senior people at Google. I remember giving him our offer and he said repeatedly, “I’m not interested in being a manager.” I kept saying, “ok, ok, no manager position for you.” But he’s decided to run the Kirkland and Seattle Test Teams, in addition to several groups in Mountain View. He loves it. I think that might even be a surprise even to him.

uTest: Google seems to break into another market every week, with the new Nexus One phone being the most recent example. How much of your time is spent looking forward to new apps and categories, as opposed to concentrating on existing products?

PC: At Google we have a 70-20-10 rule for investing in technology. While the numbers are not carved in stone, the basic concept is that we invest ~70% on our existing core products, ~20% in related but new categories and ~10% on riskier bets or areas outside our existing core competencies.

Innovation is a high order bit of course. We’re having to do a lot of innovation on supportive infrastructure to keep up with the pace. For instance, we’ve created an infrastructure that allows us to innovate very rapidly while maintaining high quality despite a huge and complex code base. On any given day, there are thousands of projects under development with over 5000 CPU cores producing 65K compilations of build targets, and over 100M executed test suites. The build and test system accommodates the load, and an annual growth rate of 75%, using distributed execution, caching, and work avoidance techniques. The average build and test cycle take just 4 minutes and we are quickly moving toward a model where developers can receive results before they formally check-in code changes (‘pre-submit checking’).

In our model, product teams maintain a build that’s always green (compiling and testing without error). We built the supporting system based on three priorities:

• Speed: All test and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data.
• Feedback: The focus of test systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not fixing code that is broken.
• Simplicity: Engineers should not have to understand how the underlying systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workflow that allows them to take appropriate action.

Thanks for reading Part I of our sit-down with Google’s Patrick Copeland. Check back Tuesday for Part II of our interview, where we’ll cover the challenges of managing a global engineering team; rewarding developers with food pellets; the difference between a good tester and a great tester, and more.

In the third and final part of the Michael Bolton trilogy, we cover advice for new testers, his hypothetical banishment from Software Land, the blogs he reads and more. Did you miss our earlier interviews? Here’s Part I and Part II.

uTest: Hypothetical: You’ve been banished from testing – nay, ALL software-related activities – for the rest of your days. What will you to earn a living?  What hobbies would you pick up to fill the intellectual void?

MB: Who knows?  For fun, I’d keep playing mandolin, probably. Teach, maybe. Write. I’ve worked in theatre stage management, been a book-keeper, tended bar, worked in a comedy club. In high school I worked in mail rooms during the summer. Whatever I’ve picked up in life, it was because something needed to be done and I was there to do it.  If it didn’t seem like much at first, I started to learn about it quickly. When you invest a little bit of effort to figure out your job, you learn how to makes it faster and better and more interesting. It turns into this great feedback loop. Any job can be more fun when you set out to master it.

uTest: Tell our testing community something about you that your most avid readers don’t know.

MB: While walking through the woods on an island near Vancouver recently, I found myself being quiet and brief, which I like from time to time. Practically nobody knows that.

Lots of people probably don’t know how much I’m eager to help people out. All of my work—courses, articles, conference presentations, this interview—comes with lifetime free technical support. Have a question? Just ask. I might not answer right away—supporting the family with paying work takes precedence over supporting the community—but I’ve never knowingly turned anybody down, so if I don’t answer right away, be persistent. James Bach makes the same offer, by the way. We’ve found that it’s a great way not only to help people, but also to explore problems and come up with solutions and learn things that can help our clients.

uTest: If you were talking to a newbie tester, what advice would you give him or her to set their professional journey off on the right foot?  How about for a 10-year veteran tester?

MB: To both: learn, and don’t stop learning. Practice in your work, but also on open source projects, and in your life generally. Study testing, but note that the great ideas are likely to come from outside the field—at least the narrow vision of the field that the process enthusiasts and “certificationists” present. Testing is a marvelously interdisciplinary craft.  One implication is that whatever you bring to the table from your life and your experience and your education can inform new ideas about testing.  Join or start communities of skilled testers in your area. Join online groups; I like the Software Testing Mailing List and the Software Testing Club particularly. Learn about your craft and build your reputation by asking questions and seeking answers. As with testing itself, the really excellent work starts with asking questions.

uTest: In your opinion, what characteristics, skills or experience separate a good tester from a great tester?  How about in a testing manager?

MB: My colleagues and I have been talking about skills of great testers lately.  (In term of public postings, James and Jon Bach have podcast a nice pair of conversations which you can find here. James also did a CodingQA podcast here. One key skill is the ability to identify context quickly, which helps to avoid all kinds of pathologies—wasting time, failing understand the mission, testing the wrong stuff, investigating unimportant risks, and so forth.  Get the context right, and you’ve got a better shot at doing excellent testing. We teach people to consider and ask questions about the test lab, the test team, the development model and the programmers, the requirements, and the mission to help with that.

Another skill—which we also teach—is something that James has named test framing, which is the same kind of thing in the other direction. It’s the high-level skill—a skill set, really—associated with linking the context, your motivation, the actions you perform, the observations you make, the inferences you draw, and the story you tell about all that stuff.  It’s the ability to follow and describe the threads of logic forward and backward, from the mission to the outcome of the test and back again. It’s the ability to competently address and answer the questions “How, specifically, does this test relate to your mission?” as you prepare to test and “How does the outcome fulfill your mission?” after you’re done.

We’ve seen some good testers whom we can’t quite call great, because they have a hard time with some aspect of that skill set. They seem to lose the ability to link premises and conclusions at some point.

I think there’s a strong relationship to the skills of a great test manager there. A great test manager needs to be able to do the same kind of framing with respect to the test project and the test strategy. A manager that can frame the project can negotiate for people and resources more effectively, can train and deploy her people more efficiently, can explain the work of her team, can defend herself skillfully, can adapt to rapidly changing circumstances. That’s really important, because most projects are messy and confusing. If they weren’t we wouldn’t need testers.

Here’s an example: Test managers get asked questions like “When is testing going to be done?” That question is hard for some test managers because it’s not the test manager’s job to make the decision. They feel pressured to provide an answer, but they can’t frame the problem. A great test manager can deal with that pressure. First: Problems in a product get introduced almost entirely by people other than the testers. Second: Neither those people nor the testers know what problems are there, or where they are. Third: Testing is done in cycles that reveal as-yet unknown information about problems that may or may not need to be fixed. Fourth: The decision to fix the problems depends on the programmers and on the product owner. Fifth: Testing isn’t done until the problems in the product are fixed to the product owner’s satisfaction.

If those things are true (and they are), the amount of investigation and reporting and bug verification testers have to do is unpredictable until you’ve actually tested the product, made decisions about what’s to be fixed, and fixed the problems. Well… it’s predictable, but the prediction doesn’t have any real validity, because there are too many variables that could throw the whole prediction off. “Accurate” predictions in this business come mostly self-fulfilling prophecy—and from ignoring changes in scope or staffing or schedules or budgets or tools that just happened to make the prediction come true. If you set only a couple of data points for your targets, all the other stuff can change without people noticing much.

So testing cycles can be scheduled, but testing cannot be done until the last problem deemed important by the product owner is fixed, by the programmers, to his satisfaction. The great test manager can help the product owner to identify and decide upon what information he needs to know about the product. His goal is to be satisfied in his technical awareness of the product—in relation to the business needs—such that it makes business sense to ship or deploy the product. The test manager and the product owner collaborate in identifying the testing tasks that might help in providing that information.  That turns into the mission for that cycle of testing, test manager and her team can then set about fulfilling the mission—and adapting it when the product owner changes his mind, or when other things don’t go according to plan.  But in the end, the question of when testing will be done depends very little upon the test manager, and far more upon the programmers, the product owner, and a bunch of information that is presently unknown.  A great test manager can explain that in a way that satisfies the product owner, without being trapped into an unachievable commitment.

uTest: At STPCon, you also said that “Testing can be assisted by automation, but automation is not the arbiter of value. We are.” Does this mean that testing will never be replaced by machines? And does this include cyborgs?

MB: Automation is a tool; it’s a means to an end.  It’s a medium, in the McLuhan sense—it extends or enhances or enables or accelerates or intensifies some human capability, but it doesn’t replace that capability. Writing and email and wikis extend our ability to communicate, but they don’t replace talking.

Machines can’t make decisions about what’s best for us, because they’re only extensions of some human faculty. Value is about what humans want, what they like, what they’re willing to pay or do. People don’t even like other people making those decisions for them.

uTest: You’re an active blogger, teacher and speaker – do you enjoy these things more than testing itself?

MB: That’s a little like asking about someone’s favorite Beatle album. I love testing, I love teaching it, I love talking about it. It’s a feedback loop. I wouldn’t be able to do the other stuff if I didn’t test, and the more testing and research about testing that I do, the more I find that’s interesting to talk about. When we teach Rapid Testing, we offer a free day of hands-on testing and coaching. We offer that separately too, and it’s always a source of great learning and great fun.  I’d be delighted for clients to involve me more directly more often.

uTest: Who do you read in the world of testing (blogs, sites, journalists, et al)?

MB: I have a very expansive view of the world of testing. To me, the world of testing is the world of human knowledge and inquiry, of critical thinking, of science. That includes software-related stuff, but it includes a ton of other disciplines too:  medicine (Jerome Groopman, Atul Gawande), aviation (Patrick Smith on Salon.com), education and other social issues (David Cayley), advertising and marketing (Terry O’Reilly), economics (James Surowiecki, John Cassidy, Herbert Simon), security (Bruce Schneier), risk (Nassim Taleb).  All those folks and their specialties are Googleable. I really like Malcolm Gladwell’s writing, too; he’s harder to pigeonhole, but he’s good at finding intriguing sides to the basic story. I first encountered most of the writers via The New Yorker and most of the broadcasters via CBC Radio. I like starting with very accessible stuff, and then if something interests me, I follow the lines of references, bibliographies and notes.

A couple of testing books that came out recently have few notes and no bibliography. That’s like publishing malpractice, so far as I’m concerned. You’ll notice that I’ve mentioned a lot of names already. That’s specifically so that people who are reading this interview can check out the stuff that’s fascinated me and helped to spark new ideas. I hope it helps to inspire people to talk about their own discoveries. Thank you for this opportunity to chat about mine!

Editor’s note: We hope you’ve enjoyed our three-part chat with Michael Bolton. If there’s a question you wanted us to ask that we didn’t get to, you can ask Michael directly at: michael@developsense.com.

Have suggestions for our next interview? Send those along as well!

In the first part of our interview with Michael Bolton, we grilled him on the emergence of the Weekend Testers, sensible metrics, Michael Bolton the pop star and a bunch of other topics. In part “deux” of our interview, we tackle the necessity of tester passion, how emotions affect testing, and the greatest threats to the profession. Check back tomorrow for the final segment.

uTest: There’s a lot of passion amongst testing thought leaders about the best way to test, or the best way to manage or train testers.  Often that passion overflows into heated debates.  How can this passion best be channeled to improve the state of testing?

MB: First of all, we should welcome debate, and we should welcome skilled argumentation as part of the art of construction and practice of persuasion. I’ve found, though, that it helps to remember that we’re exploring and challenging ideas. That means it’s good not to get too personally invested in certain ideas, because we’re always learning more, and because changes in context can mean big changes in what needs to be done.

That said, there are some ideas that seem robust for me. I believe that it’s unethical to dumb down people or the work that they do. I believe that we should focus our craft on learning, and learning how to learn rapidly. How can we improve the state of testing? By recognizing that software development is a web of people who are related in service to each other. That means putting people and social issues first. Get that right, and everything else will follow.

Suggestions are cool.  Standards are something else.  No group should be dictating to other people how they must test unless there are compelling human health and safety reasons for it. Do you really believe that the standards people know anything useful about your business? That the force of government-supported regulation, created by busybodies, should weigh on how you do your daily work? And if your answer is No, what are you going to do to get it stopped?

I talked to one fellow at a conference who said that managers would have no way to filter candidates without certifications.  “No way?” I asked  “Really?”  “Well, what other way is there?” he asked. “What other ways can you think of?” I asked him. He was stumped.  He didn’t seem to know about personal references, well-crafted resumes, networking, interviewing, auditioning, hiring from within and training people.  He did know about those things, but he was so fixated on certification that he couldn’t come up with a single one of them on the spot. Now:  do you really want someone like that, with that kind of limited mindset, setting the standards for how you test your product? Big money and people’s livelihoods are riding on your answer.

uTest: We sat in on your STPCon presentation in October, where you said “Pay attention to your emotions while testing. If something strikes you as being off, then chances are it probably is.” How did you come up with this piece of advice? And can you give testers a brief real-world example?

MB: I fly a lot, and so I often have to book travel using buggy airline Web sites. A couple of years ago, I had to book a flight to Europe. In the course of five minutes, I went through surprise (the site kept revising the dates I had chosen), confusion (why was it doing that?), frustration (the airline’s own Web site didn’t offer flights directly from Toronto to Amsterdam, even I knew though travel agency sites and personal experience that there were two such flights a day), impatience (why is this so hard and taking so long, for crying out loud?) and annoyance (I don’t liked being stymied and having my time wasted). I wrote an article about the experience; you can find it here.  

We all want to do testing better and faster, right? Tests of the business rules and the calculations are really important—no question about that—but emotional responses are important too, and they have the advantage of being immediate, both in the “right now” sense and the “no intermediary” sense; they’re direct experience.  Our customers will have emotional responses to their experiences too, and you never get a second chance to make a first impression.

In the Rapid Testing course that James Bach and I write and teach, we say that a bug is “something that bugs somebody who matters”.  People, including testers, are bugged by stuff that doesn’t work.  Being bugged is a feeling that starts with an emotional reaction.  The feeling is a signal.  It’s data, but not information; you still need to interpret the signal’s meaning and significance.  So instead of saying “If something strikes you as being off, then chances are it probably is,” I’d prefer not to be too certain about that.  Look into what you’re feeling, for sure, but consider the possibility that something else is the source of the feeling.  Maybe what you’re seeing is okay, but you need to learn something about the business rules.  Maybe it’s not consistent with an older version of the product, but the new way of doing things is really better and you’re just not used to it. Maybe it feels like a big deal for you, but it’s okay for your client or your end users.

We teach people to pay attention not only to the product, but also to their testing. If you’re bored while testing, use that feeling as a trigger heuristic to prompt a question:  why am I bored? Is it because my testing isn’t revealing anything new? Is it because I’m doing repetitive and un-engaging work that I could delegate to a machine? Is it because this is useless work that I shouldn’t bother with at all? Am I missing the point or paying attention to the wrong things?  If you’re uncomfortable for some reason—if you feel like you’re being pressured, for example—there’s a possibility that you’ll be distracted, or that you’ll rush, or that you’ll be biased towards noticing some kinds of problem at the expense of others. But maybe the pressure is telling you something about the significance of risk, or maybe it’ll remind you to work rapidly, or something else that’s good in your context.

In Jerry Weinberg’s Quality Software Management Vol. 1, Systems Thinking, he points out that decisions about quality are always political and emotional, but people want to appear to be rational. In fact, that in itself causes an emotional reaction in people who want to preserve their own illusions that they’re rational. At a conference recently, a tester approached me and said that she had been intrigued by the role of emotion in software development. She mentioned this to her boss. He replied, “There’s no room for emotion in software development.”  She said, “Really?  No room at all?” He got very agitated, and repeated, shouting this time, “THERE’S NO ROOM FOR EMOTION IN SOFTWARE DEVELOPMENT!” She told me he was quite serious. Naturally, we laughed about that. Our emotional reaction was a pointer to the incongruence between what he was claiming and how he was feeling.

uTest: What are the greatest threats right now to the software testing discipline?  What are the greatest hopes for a brighter testing future?

MB: The greatest threat at the moment, I think, is a systemic set of misconceptions about testing, especially at the middle management level. We don’t do quality assurance; people with the power to make decisions about the nature of the product and the project do that. We call those people managers. Software testing is not like manufacturing quality assurance, where we’re trying to make a zillion identical copies of something, and then checking to make sure that each one is just like the prototype. Every software development project is an attempt to build something new—otherwise we’d use a product that had been built already. So software testing has more in common with design quality assurance, where we’re trying to figure out the relationships between a product and its stakeholders, and trying to understand the product as we’re designing and building the first instance of it. But even then, we don’t assure quality; we question it on behalf of our clients. The people who are building and managing the product assure quality.

When we model software testing on manufacturing quality assurance, we emphasize checking at the expense of testing.  I’ve written a lot on my blog about the distinction between the two since August of 2009.  (You can see it at http://www.developsense.com/blog.shtml). Checking is important, but it’s mostly a confirmatory activity, verifying what we knew to be true before, or what we hope is still true, making sure that we’re getting the right answers. Testing is a bigger deal, focused on finding new information, identifying new risks, asking new questions.

Lots of people—managers, programmers, and even many testers—conflate testing and checking. Again, checking is really important, and it’s important to do it skillfully.  In particular, I think it’s important to automate it skillfully.  But when people confuse testing and checking, they don’t think clearly about cost, value and risk, and they tend to dumb testing down.  That’s a big problem.

uTest: Looking back over your testing career, you’ve seen a lot of changes to tools and techniques and trends.  What do you think software testing will look like in 2020?

MB: 2020 the year, or 20/20 the vision? I predict that, in 2020, there will have been a lot of changes to tools, techniques, and trends, and that someone will ask a tester to predict what testing will be like in 2030.

At this point in history, no sensible person should make specific predictions about technology beyond dinner time; read The Black Swan for more about that. But as for testing, the fundamentals will still be the same. People will want stuff, and programmers and engineers will design it. There will be miscommunication, misunderstanding, unexpected incompatibilities, delays, mistakes, emergent properties. So we’ll still need to investigate requirements and designs and products and problems skillfully.

It’s like writing. After a few thousand years of development on paper, we’ve got people writing for print and online media and television and radio. People’s work can be prepared and consumed anywhere in the world, 24/7, in any number of languages. But we still have people to manage the writers, and we still have people to help the writers—editors and research assistants and designers and so forth.

Editor’s note: Check back in tomorrow for Part III.

We’re thrilled to have Michael Bolton as the latest victim of our Testing the Limits series. As the founder of DevelopSense, Michael has traveled the world teaching the craft of software testing to businesses and individuals alike. Since 2005, he has specialized in courses on Rapid Software Testing – which he co-authored with James Bach. Michael is also a prolific writer, and his publications include hundreds of articles, essays and columns. Aside from his blog, you can keep tabs on his latest work through Twitter.

In Part I of the “trilogy” we discuss the Weekend Testers, testing abroad, how numbers can enslave managers, and of course, his pop-star namesake.

uTest: You’ve been a thought leader in the testing space for a while now, but people still seem to get you confused with Michael Bolton (the singer) on Twitter. Ever thought about creating a tester alias? Or have you considered asking him to change his name since “he’s the one that sucks.” Assuming you (and our readers) have seen Office Space, I bet this joke never gets old.

MB: Yeah, it never gets old.  Try renting a car with this name.

A couple of things on that. First, Office Space captures very well what it’s like to have my name. Second, it’s not his real name; he changed it already. Way back when, before Office Space, I was working in tech support at Quarterdeck Canada.  American callers would occasionally turn north when there were long phone queues in Santa Monica. On one call, when I introduced myself to the customer, he laughed. “Really? That’s your real name?” “Yes, really,” I said, expecting one of the usual jokes. He said, “You know, it isn’t his real name. I used to be his bass player.”  The singer’s real name is Bolotin, but according to the bass player, there was no hope that radio DJs would ever pronounce “Bolotin” right, so he changed it.

uTest: We recently interviewed your friend and colleague James Bach, who had high praise for a group called the Weekend Testers. Can you give our readers a quick recap of what this group does, and whether or not you’re on board with their testing philosophy?

MB: James hadn’t heard about Weekend Testers (or maybe he had heard, but it hadn’t registered) until I raved about them to him. When I was in India in November 2009, I attended a talk by Ajay Balamurugadas, who told an amazing story. They’re a bunch of fairly young testers from Bangalore. The core organizers (Ajay plus Sharath Byregowda, Manoj Nair, and Parimala Shankaraiah) are students of our colleague Pradeep Soundararajan. They gather online every week to spend an hour or so testing some freeware or some Web service, and then they spend an hour or so debriefing each other.

There’s a weekly leader who sets a mission for the session. Sometimes the focus is on finding bugs quickly; sometimes it’s on choosing a particular approach, sometimes it’s on note taking. In the talk, Ajay described testers taking responsibility for their own skills development; testers building a self-critical community; testers overcoming obstacles like power cuts and buggy messaging tools; testers ignoring meaningless certifications; testers providing service to the open source development community; testers discovering testing for themselves, owning it.

At the end of the presentation, I whooped, stood up and applauded like I was at a rock concert. At Q & A time, I raised my hand to say that I didn’t have a question, but that this was the most exciting talk on testing I had seen in years, maybe ever.  At the time of the presentation, there were already chapters forming in other cities in India.  At the beginning of this year, Markus Gartner and Anna Baik announced a European chapter.  It’s spreading!

Am I on board with their philosophy?  Hell yeah!  These people, and people like them, are the future of skilled testing.

uTest: It looks like you’ve been “testing abroad” for quite some time now. What’s the biggest thing you’ve learned about testing in locales other than the US and Canada (your two primary residences)?

MB: Scandinavia—Sweden in particular—and New Zealand seem to be percolating more excellent testing than other places.  Meanwhile, I observe that many Western firms—mostly the Americans—are making life difficult for testers in India and other developing countries. These firms didn’t know much about testing to begin with. They viewed it as a rote, clerical activity, piecework, checking work, commodity work that delivered little value. They knew how to do checking, sort of, very slowly and very expensively. But they didn’t know how to do testing, or how to increase its value, so they focused on cost and outsourced it.

The developing countries have millions of intelligent people who want to develop skills, but the West is still requiring these overprescribed, expensive, low-value, confirmatory approaches in which smart human testers are being asked to behave like slow, dumb machines. Confirmatory tests do find problems, but to a great degree, programmers should be pairing and low-level automated tests to squash those problems before testers ever see them. Then free up the testers to look for higher-level problems and previously unanticipated risks.

uTest: Your testing philosophy seems to draw a lot of heat from some circles. For instance, a lot of people seem to think you’re an anti-numbers guy (as you mentioned recently on our blog). What is it these people are so opposed to? Or are they simply misinterpreting your approach to testing?

MB: Excellent testing, the way we teach it, involves thinking critically about the product and developing a story about it. In order to do that expertly, we have to think critically about how we’re getting that story and how we’re telling it in ways that are important to our clients. Testing is about trying to make sure that our clients aren’t being fooled. If we’re fooling  ourselves, we’re likely miss important problems.

When I object to people using numbers badly or using numbers excessively, some people perceive that I object to using numbers at all – not so. It’s just that I love numbers so much that I hate to see the poor things abused.

Some people enslave numbers.  They make numbers work too hard, and too often. I’ve seen organizations collect piles of data about defect escape ratios and defect detection percentages. They hire market research firms and calculate the ratio of happy customers to unhappy customers. But the aggregated data doesn’t tell you anything specific on how to make things better for the unhappy customers.

When you enslave numbers, they eventually rise up, revolt, and enslave you. These organizations spend so much time collecting the data and talking about it and justifying it and trying to duck blame that they don’t seem to have time to do anything about the actual problems, which generally fall into two categories. One:  the organizations are trying to do more work than they can handle with the approaches they’re using. Two:  they’re not listening to people that matter—neither to their customers, nor to their own front-line staff, many of whom are closest to the customers. VPs could learn a ton of useful business information from their own customer service and technical support reps, and they could learn plenty about the project by listening to their programmers and their testers. Product and project knowledge gets mediated by middle managers and numbers; it turns from information into data. When your car is about to go off a cliff, it’s a weird time to be thinking about gas mileage and drag coefficients; better to take the right control action—look out the window and steer or use the brake until you’re back on course. Once you’re back to being productive, then you can start thinking about optimizing, if you think you need to. I wrote about that here and here .

Besides, people don’t decide things based on numbers anyway. They decide based on how they feel about the numbers.  On their own, numbers don’t tell you what is or isn’t okay. Your judgment does that. Your judgment is governed by the synthesis of observations and inferences and facts and feelings, in your head and in your gut.  Effective decisions require both head and gut. Each can mislead the other, which ends up with someone being fooled, or worse. Neuroscientists and psychologists and economists (people like Antonio Damasio, Dan Ariely, Daniel Kahneman and the late Amos Tversky, Daniel Gilbert, Gerd Gigerenzer, Daniel Goleman) have been looking into each others’ fields to discover fascinating stuff about the links between emotion and intelligence. (If you want to do research in that area, apparently it helps to be named Dan.)

Some people seem genuinely scared by the human issues and the uncertainty that’s inherent in software development.  They want testing to be this simple technical problem, a confirmatory thing. Questions related to exploration and discovery and investigation and learning emphasize the fact that we don’t know everything from the outset when we’re dealing with complicated systems. We can’t. We prepare what we think are the requirements, but we often don’t understand what we want until we can compare it to something we’ve got. We have to live in this messy, uncertain, continuously changing world that we can’t control very well.

In that kind of world, repeatability isn’t that big a deal; computers are good at that. The big deal is adaptability; can our software help people to solve their problems, not only reliably but also flexibly?  Designing software to do that, putting all the design work up front, is difficult—I’d say impossible. But we can do a bit of focused work, gather information (that is, test), and then tune things where they need tuning, add things where they need to be added, drop them where they’re not needed any more. Then we repeat the cycle often, with lots of little variations.  That’s what Agile development is about. That’s what evolution is all about, really. In that kind of world, testing is an important service, and the kind of testing that James and I teach is designed to make that service sophisticated and powerful and fast. I think the need for testing annoys people who think everything should be known in advance. Alas, I believe that there are lots of people, even many testers, who think like that.

Editor’s note: Check out Part II of the interview.

Writing witty, thought-provoking content is really hard.  And we’re pretty lazy, but fortunately we know some extremely smart & funny people.  So we invented the Testing The Limits series, in which we interview leaders from the worlds of testing, software, entrepreneurship and crowdsourcing.

We’re immensely grateful to these talented, busy people, and we have much more planned for the Testing The Limits series in 2010.  But before we flip the calendar, these posts from this year are worth another look:

June: James Whittaker – Author, Professor and Testing Evangelist at Google

July: Rosie Sherry — Founder of the UK-based Software Testing Club

August: Andrew Muns — President of Software Test & Performance

September: Jack Margo — SVP of Internet Operations of Developer Shed

October: Jon Winsor — Author, Crowdsourcing Expert, and Founder of Victors & Spoils

November: Matt Heusser — Software Testing Author, Professor and Testing Manager

December: James Bach — Software Testing Author, Teacher and Speaker

We have some great guests and ideas lined up for 2010, including software execs, QA thought leaders, and famous journalists & authors.  As always, the goal of Testing The Limits will be to inform, to entertain, and above all else, to help our readers get to know these thought leaders who are worth following and listening to.

Have a suggestion for a future Testing The Limits guest?  Drop us a note or tell us in the comments section.

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.