Testing Roundtable: What’s the Biggest Weakness in the Way Companies Test?

This month, in place of our standard Testing the Limits interview, we decided to hit up a few of our past guests for a “testing roundtable” discussion. The topic: What is the biggest weakness in the way companies test software? Below are some extremely insightful answers from testing experts Michael Bolton, James Bach, Noah Sussman, Dan Bartow, Rex Black, Jim Sivak and Cem Kaner. Enjoy!


Michael Bolton, Principal at DevelopSense:

So far as I can tell, most companies treat software development as implementation of highly idealized business processes, and they treat testing as an exercise in showing that the software models those processes in a way that’s technically correct. At the same time, companies treat the people who use the software as an abstraction. The consequence is that we’re creating software that delays and frustrates the people who use it or are affected by it. When testing is focused almost entirely on checking the functions in the software, we miss enormous opportunities to learn about the real problems that people encounter as they go about their business. Why are testers so often isolated from actual end-users?

Today I was traveling through the airport. When I checked in using the online service, I had accidentally noted that I’d be checking two bags, but I only brought one with me. In addition, my flight was cancelled, and I had to be put on a later flight. The customer service representative could get me onto that flight, but she had serious trouble in printing a boarding pass associated with only one bag; apparently there was a warning message that couldn’t be dismissed, such that her choices were to accept either three bags or none at all. It took fifteen minutes and two other representatives to figure out how to work around the problem. What’s worse is that the woman who was trying to help me apologized for not being able to figure it out, as if it were her responsibility. Software development organizations have managed to convince our customers that they’re responsible for bugs and unforgiving and unhelpful designs.

The success of a software product is only partly based on how it handles the happy path. That’s relatively easy to develop, and it’s relatively easy to check. Real testing, to me, should be based on investigating how the software allows people to deal with what we call “exceptions” or “corner cases”. That’s what we call them, but if we bothered to look, we’d find out that they were a lot more common than we realize; routine, even. Part of my vision of testing is to include a new discipline in which we do significant field research and participant observation. Instead of occasionally inviting customers to the lab (never mind sitting in the lab all by ourselves), we testers—and our organizations—could learn a lot through direct interaction with people who use the software every day; by close collaboration with technical support; and by testing rich and complex scenarios that are a lot closer to real life than simplified, idealized use cases.


James Bach, Author and Consultant, Satisfice:

There is a cluster of issues that each might qualify as the biggest weakness. I’ll pick one of those issues: chronic lack of skill, coupled with the chronic lack of any system for acquiring skill.

Pretty good testing is easy to do (that’s partly why some people like to say “testing is dead”– they think testing isn’t needed as a special focus because they note that anyone can find at least some bugs some of the time).

Excellent testing is quite *hard* to do.

Yet as I travel all over the world, teaching testing and consulting in testing organizations, I see the same pattern almost *everywhere*: testing groups who have but a vague, wispy idea what they are trying to do; experienced testers who barely read about and don’t systematically practice their craft beyond the minimum needed to keep their employers from firing them; testers whose practice is dominated by irrational and ignorant demands of their management, because those testers have done nothing to develop their own credibility; programmers who think their automated checks will save them from disaster in the field.

How does one learn to test? You can’t get an undergraduate degree in testing. I know of two people who have a PhD in testing, one of whom I admire (Meeta Prakash), the other one is, in my view, an active danger to himself and the craft. I personally know, by name, about 150 testers who are systematically and diligently improving their skills. There are probably another several hundred I’ve met over the years and lost touch with. About three thousand people regularly read my blog, so maybe there are a lot of lurkers. A relative handful of the people I know are part of a program of study/mentoring that is sanctioned by their employers. I know of two large companies that are attempting to systematically implement the Rapid Testing methodology, which is organized around skill development, rather than memorizing vocabulary words and templates. Most testers are doing it independently, however, or even in defiance of their employers.

Yes, there is TMap, TPI, ISTQB, ISEB, and many proprietary testing methodologies out there. I see them as crystallized blobs of uncritical folklore; confused thinking about testing frozen in place like fossilized tree sap. These models and procedures have been created by consultants and consulting companies to justify themselves. They neither promote skill or require skill. They promote what I call “ceremonial software testing” rather than systematic critical thinking about complex technology.

Just about the best thing a tester can do to begin to develop testing skill in a big way is not to read or study any test methodology. Ignore vocabulary words. Toss aside templates. No, what that tester should do is read Introduction to General Systems Thinking, by Gerald M. Weinberg. Read it all the way through. Read it, young tester, and feel your mind get blown. Read it, and meditate on its messages, and do the exercises it recommends, and you will find yourself on a new path to testing excellence.


Noah Sussman, Technical Lead, Etsy:

A surprising number of organizations seem to dramatically underestimate the costs of software testing.

Testability is a feature and tests are a second feature. Having tests depends on the testability of an application. Thus, “testing” entails the implementation and maintenance of two separate but dependent application features. It makes sense then that testing should be difficult and expensive. Yet many enterprise testing efforts do not seem to take into account the fact that testing an application incurs the cost of adding two new, non-trivial features to that application.

There also seems to be a widespread misconception that testing somehow makes application development easier. In fact the opposite is true.

If I may mangle Kernighan: testing is much more difficult than writing the code in the first place. To implement testability and then write tests, one needs first to understand the architecture of the application under test. But testing also requires doing hard things — like input partitioning and path reduction — that are beyond the scope of the application. The reality is that to get good tests, you’re going to have to ask some of your best people to work on the problem (instead of having them work on user-facing application features). Yet many organizations seem not yet to have recognized this.


Continue Reading

Essential Guide to Mobile App Testing

Who Will Be This Year’s Software Test Luminary?

Luminary: a person who has attained eminence in his or her field or is an inspiration to others

You’d be hard-pressed to find a profession with a wider range of ideas and personalities than that of software testing. This point is certainly not lost on our readers, as evidenced by the popularity of our Testing the Limits interview series. And it’s not lost on our good friends at Software Test Professionals, who have opened up nominations for the 2nd Annual Software Test Luminary Award.

More on the nomination process in a second, but first, a little bit about the award itself:

The Luminary award will honor any software testing and quality assurance professional who is determined, persistent, and committed to improving a process or methodology. They develop ideas, which when properly applied, have a positive impact on the end product, either by enhancing quality or performance or resulting in improved efficiencies for a particular process, team or organization. In addition, their contributions elevate the critical role of the software test profession within the software development process.

A luminary is someone who has inspired others by their actions and the results of those actions on the profession. They inspire others to pursue a software testing career. It is about how they have given back, and shared their knowledge and experience with others in order to advance the profession and improve the career paths of all practitioners. A luminary will typically be recognized and respected long after their days of practicing have ended.

If you recall, last year’s honor went to Gerald M, Weinberg, who edged out fellow nominees James Bach and Cem Kaner.

So who will be named this year’s Software Test Luminary? It’s your call. STP will gather nominations and submit the top 3 candidates for a final round of voting. The finalist will be announced at the Software Test Professionals Fall 2011 Conference, October 24-27 in Dallas, Texas.

Here’s a quick timeline of the events:

Continue Reading

Essential Guide to Mobile App Testing

Software Testing Classics: Bug Advocacy by Cem Kaner

Last week, I decided to go back in time to revisit a classic work of software testing theory by James Bach, on the subject of risk-based software testing. What I tried to show was that despite tremendous advances in terms of tools, techniques and technology, the fundamentals of software testing essentially remain the same. I hope that was conveyed in the post.

Anyway, in that same spirit, I’m going to quickly bring back another classic work of software testing theory for debate and discussion: Bug Advocacy by Dr. Cem Kaner.

If that name sounds familiar, it’s because Cem Kaner is one of the field’s leading thinkers, teachers and practitioners. He’s also appeared as a guest on our Testing the Limits interview series, which you can find here, here and here.

So what is “Bug Advocacy” all about? The title pretty much says it all (I hope), but let’s take a look at some key excerpts from this 100-page masterpiece to find out more. First the, the premise:

  1. The point of testing is to find bugs.
  2. Bug reports are your primary work product. This is what people outside of the testing group will most notice and most remember of your work.
  3. The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.
  4. Programmers operate under time constraints and competing priorities. For example, outside of the 8-hour workday, some programmers prefer sleeping and watching Star Wars to fixing bugs.

A bug report is a tool that you use to sell the programmer on the idea of spending her time and energy to fix a bug.

Motivating the Bug Fixer
Some things that will often make programmers want to fix the bug:

Essential Guide to Mobile App Testing

The “Jedi Knights” of Context-Driven Software Testing

The first rule of Fight Club is: you do not talked about Fight Club. Lucky for us, that rule does not apply to the Context-Driven School of Software Testing.

In case you hadn’t noticed, the context-driven school has amassed a global following in just a few short years years, despite some initial confusion on the part of newbies…What is a context-driven tester? What is the basic premise? How is it different from the other prominent “schools” of testing? And what does one have to do to become a member?

James Bach – the founding father of CDT – posted a great overview of the principles this past weekend in an article titled “The Dual Nature Of Context-Driven Testing.” He offers some key distinctions on what the term means, what it doesn’t mean, and how you can grow as a tester by learning more about its principles. Here are a few important excerpts (emphasis mine), beginning with an abridged definition:

The Context-Driven School of software testing is a way of thinking about testing, AND a small but world-wide community of like-minded testers. There are other, larger, schools of testing thought. But CDT represents my paradigm of testing. By paradigm, I mean an organizing worldview, an ontology, a set of fundamental beliefs.

CDT is not a style of testing. It’s not a toolbox of methods. It’s more fundamental than that. You could think of  CDT partly as an ethical position about testing. All methods or styles are available to Context-Driven people, but our selection of methods and reactions to testing situations are conditioned by our ethical position. This position is defined here.

Reading further, it occurred to me that the context-driven school is well-represented on the uTest Blog. To illustrate this alliance, I’ve included links to the names of those “Jedi Knights” who have made contributions on this site. Here’s the excerpt:

Continue Reading

Essential Guide to Mobile App Testing

Picture Quiz: Should You Become A Software Tester?

We write frequently on the subject of what it takes to become a top tester – in both the uTest community and the industry as a whole. We ask the testing giants their thoughts on the matter (see quotes below) and publish guest posts and Crash Courses in an effort to help you become a better software tester.  Please hold your applause.

But what if software testing isn’t for you? What if after all the education, training and job-searching, you discovered that you really had no knack for the craft? Wouldn’t it have been nice to know that a little sooner? Lucky for you, I’ve designed this picture quiz as a humorous supplement to the Jung Career Indicator Test. Here’s how it works: If you don’t see anything wrong with these photos, then software testing is definitely NOT for you. Far from scientific, but hey, it’s a start.

Almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.” – Cem Kaner

Continue Reading

Essential Guide to Mobile App Testing