Software Testing Blog

In part II of our Testing the Limits interview with Ben Simo, we’ll discuss whether you should trust automated testing tools; the proliferation of testers on Twitter; the true meaning of “QA”; how testing evolves differently in each company; the long lost Bach brothers and much more. You can catch up on the conversation by reading part I. We’ll wrap things up tomorrow with part III.

********

uTest: Jon Bach mentioned that changing the meaning of “QA” to Quality Assistance would help outsiders (engineers, executives, et al) better understand the role of this discipline.  Agree or disagree?

Simo: I believe I first heard  “Quality Assistance”  from Cem Kaner.  I agree with Jon. When testers bear the title Quality Assurance, it often implies that they actually assure the quality of other people’s work. Testers are in a position to help assist quality; not assure it. Let’s not assist the setting of unrealistic expectations with inappropriate titles.

uTest: While we’re on the subject, are you anyway related to James and Jon Bach? The resemblance is uncanny.
Simo: I don’t think so. I’m available for adoption if the Bach family is interested.

uTest: You’ve said that you frequently use automated tools, but that you don’t trust them entirely (back to that whole defensive pessimist thing again). What advice do you have for testers and managers wanting to strike a healthy balance? And what’s currently in your arsenal of automated tools?

Simo: My mistrust in tools is based on the fact that tools can’t think for me. Automated checking can only process whatever decision rules someone thought to program when the checks were created. Automation will consistently do what it is programmed to do and consistently not do what it is not explicitly programmed to do. I find test automation to be useful. In fact, there are some things I’d not want to even try to do manually. I do, however, distrust the green bar. When automated checking passes, I ask myself what the automation does not tell me. I also try to keep aware that people who don’t understand what the automation does are likely to assume that it does more than it does.

Tools are much more than test automation. Tools are essential for testing. I don’t want to test without tools. I have some old programming books that promote testing in which a programmer manually executes code, step-by-step, with pencil and paper in order verify that the code works as expected. This is manual testing. This is a testing practice that came from a time when computer time was rare and cost more than people. We’d now laugh at someone proposing testing in this manner.

Today, whether we call it “manual” or “automated”, tools and automation are part of coding and testing software.
Programmers use fancy integrated development environments that do a great deal of the tedious work of managing and checking code. Programmers have unit testing tools that allow them to write tests as they write the code. We use tools to track and version source code. We have continuous integration tools that automatically run pre-defined tests and report results when programmers save code into shared code repositories. Today, programmers rely on interactive tools to do work that used to be done manually. These tools support people coding software rather than trying to replace them with batch processes.

Many of the tools built for testers seem to still be stuck in the world of batch processing. Rather than assist testers by helping with tedious tasks, they try to replace test execution. Some of this comes from the design-all-tests-up-front-in-procedurally-scripted-detail thinking that made sense when computer time cost more than people.

I worked as a tester for nine years before I encountered a record and playback type test automation tool. In those nine years, tools played an essential role in my testing. We built all our tools in-house. We wrote macros and scripts to do tedious data processing. We wrote code to verify data formats. We created test interfaces for databases. We created test management systems. We depended on tools but did not have a single test case that was fully automated. This automation made testers both more efficient and effective.

Rather than look at testing as something to be either manual or automated, I encourage people to look at individual tasks that are part of testing and try to identify ways that automation can help testers evaluate software. Rather than limit tools to automation of test execution, I look for ways that automation can assist testers – to turn them into powerful cyborgs.

These tools include automated test execution tools, data setup and teardown scripts, communication tools, test management tools, test logging and reporting tools, test data manipulation tools, and more. Along with test execution automation tools (take your pick), my most-used testing tools are Excel, Picasa (for screenshots), a screen recorder, and the GNU Unix utilities (also available for Windows).

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

Simo: I’ve not noticed great distinctions in testing practices based on industry.

What I have noticed is that testing practices seem to evolve within individual companies and test groups. Rather than learn from what others have done, it often appears that each tester, each test organization, and each company starts in some stone age and evolves on its own – making the same mistakes that others did long before.  Lessons learned don’t seem to be shared much across companies.

I’ve also noticed that “quality assurance” groups in large commercial businesses, regardless of industry, tend to be dominated by factory school thinking. Instead of viewing testing as cognitively complex work that requires critical thinking, good communication, and continual learning: testing is thought to be predictable repeatable validation work that can be done by automation or cheap labor.  These companies tend to view quality and process as things to be scripted, controlled, and quantitatively measured. In doing so, I believe these companies dehumanize their people and their customers.  These companies tend to have what Jerry Weinberg called “overstructured management” in a paper written nearly 30 years ago; management which attempts to manage people like code: with sequential work models, binary either-or choices, and mechanical repetition. These things may be good for computers, but aren’t great for people.

While many companies are still recursively calling on the mistakes decades past, others are recognizing that people aren’t to be controlled like machines.  While they may not be familiar with either document, these companies exhibit the values expressed in the Agile Manifesto and practice the principles of the Context-Driven School. These companies value people over systems.

uTest: Who should testers be following on Twitter to stay current on the latest industry trends (I mean, besides you and us, of course)?

Simo: I’m not as interested in trends as much as I am in experience reports, and new ideas and challenges. Some of my favorites who are active on Twitter are Pradeep Soundararajan (@TesterTested), Shrini Kulkarni (@shrinik), Matt Heusser (@mheusser), Lanette Creamer (@lanettecream), Elisabeth Hendrickson (@TestObsessed), Michael Bolton (@MichaelBolton), Jon Bach (@jbtestpilot), and James Bach (@JamesMarcusBach). Beyond that, I suggest people see who these people, and I, follow and interact with on Twitter. Use Twitter’s search feature to find people tweeting about things that interest you. Don’t limit yourself to only testers or those people you like. We can learn a great deal from those outside our field or with whom we disagree.

Editor’s note: Read part III of the Ben Sim Testing the Limits trilogy.