Testing the Limits With James Bach – Part I

JamesBach150James Bach is synonymous with testing, and has been disrupting the industry and influencing and mentoring testers since he got his start in testing over 25 years ago at Apple. Always a great interview, James is one of our most popular guests and we’re happy to have him back for his first Testing the Limits since 2011. For more on James’ background, his body of work and his testing philosophy, you can check out his blog, website or follow him on Twitter.

In Part One of our latest talk with James, he talks about a future that involves a ‘leaner’ testing world, the state of context-driven testing outside of the United States, and why you’re “dopey” if you’re a manager using certain criteria in hiring your testers.

uTest: We know you don’t enjoy certifications when it comes to testers. In fact, in a recent blog, you mentioned that ‘The ISTQB and similar programs require your stupidity and your fear in order to survive.’ Do you feel like certifications are picking up steam when it comes to hiring and if they’re becoming even more of a pervasive issue?

JB: I don’t have any statistics to cite, but my impression from my travels is that certifications have no more steam today than they did 10 years ago. Dopey, frightened, lazy people will continue to use them in hiring, just as they have for years.

uTest: Speaking of pervasive problems, what in your opinion has changed the most – for better or for worse – in the testing industry as a whole since we talked with you last almost 3 years ago?

JB: For the better: the rise of the Let’s Test conference. That makes two solidly Context-Driven conference franchises in the world. This is related to the general rise of a spirited European Context-Driven testing community.

Nothing much else big seems to have changed in the industry, from my perspective. I and my colleagues continue to evolve our work, of course.

uTest: In a recent interview, you mentioned that you see the future of testing, in 2020 for instance, as being made up just of a small group of testing “masters” that jump into testing projects and oversee the testing getting done…by people that aren’t necessarily “testers.” Do you see QA departments going completely by the wayside in this new reality of a leaner testing world? Wouldn’t this be a threat to the industry in general?

JB: I’m not sure whether you mean QA groups, per se, or testing groups (which are often called QA). I don’t see testing groups completely going away across all the sectors of the industry, but for some sectors, maybe. For instance, it wouldn’t surprise me if Google got rid of all its “testers” and absorbed that activity into its development groups, who would then pursue it with the ruthless efficiency of bored teenagers mopping floors at McDonald’s (a company as powerful as Google can do a lot of silly things for a very long time without really suffering. Look at how stupidly HP has been managed for the last 20 years, and they are still, amazingly, in business).

Continue Reading

Are There Enough ‘Intellectual’ Software Testers?

imagesJames Bach is no stranger to tackling heated topics, and in general, being one of the most influential disruptors in the in the testing industry.

So it comes as no surprise that in a recent blog, James provided some fodder for a great discussion in the uTest Forums, arguing that there aren’t enough intellectual testers in the field — that is, testers that are willing to challenge themselves or the status quo:

“The state of the practice in testing is for testers NOT to read about their craft, NOT to study social science or know anything about the proper use of statistics or the meaning of the word ‘heuristic,’ and NOT to challenge the now 40 year stale ideas about making testing into factory work that lead directly to mass outsourcing of testing to lowest bidder instead of the most able tester.”

While there was a fair amount of pushback to this, a surprising amount of uTesters tended to agree, including one tester that even went so far as to call it a “pet peeve” of his. However, while agreeing with Bach’s assessment, these same testers argued that it isn’t necessarily their fault — it’s a product of their environment:

“To conclude, I believe that the issue lies with how projects are managed. If no time is left for more robust testing, then it almost doesn’t matter how intellectual or technically savvy a tester is if all he/she is going to have time to do is create and execute tests against specifications. In other words, intellectual testers don’t have much opportunity for more intellectual testing. A strong tester would not be able to showcase those skills in this environment.

Continue Reading

Q&A: Context-Driven Testing Champions Talk Trends, Preview Let’s Test Oz

Henrik Andersson and David Greenlees are two well-known contributors to the context-driven testing community and together co-founded the Let’s Test conferences, which celebrate the context-driven school of thought. Let’s Test Oz is slated for September 15-17 just outside Sydney, Australia, and uTest has secured an exclusive 10% discount off new registrations. Be sure to email testers@utest.com for this special discount code if you plan on attending.

In this interview, we talk with Henrik and David on trends in the context-driven community, and get a sense of what testers can expect at Let’s Test Oz.

19c4175HenrikAndersson

uTest: Like James Bach, you’re both members of the ‘context-driven’ testing community. What drove each of you to context-driven testing?

HA: Actually, James did. I had close to no awareness of the context-driven testing (CDT) community before I hosted James’ RST class in Sweden in spring of 2007. During my discussions with James, I found that we shared lots of fundamental views on testing, and he insisted that I should meet more people in the CDT community.

James told me about the CAST conference that took place in the States, and that just before this, there would be a small peer conference called WHET 4 that his brother Jon hosted. A few days later, I got an invitation from Jon Bach to attend. At this workshop, where we spent a weekend discussion on Boundary Testing, I met testers like Cem Kaner, Ross Collard, Scott Barber, Rob Sabourin, Michael Bolton, Dough Hoffman, Keith Stobie, Tim Coulter, Dawn Haynes, Paul Holland, Karen Johnson, Sam Kalman, David Gilbert, Mike Kelly, and, of course, Jon and James Bach. From then on I was hooked!

DG: Difficult question to answer without writing a novel! I wrote about my testing journey some time back, however, that doesn’t really touch on my drivers toward the CDT community. If I was to pinpoint one thing, it would be the book Lessons Learned in Software Testing (Bach, Kaner, Pettichord). This was my first introduction to the community and to what I believe is a better way to test…in fact…the only way to test.

What keeps me here is the fantastic people I come across each and every day. We challenge each other, we’re passionate, and we’re not afraid to put our opinions out there for the world to hear and critique. This all adds to the betterment of our craft, which is our ultimate goal. I’m a firm believer that there is no ‘one-size-fits-all’ approach to testing, and when you add that to my natural tendency to explore rather than confirm, I find that the CDT community is a great fit for me.

uTest: And speaking of James Bach, he’s one of the keynote speakers at Let’s Test Oz in the Fall. Can you tell us a little bit about the idea behind the show, and why you felt it was time for context-driven conferences in Europe and Australia?

HA: Let’s Test is all about building, growing and strengthening the CDT community. We have successfully arranged Let’s Test three years in a row in Europe, but the attendees are coming from all over the world. The idea behind Let’s Test is to create a meeting place for testers to learn, share experiences, grow, meet other testers, do some real testing, and, of course, to have a whole lot of fun.

When David Greenlees and Ann-Marie Charrett told me about what they were looking to achieve, I immediately felt that it was in line with Let’s Test, and believe Let’s Test can be a great vehicle to grow the CDT community in Australia.

Last year, we did a one-day tasting of Let’s Test in Sydney, and this year, we did one in the Netherlands. In November, we will be hosting one in Johannesburg, South Africa. The purpose of the small tastings of Let’s Test is for testers to get a glance at the Let’s Test experience, at a really low cost. If you cant come to the real Let’s Test, this is a great alternative to check out what it is all about.

DG: From the Australian point of view, it’s fair to say that the CDT community is very small. We refer to the area as ‘Downunder’ — this is our way of saying Australia and New Zealand. I felt it was time to change that, and one way to help the CDT community thrive is to hold a CDT conference.

For quite a few years now, I’ve felt that Downunder needed a different style of software testing conference, one where conferring is the ultimate goal, and so I emailed Henrik, and he was extremely positive and encouraging…so here we are.

Continue Reading

10 Quotes for Software Testers…Since the Last Time

So we had to dig back deep into the archives to see when the last time was that we featured some testing quotes worthy of hanging up on the ol’ refrigerator. To our horror, it was over three years ago, so we decided it was time again for another roundup. Without further ado:

Testing means learning. Learning requires faith in one’s ignorance combined with the confidence that it can be extinguished.”James Bach

“Testing is organized skepticism.”— James Bach (A double dose of Bach!)

“Be a yardstick of quality. Some people aren’t used to an environment where excellence is expected.” – Steve Jobs

“There are only two things that seem to be even close to universally true when it comes to testing – things are constantly changing, and if you put three testers in a room with a testing term or topic to discuss, no more than two of them will ever agree at the same time.”Scott Barber

“A ‘passing’ test doesn’t mean ‘no problem.’ It means no problem *observed*. This time. With these inputs. So far. On my machine.”Michael Bolton

Continue Reading

In Videos: The Best in Software Testing From James Bach

He may need no introduction to software testing vets, but for those starting off, there’s no better guy to turn to for testing advice than James Bach.

James has been disrupting the testing industry since 1987 with his blogs, lectures, courses and books on software testing and the context-driven school of thought. For more on James’ background, his body of work and his testing philosophy in general, you can check out his blog or follow him on Twitter.

In the meantime though, we’ve rounded up some of the best (and animated in discussion!) videos and lectures from the one-and-only Bach.

Open Lecture by James Bach

Continue Reading

Software Engineers: “Forgive Me Testers, For I Have Sinned”

A few days back, GigaOM posted terrific article on the 7 Sins of Software Development. When you read it, which I strongly suggest, I think you’ll see that testers play a huge role in absolving the various “deadly” sins of software engineers.

If you’re too apathetic to read the article (sloth is a sin, FYI) then check out the excerpts below:

Sloth
Sloth is apathy, not laziness. An apathetic programmer is the arguably the most detrimental, because he has zero interest in quality. On the other hand, a lazy programmer can be a good programmer, because laziness can drive long-term efficiencies. For example, if I’m too lazy to type in my password everywhere, I might create a single sign-on feature. Or, if I’m too lazy to manually deploy software, I will instead write an automatic deployment tool. Laziness and scalability go hand in hand.

Wrath
Although many software engineers seem peaceful, underneath the surface often lurks a passive aggressive personality. Take a look at source code comments to see examples of this hidden hostility. Usually profanity in source code is proportional to technical debt. However, it is vital that your engineers are not milquetoasts. Beware of the programmer who does not ask questions or who will use any text editor willingly. Good programmers have strong opinions, but they also appreciate lively debates.

Envy
Envy can be very dangerous in software development. Envy for other products often leads to feature creep. If someone mentions feature parity, you should ask, “But do we need it?” The ultimate killer feature is simplicity, but simple to use is hard to design. Also, it is easy to lose focus when you are constantly watching what other companies are doing. Imagine building towers out of Legos. Would you rather build one tower at a time or many towers in parallel? The parallel approach only works if the towers are identical. Otherwise, you spend too much time context switching. Agility is not the same as half-baked. And doing one thing well is still underappreciated.

Continue Reading

8 Tips For Becoming a Dedicated Tester

Become a top software testerOur old friend James Bach recently fielded a question on his blog from a new tester seeking advice on what her daily routine should include so that she can grow in her new field. James seems impressed by the new tester’s discipline (she did willingly ask for daily testing “homework” after all) and dedication to the craft. He outlined five tasks he believes every tester should practice on a daily basis, here’s a quick summary of his tips:

Write every day
Whenever I find myself with a few moments, I make notes of my thoughts about testing and technical life.

Watch yourself think every day
While you are working, notice how you think. Notice where your ideas come from. Try to trace your thoughts.

Question something about how you work every day
Testers question things, of course. That’s what testing is. But too few testers questions how they work. Too few testers question why testing is the way it is.

Explain testing every day
Even if no one makes you explain your methodology, you can explain it to yourself.

I like these tips because they aren’t the typical recommendations you run across, like “test whenever you can,” “read an array of testing books” and “be open-minded when it comes to techniques.” Those are great tips too, just nothing special. Of course, James didn’t just give one sentence explanations for each of his pointers, so take a few minutes and read his complete blog post to get the full impact of these smart tips.

And as a little extra, here are a couple more tips James’ readers left in the comments section.

Continue Reading

5 Myths of Software Testing

As I scan the software testing stories of the day, I’m amazed at the frequency of certain misconceptions. While there are too many to list, I wanted to share five of the most common testing myths (in my brief experience). The first three I find to be prevalent in mainstream news articles, while the other two are more common within the tech industry in general.

Take a look and see if you agree with me.

Myth 1. Testing is boring: It’s been said that “Testing is like sex. If it’s not fun, then you’re doing it wrong.” The myth of testing as a monotonous, boring activity is seen frequently in mainstream media articles, which regard testers as the assembly line workers of the software business. In reality, testing presents new and exciting challenges every day. Here’s a nice quote from Michael Bolton that pretty much sums it up:

“Testing is something that we do with the motivation of finding new information.  Testing is a process of exploration, discovery, investigation, and learning.  When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated, we’re testing.  We’re testing when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before.”

Myth 2. Testing is easy: It’s often assumed testing cannot be that difficult, since everyday users find bugs all the time. In truth, testing is a very complex craft that’s not suited for your average Joe. Here’s Google’s Patrick Copeland on the qualities of a great tester:

Continue Reading

Testing Roundtable: What’s the Biggest Weakness in the Way Companies Test?

This month, in place of our standard Testing the Limits interview, we decided to hit up a few of our past guests for a “testing roundtable” discussion. The topic: What is the biggest weakness in the way companies test software? Below are some extremely insightful answers from testing experts Michael Bolton, James Bach, Noah Sussman, Dan Bartow, Rex Black, Jim Sivak and Cem Kaner. Enjoy!

*********************

Michael Bolton, Principal at DevelopSense:

So far as I can tell, most companies treat software development as implementation of highly idealized business processes, and they treat testing as an exercise in showing that the software models those processes in a way that’s technically correct. At the same time, companies treat the people who use the software as an abstraction. The consequence is that we’re creating software that delays and frustrates the people who use it or are affected by it. When testing is focused almost entirely on checking the functions in the software, we miss enormous opportunities to learn about the real problems that people encounter as they go about their business. Why are testers so often isolated from actual end-users?

Today I was traveling through the airport. When I checked in using the online service, I had accidentally noted that I’d be checking two bags, but I only brought one with me. In addition, my flight was cancelled, and I had to be put on a later flight. The customer service representative could get me onto that flight, but she had serious trouble in printing a boarding pass associated with only one bag; apparently there was a warning message that couldn’t be dismissed, such that her choices were to accept either three bags or none at all. It took fifteen minutes and two other representatives to figure out how to work around the problem. What’s worse is that the woman who was trying to help me apologized for not being able to figure it out, as if it were her responsibility. Software development organizations have managed to convince our customers that they’re responsible for bugs and unforgiving and unhelpful designs.

The success of a software product is only partly based on how it handles the happy path. That’s relatively easy to develop, and it’s relatively easy to check. Real testing, to me, should be based on investigating how the software allows people to deal with what we call “exceptions” or “corner cases”. That’s what we call them, but if we bothered to look, we’d find out that they were a lot more common than we realize; routine, even. Part of my vision of testing is to include a new discipline in which we do significant field research and participant observation. Instead of occasionally inviting customers to the lab (never mind sitting in the lab all by ourselves), we testers—and our organizations—could learn a lot through direct interaction with people who use the software every day; by close collaboration with technical support; and by testing rich and complex scenarios that are a lot closer to real life than simplified, idealized use cases.

*********************

James Bach, Author and Consultant, Satisfice:

There is a cluster of issues that each might qualify as the biggest weakness. I’ll pick one of those issues: chronic lack of skill, coupled with the chronic lack of any system for acquiring skill.

Pretty good testing is easy to do (that’s partly why some people like to say “testing is dead”– they think testing isn’t needed as a special focus because they note that anyone can find at least some bugs some of the time).

Excellent testing is quite *hard* to do.

Yet as I travel all over the world, teaching testing and consulting in testing organizations, I see the same pattern almost *everywhere*: testing groups who have but a vague, wispy idea what they are trying to do; experienced testers who barely read about and don’t systematically practice their craft beyond the minimum needed to keep their employers from firing them; testers whose practice is dominated by irrational and ignorant demands of their management, because those testers have done nothing to develop their own credibility; programmers who think their automated checks will save them from disaster in the field.

How does one learn to test? You can’t get an undergraduate degree in testing. I know of two people who have a PhD in testing, one of whom I admire (Meeta Prakash), the other one is, in my view, an active danger to himself and the craft. I personally know, by name, about 150 testers who are systematically and diligently improving their skills. There are probably another several hundred I’ve met over the years and lost touch with. About three thousand people regularly read my blog, so maybe there are a lot of lurkers. A relative handful of the people I know are part of a program of study/mentoring that is sanctioned by their employers. I know of two large companies that are attempting to systematically implement the Rapid Testing methodology, which is organized around skill development, rather than memorizing vocabulary words and templates. Most testers are doing it independently, however, or even in defiance of their employers.

Yes, there is TMap, TPI, ISTQB, ISEB, and many proprietary testing methodologies out there. I see them as crystallized blobs of uncritical folklore; confused thinking about testing frozen in place like fossilized tree sap. These models and procedures have been created by consultants and consulting companies to justify themselves. They neither promote skill or require skill. They promote what I call “ceremonial software testing” rather than systematic critical thinking about complex technology.

Just about the best thing a tester can do to begin to develop testing skill in a big way is not to read or study any test methodology. Ignore vocabulary words. Toss aside templates. No, what that tester should do is read Introduction to General Systems Thinking, by Gerald M. Weinberg. Read it all the way through. Read it, young tester, and feel your mind get blown. Read it, and meditate on its messages, and do the exercises it recommends, and you will find yourself on a new path to testing excellence.

*********************

Noah Sussman, Technical Lead, Etsy:

A surprising number of organizations seem to dramatically underestimate the costs of software testing.

Testability is a feature and tests are a second feature. Having tests depends on the testability of an application. Thus, “testing” entails the implementation and maintenance of two separate but dependent application features. It makes sense then that testing should be difficult and expensive. Yet many enterprise testing efforts do not seem to take into account the fact that testing an application incurs the cost of adding two new, non-trivial features to that application.

There also seems to be a widespread misconception that testing somehow makes application development easier. In fact the opposite is true.

If I may mangle Kernighan: testing is much more difficult than writing the code in the first place. To implement testability and then write tests, one needs first to understand the architecture of the application under test. But testing also requires doing hard things — like input partitioning and path reduction — that are beyond the scope of the application. The reality is that to get good tests, you’re going to have to ask some of your best people to work on the problem (instead of having them work on user-facing application features). Yet many organizations seem not yet to have recognized this.

*********************

Continue Reading

Testing the Limits With Anne-Marie Charrett – Part I

Testing the Limits with Anne-Marie CharrettTo kick off another amzing year of Testing the Limits we reached out to Anne-Marie Charrett, an independent tester who has worked for the likes of Mercury Interactive, IBM (twice) and Nortel – just to name a few. She also arranges for speakers to visit Ireland as part of Softtest Ireland and blogs about her testing experience and offers coaching at mavericktester.com

In part I of this month’s interview, we learn what motivates Anne-Marie to coach via Skype, what’s caught her interest lately, how her book with James Bach is coming and what the biggest mis-conception about testing is. Come back tomorrow for part II.

uTest: In terms of writing, speaking and researching, you are one of the most active testers in the business. So we’ll start by asking you this: What hot topics within testing have captured your interest recently?

AMC: 2012 has kicked off with a flurry of activity. Key topics appear to be, How we learn, Rapid Test Management and more recently James Bach has been looking Exploratory Test Documentation.

It goes like this. Typically we write tests and charters as artifacts for other people as evidence of work performed. But writing is a lot more powerful than that, it has the ability to assist in design (think brainstorming in mind maps). Exploratory Test Documentation is about changing the purpose of writing from an end product to a by product.

I also like the way new conferences and peer workshops are happening at a grass roots level, for example Lets Test in Stockholm. These are not necessarily big conferences, but ones that offer value to testers and that encourage participation. I hope that this will be the conference circuit of the future!

uTest: You’ve made quite a name for yourself as a testing coach; offering advice to testers free of charge via Skype. In your experience, what areas require the most coaching on your part? In other words, what does a typical tester coaching session cover?

AMC: Often testers come looking for coaching in a particular skill (e.g Test Automation), but many fail to understand basic testing concepts such as: “What is testing?” and “How do you determine bugs?”

Understanding testing is key to improving your testing skill.  After all, if you don’t understand something, how can you improve it?

Software delivery typically doesn’t allow for this type of introspection. Our jobs demand we focus on delivery, often to the detriment of how well we are doing our testing.

Coaching is the breathing space that all testers need to learn and grow.

In coaching I encourage testers to work through tasks to acquire skill. I’m there to guide and help them, but they need to work out the answers. That way, their learning experience is deeper and more meaningful and empowering.

Continue Reading