Software Testing Blog

To kick off another amzing year of Testing the Limits we reached out to Anne-Marie Charrett, an independent tester who has worked for the likes of Mercury Interactive, IBM (twice) and Nortel – just to name a few. She also arranges for speakers to visit Ireland as part of Softtest Ireland and blogs about her testing experience and offers coaching at mavericktester.com. 

In part I of this month’s interview, we learn what motivates Anne-Marie to coach via Skype, what’s caught her interest lately, how her book with James Bach is coming and what the biggest mis-conception about testing is. Come back tomorrow for part II.

uTest: In terms of writing, speaking and researching, you are one of the most active testers in the business. So we’ll start by asking you this: What hot topics within testing have captured your interest recently?

AMC: 2012 has kicked off with a flurry of activity. Key topics appear to be, How we learn, Rapid Test Management and more recently James Bach has been looking Exploratory Test Documentation.

It goes like this. Typically we write tests and charters as artifacts for other people as evidence of work performed. But writing is a lot more powerful than that, it has the ability to assist in design (think brainstorming in mind maps). Exploratory Test Documentation is about changing the purpose of writing from an end product to a by product.

I also like the way new conferences and peer workshops are happening at a grass roots level, for example Lets Test in Stockholm. These are not necessarily big conferences, but ones that offer value to testers and that encourage participation. I hope that this will be the conference circuit of the future!

uTest: You’ve made quite a name for yourself as a testing coach; offering advice to testers free of charge via Skype. In your experience, what areas require the most coaching on your part? In other words, what does a typical tester coaching session cover?

AMC: Often testers come looking for coaching in a particular skill (e.g Test Automation), but many fail to understand basic testing concepts such as: “What is testing?” and “How do you determine bugs?”

Understanding testing is key to improving your testing skill.  After all, if you don’t understand something, how can you improve it?

Software delivery typically doesn’t allow for this type of introspection. Our jobs demand we focus on delivery, often to the detriment of how well we are doing our testing.

Coaching is the breathing space that all testers need to learn and grow.

In coaching I encourage testers to work through tasks to acquire skill. I’m there to guide and help them, but they need to work out the answers. That way, their learning experience is deeper and more meaningful and empowering.

Read more…

So you just unwrapped your brand new (insert gift here). You loved the old version and can’t wait to try out the latest one with all the new bells and whistles you’ve heard so much about. But shortly after opening said gift, you realize there’s a problem. Something’s different.

If your brand new gift was a Kindle Fire, that problem can likely be summed up in one (hyphenated) word: over-engineering.

Jakob Nielsen, the “King of Usability” and former Testing the Limits guest, recently published his usability findings on the Kindle Fire – and he’s not impressed. The report covers multiple aspects of the new tablet, but one area that caught my attention was his discussion of how the Fire compared to the first generation Kindle. Here’s his take (my take after the jump):

The Fire is a heavy object. It’s unpleasant to hold for extended periods of time. Unless you have forearm muscles like Popeye, you can’t comfortably sit and read an engaging novel all evening. The lack of physical buttons for turning the page also impedes on the reading experience for fiction. On the older Kindles, it’s easy to keep a finger on the button when all you use it for is to turn the page. In contrast, tapping an area of the screen disrupts reading enjoyment, is slightly error-prone, and leaves smudges on the screen. The Fire screen also has more glare than the traditional Kindle.

For reading fiction, the older Kindle design wins.

Read more…

Testers and programmers are two groups of people who should get along, but often don’t. It’s a sad fact of life that testers (by virtue of what they do) often bring bad news. And programmers, by virtue of what they do, are the source of the defects that create the bad news. Rather than both accepting that this is a reality of life and working together, they allow the relationship to become acrimonious.

James Bach is no stranger to this problem, and his latest blog post is a blueprint for making that relationship more productive and professional. Titled A Tester’s Commitments, James starts by writing:

Dear Programmer,

My job is to help you look good. My job is to support you as you create quality; to ease that burden instead of adding to it.

What follows are twelve commitments a tester should make to their programmers. They include things like:

• I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied.
• I will learn the product quickly, and make use of that knowledge to test more cleverly.
• I will not carelessly waste your time. Or if I do, I will learn from that mistake.

But James is not in usual form unless he invites controversy, and that first bullet struck quite a chord with some of his readers. Testers provide a service!? Since when?

Read more…

Our Testing the Limits “reunion tour” rolls on this month with Michael Bolton, back for another lively session of Q&A. Michael is best known as the founder of DevelopSense, his Toronto-based testing consulting firm, and as a leading figure in Rapid Testing and the Context-Driven school of testing. In short, he’s one of the industry’s most highly regarded writers, speakers and teachers – and it’s a real pleasure to have him back. For more on Michael, be sure to check out his website, blog or follow him on Twitter.

In part I of our healthy two-part interview, we get his thoughts on test cases not being related to testing; the sub-par debate skills of testers; the quality chain of command; objections to Rapid Testing and much more. Be sure to check back tomorrow for Part II. Enjoy!

uTest: It’s been almost two years since our last interview. Where does the time go? We’ve followed you pretty closely during that time (on Twitter, don’t worry), but for those who haven’t, what have they missed? New publications? New courses? New ideas on testing? What’s new with Michael Bolton?

MB: I’ve been traveling like crazy this year, and I’m booked pretty heavily through the end of the year.  I’m beginning to set up my schedule for next year—so if people would like to schedule an in-house class, now is a great time to ask.  For new publications How to Reduce the Cost of Testing, a new book edited by Matt Heusser and Govind Kulkarni, has just been released. I’m pleased to say that I’ve got a chapter in there, with a number of other members of our community.

I don’t specialize in new ideas in testing so much, but rather in refining and reframing ideas we’ve had for years in more specific and, I hope, more useful ways. The other thing that I love to do is to bring ideas from elsewhere into testing.  Currently I’m fascinated by the work of Harry Collins, who studies the sociology of science and the ways in which people develop knowledge and skill. Tacit and Explicit Knowledge is his most recent book; The Shape of Actions is older.  I’m most interested in the idea of repair, which is Collins’ notion for the ways in which people fix up information as they prepare to send it, or as they receive and interpret it.

As an example, I’m 5’ 8” tall.  If I ask you how tall I am in centimeters (and provide you with the ratio of 2.54 centimeters to the inch), you’ll probably do a little math in your head to translate 5’ 8” into 68 inches.  If you do that, it’s because you have tacit knowledge that a foot is 12 inches, and it’s quicker to do five times 12 in your head and add eight than to work it out on the calculator.  Then you’ll report that I’m 173 centimeters (or 172), rather than what the calculator tells you:  172.72.  If you round the answer up or down to a whole centimeter, it’s because you have tacit knowledge that the extra precision is useless when my height changes more than that with every breath. The calculator doesn’t know that, but people often fix up the interaction with the tool, applying that kind of tacit knowledge without noticing that they’re doing it.  Collins argues that we give calculators and computers and machines more credit than they deserve when we ascribe intelligence or knowledge to them, even when we do it casually or informally.

My latest hobby horse is definitely not new, but I’d like to have a go at it anyway.  I’d like to skewer the idea of the test case having any serious relationship to testing.  Test cases are typically examples of what the product should do. That’s important; we often need examples to help explicate requirements and desires. But examples are not tests, so I’d like to call those artifacts example cases or examples rather than test cases. They’re confirmatory, not exploratory; checks, not tests. Brian Marick has written a lot about examples; Matt Heusser has too; so has Gojko Adzic. James Bach has been railing about test cases for a long time.  Often test cases are overly elaborate, expensive to prepare and maintain.  They’d be even more expensive if testers didn’t repair them on the fly, inserting subtle variations making observations that the test case doesn’t specify.  Just as Collins suggests about machines, test cases get more credit than they deserve.  As Pradeep Soundarajan would say, the test case doesn’t find the bug.  The tester finds the bug, and the test case has a role in that.  Now: the development of checks and the interpretation of checks—those things require all kinds of sapience and skill.

A test, to me, is an investigation, not a bit of input and output for a function.  Yet people tend to think of testing in terms of test cases.  Even worse, people count test cases; and even worse than that, they count passing and failing test cases to measure the completeness of project work or testing work.  It’s like evaluating the quality of a newspaper by counting the number of stories in it without reference to the content, the quality of the writing, the quality of the investigation, the relevance of the report, whether a given article contains one story or a dozen, and so forth.  Counting stories would be a ludicrous way of measuring either the quality of the newspaper or the state of the world. Yet, it seems to me, many development and testing organizations try to observe and evaluate testing in this completely shallow and ridiculous way. They do that because seem to think about things in terms of units of production. Learning, discoveries, threats to value, management responses… none of these things are widgets. They not things, either, for that matter.

uTest: In a recent blog post, you wrote about the inability of some testers to properly frame tests, mainly because they haven’t been asked to. Generally speaking, what other qualities or skills do you find testers to be lacking in?

Read more…

In our recent interview with James Bach, we asked him (among other things) what skills he believed that most testers lacked. His answer: practical mathematics. With that in mind, our next Tester Challenge will, um, challenge you to solve these mixed operations problems. Good luck!

In part II of our latest Testing the Limits interview with James Bach, we discuss what it would take to get him back on the client side of testing; free testing tools that he’s currently using; required reading for new testers; his upcoming speaking/book tour, sniper rifling in Middle Earth and more. Enjoy!

If you missed part I, you can find it here.

uTest: Your brother recently made the leap to the client side at eBay….will we see James Bach cross back over as well?

JB: Well, maybe a super-villain out there buys out my company and hires me full time to be chief tester for their front corporation. I would wonder at first why I was being paid three million dollars a year to test a variation of Angry Birds, but the money would blunt my natural inquisitiveness (I have my wife’s medical bills to pay, after all).

Only after a British secret agent drops in (literally) at my office to confront me with the horrible truth (that the program I was testing was actually a doomsday device) would I and Cortana (an AI construct rendered as a perky young female that switched sides after spontaneously evolving a sense of ethics) manage to skillfully mis-report critical bugs prior to sign-off. This would lead to the destruction of the doomsday machine seconds after our daring escape through the test data exhaust port.

But I doubt that will happen.

Basically, it would take an unreasonable amount of money to make me give up my independence.

uTest: You wrote a great blog post recently on the subject of tool vendors – specifically, how they can avoid your wrath. We’ve also heard you recommend free tools in the past (“because they can be freely abandoned”). What tools (either paid or free) have you discovered recently and how have hey helped your testing efforts?

JB: I’ve been playing with “R”. It’s a free statistical analysis system.

There are several books on it. I love tools that help me work with complicated data.

The tool that has helped me most recently is my Canon solid state video recorder combined with my micro tripod. I think I will never do professional testing again without a camera rolling. It’s so helpful to be able to roll back the film and watch what I just did in order to reproduce a difficult problem.

uTest: You’ve been following the testing of medical devices rather closely, recently highlighting that the FDA had come to recognize the value of exploratory testing. Are there other particular industries that need to make this same realization? If so, which ones?

JB: Banking, insurance, and aerospace come to mind. I’m working with a major world bank as well as a major insurance company right now, so that feels good. They are both “standardizing” on Rapid Testing. Which means that thinking for one’s self is becoming their standard.

uTest: We know you’re not much of a proponent of testing textbooks, but we know you’re a big fan of books and literature in general. Do you have a “required reading” list for new testers? If so, what are some of titles we could expect to find on it?

Read more…

Back for another session on the Testing the Limits hot seat is James Bach – one of our most popular guests and one of the most widely recognized figures in testing today. A prolific author, speaker, coach and consultant, James has been disrupting the testing industry since 1987. For more on James’ background, his body of work and his testing philosophy in general, you can check out his blog, website or follow him on Twitter.

In part one of our latest interview, we get his thoughts on his previous life as a developer; how testers should interact with engineers; the biggest waste of time in testing today; the skills that most testers lack and more.

uTest: In a recent lecture you said that you hated being a developer, saying it was like completing 25 crossword puzzles every day (i.e. boring and repetitive). What’s the one piece of advice you would offer to those are who are thinking of making the transition from dev to testing?

JB: I advise: make that transition– but please, study testing as you do.

Testing is not just another programming problem. Yet, too often, a programmer mentality sees testing as just the process of manipulating software. Sometimes, programmers dream of robot armies to do their “testing.” The quest for the clever tool then eclipses the mission of excellent testing. This is a little like trying to invent an android that can talk to your wife so that you don’t have to.

You can use all your tech skills, programmers. It’s fine to write software. But what testing is really about is rapid, deep learning. To do that, you have to get your face, and all the rest of you, right up close to that product. You must wrestle with the product– yes– by hand. Or as we like to say where I come from: you need to do your testing sapiently.

uTest: In that same presentation you also talked about the importance of making friends with developers. Why is the tester-developer relationship so adversarial at times?

JB: I don’t know, why does my wife not like it when I report defects in her? People are strange that way…

Seriously, it’s easy to see the dynamic. Anyone who creates a piece of work and submits it for judgment is going to feel judged. That’s not a pleasant feeling. And the problem is compounded by testers who glibly declare that this or that little nit or nat is a “defect,” as if anything they personally don’t like is a quality problem for everybody. It is further compounded when programmers and testers are separated by large distances or other communication barriers, not to mention the process barriers.

Even though technical people are renowned and sought after for their social skills (I understand United Nations programmers were just about to solve the Arab-Israeli situation a few months ago, had it not been for the need to calm tensions between the Koreas.) testers and developers are people who see the world very differently. It’s a special challenge to relax and smile when a bug report seems to have been ignored.

Personally, I make a set of explicit commitments to any developer I work with on a project. I write them down and deliver them. This seems to help.

uTest: You’ve said many times that belief is a sin for testers – something we assume you’ve learned from experience. Was there anything in particular that led you to this truism? Or was it just years of experience?

JB: It’s not a matter of experience, but reason. Our job is vigilance. To lose vigilance is to abdicate our responsibility. Vigilance, in testing, means being a good skeptic. We must reject certainty in any form. We’re the Knights of May Be. To believe is to cease questioning; to fall asleep at our posts.

I use lots of information. I work hard not to believe it. This is my discipline.

uTest: What’s the biggest waste of time in testing today?

Read more…

Luminary: a person who has attained eminence in his or her field or is an inspiration to others

You’d be hard-pressed to find a profession with a wider range of ideas and personalities than that of software testing. This point is certainly not lost on our readers, as evidenced by the popularity of our Testing the Limits interview series. And it’s not lost on our good friends at Software Test Professionals, who have opened up nominations for the 2nd Annual Software Test Luminary Award.

More on the nomination process in a second, but first, a little bit about the award itself:

The Luminary award will honor any software testing and quality assurance professional who is determined, persistent, and committed to improving a process or methodology. They develop ideas, which when properly applied, have a positive impact on the end product, either by enhancing quality or performance or resulting in improved efficiencies for a particular process, team or organization. In addition, their contributions elevate the critical role of the software test profession within the software development process.

A luminary is someone who has inspired others by their actions and the results of those actions on the profession. They inspire others to pursue a software testing career. It is about how they have given back, and shared their knowledge and experience with others in order to advance the profession and improve the career paths of all practitioners. A luminary will typically be recognized and respected long after their days of practicing have ended.

If you recall, last year’s honor went to Gerald M, Weinberg, who edged out fellow nominees James Bach and Cem Kaner.

So who will be named this year’s Software Test Luminary? It’s your call. STP will gather nominations and submit the top 3 candidates for a final round of voting. The finalist will be announced at the Software Test Professionals Fall 2011 Conference, October 24-27 in Dallas, Texas.

Here’s a quick timeline of the events:

Read more…

With all the talk about the end of the world (hope you didn’t make plans this weekend), it seems fitting that we’d find an article proclaiming  the end of manual software testing. That’s right, a researcher from the University of Twente has developed a system that claims to eliminate the need for manual software testing. In other words, it eliminates the need for YOU! I’ll let that sink in….

The system is said to facilitate quick and accurate testing – and here’s how it supposedly works (via Forbes.com):

The testing phase for new software consists of three steps: developing the tests, running the tests and evaluating the results. These three steps are generally performed manually. Model-Based Testing is a method that automates all steps in the software testing process. When used properly, the method completely eliminates the need for manual software testing.

Model-Based Testing has a number of major advantages: it makes the software testing process faster, cheaper and more accurate. It is not uncommon for manual software testing to take anywhere from several months to years. Van der Bijl’s new system can significantly reduce the duration of the testing period and thus reduce costs. “We can reduce the duration of the testing phase by at least thirty percent. We were even able to reduce overall software development time for one of our customers by a factor of four.” Model-Based Testing is more accurate, because in principle there is no limit to the number of tests you run, says Van der Bijl. “If you want, you can even run a million tests.”

So is this really the end of manual software testing? I suspect you know the answer to that question. If not, here are a few great responses on this debate from StackOverFlow.com:

Read more…