Software Testing Blog

Yesterday we posted Part 1 of our interview with James Bach, where he discussed tester certifications, faking test projects, his latest book and wide range of other topics (including life as a freelance sentry in a parallel universe). Today, for Part 2, we discuss tips for automated checking, what makes a good tester a great tester, his flying lessons and much more. Enjoy!

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

Read more…

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

Read more…

Today, we finish up our “Testing The Limits” interview with Matt Heusser.  Be sure to check out part 1 of this interview.  In this installation, we’ll discuss which mobile app testing, his OS of choice, and the testing blogs and sites Matt reads.

What other blogs, sites or message boards do you read to stay on the leading edge of all things testing?
MH: I’ve got a bit of a bias there, as I write a monthly column for Software Test and Performance Magazine. You can get the PDF version for free every month. I also subscribe to Better Software Magazine. For communities, I like softwaretestingclub.com and the agile-testing discussion list. For blogs, well, there’s James Bach, Cem Kaner, and Michael Bolton. Adam Goucher has been doing a lot of writing lately, including editing the new book, Beautiful Testing, which I contributed a chapter to. And I spent a fair amount of time working on my own blog, “Testing At The Edge of Chaos.”

Recently I’ve been getting to know people by working on projects with them; my friend Chris McMahon started a Google Group on Writing About Testing which I found to be a blast. Through that group (and the Miagi-Do School) I’ve met quite a few new bloggers: Markus Gaertner, Lanette Creamer, Marlena Compton, and Ajay Balamurugadas come immediately to mind.

What OS are you running right now? What’s your browser of choice? Anti-virus? Inquiring minds would like to know.
MH: Max OS X, and I have to support all of them, so I run all browsers. Anti-Virus? Dude, I told you, I use a mac, and I try to avoid the questionable websites that host the viruses. What have you been browsing lately, Mr. Johnston?  (Ed. Note: I’ll ask the questions around here.)

Read more…

In this month’s installment of “Testing The Limits”, we sit down with Matt Heusser (@mheusser) — prolific blogger for STPCollaborative, thought leader and testing extraordinaire.  We’ll discuss the state of software testing, SpeedGeeking, the role of chaos in testing software, and the lack of fistfights at STPCon 2009

uTest:  We loved the SpeedGeeking session you led at STPCon, so we’re going to flip it on you – If you had just five minutes to teach, motivate or inspire the uTest audience about software testing, what would you say?
MH: Well, I’d start by asking the audience what they are doing today – what’s the greatest point or opportunity they feel – and asking what options they see to improve. Most of the time, I hear that testing is “too slow” or “the bottleneck” or something like that.

So I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting – it is very important the team stop the time tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the numbers for improvement. For example, many of the people I talk to can spend 80% of their time or more in meetings, working on documentation, working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from 80/20 to 60/40 will double the amount of time the team spends actually doing testing.

Another thing to look at is the amount of time spent trying to reproduce defects, document defects, file bug reports, “verify” fixes, and so on. We think of these activities as testing, and they can take a substantial chunk of that 20% – but they are really accidental. That’s not a testing bottleneck – it is a development bottleneck. If test can work with development to improve the quality of the software prior to code complete, that will improve the speed of the whole system. Realizing this, and having a little bit of data to “prove” it, can help the entire system improve.

So if I had five minutes, I would say start with measuring how you track your time, and ask yourself if this is the best use of your time and what can change. Sometimes, the big boss will say “no, we absolutely need you to fill out all seven pages of documentation per test run”, and you can say “ok.”  Six months from now, when someone asks why the big project is late, you can point out that the business made an explicit decision to pay the full price of defined process. You presented options and those were not accepted.

That won’t save this project — but it might save the next.  It also turns out that actually testing tends to be much more fulfilling than documentation and compliance activities. Who could have guessed?

Lots of contrasting opinions at last month’s STP Conference. While there were no fist fights (that we heard about anyway), what did you see as the most contentious issue? And where do you fall on this issue?

Read more…

Having grilled some of the top minds in the software business, this installment of Testing the Limits will deviate slightly from the norm. With us this month is John Winsor – author, entrepreneur and crowdsourcing expert.

After a successful career as a journalist and magazine publisher, John founded Radar Communications in 1998, where he implemented a variety of academic-based market intelligence tools to help some of the country’s most progressive companies learn from key voices in their communities. Today, he offers that same advice as the VP/Executive Director of Strategy and Product Innovation at Crispin, Porter + Bogusky.

John has written extensively on the subject of crowdsourcing, having published the popular 2005 book Spark: Be more Innovative through Co-Creation. With his latest book Baked In: Creating Products and Businesses That Market Themselves now hitting the shelves, John was kind enough to sit down with us to discuss the future of crowdsourcing, the premise of his new book, and the best (or worst) rock-climbing movies of all-time.

uTest: The hottest debate in crowdsourcing right now is the “fall” of traditional advertising or design firms and the “rise” of crowdsourced services. In your opinion, what does the future of crowdsourcing look like? Is the world ready for what you call the “digital tsunami?”

JW: Well the future of crowdsourcing is definitely bright, but there are still a lot of unanswered questions in people’s minds. Those who are skeptical of crowdsourcing question its ability to truly connect people. With crowdsourcing, you no longer have all of these professionals working together in the same building – that alone is often too much for some people to come to terms with. The idea of a crowd aggregating to solve business problems in a virtual environment is entirely new to most people, even though the underlying trend has been developing for years. The difference now is that it simply can’t be ignored.

uTest: So you see crowdsourcing as eventually obtaining mainstream acceptance?

JW: Absolutely. People are starting to see the full potential of this model, especially on the client side of the equation. There was a time when most people viewed crowdsourcing as chaos – like the inmates running the asylum, and that’s no longer the case for a growing number of people. So I think we’re just getting started.

Read more…

Today, we’re wrapping up with our September ‘Testing the Limits’ with Jack Margo, SVP of Internet Operations at Developer Shed.  If you’re not familiar with them, Developer Shed operates 20+ tech sites — many devoted to open source products, developers and communities.  They serve millions of visitors per month, for every breed of developer or open source project.

Today, we talk about the future of crowdsourcing, whether mobile apps are stealing web apps’ mojo, and who would win in a fight between the Agile methodology and a recession.  Also, Jack causes everyone to look up the word “bifurcated“.  In case you missed it, check out Friday’s part 1 of the interview with Jack.

uTest: This question was recently posed by blogger Andy Beal: “Crowdsourcing is hot now, but will participation fatigue set in?”
J: Sure.  And Facebook is a fad.  And Twitter won’t last 6 months.  I’ve seen crowdsourcing in action… and a good friend of mine, Chuck Lin, has used a site called crowdspring.com to get a ton of great, affordable designs.  People in our industry are motivated by three things – Money, Notoriety and Discover.  Crowdsourcing, by it’s nature, filters out the not-so-good and leaves you with the best ideation that a group of collaborating individuals can provide.  (Editor’s note: we heart crowdSPRING too!)

uTest: Some have suggested that the focus on mobile apps will weaken web applications. Any truth to this, in your opinion?
J: Not at all.  It’s just a different methodology to the build itself, but there are ample developers and specialists to go around.  I’ve not done a lot of mobile application development but the ramp up of something like Objective-C (one of the most crazy languages, in my opinion) is steep for most.  I think you’ll see a convergence when Android and other phones start to offer Flash as a development platform.  The main weakening, if there is one, is mainly from adhoc rules that cell phone manufacturers impose.  Make a phone that can run PHP applications, and you’ll have a ton of web/mobile-app developers.  Likewise, make a website that can run on Objective-C and you’ll get the converse.  It’s one’s chosen discipline, but I definitely do not see a weak web app dev community.  Bifurcated?  Yes.  But not weaker.

uTest: Which development methodology (waterfall, agile, XP, etc) has the brightest future?

Read more…

In recent months, we’ve ‘tested the limits’ with QA notables like  James Whittaker, Rosie Sherry and Andrew Muns.  This month, we’re jumping over to the dev side of the aisle by sitting down with Jack Margo, SVP of Internet Operations at Developer Shed .

Developer Shed is owned by Ziff-Davis and manages a bunch of tech sites — many devoted to open source technologies and communities.  They serve millions of visitors per month, for every breed of developer. Topics range from troubleshooting an Apache web server to programming a complicated Java application to successfully marketing a website.  Their tagline says it all: “Tools for Geeks!”

Today, we talk about what developers really think about testers, Jack’s take on Microsoft vs. open source, the reason he’s mad at Java, why net books are a fad, and which programming language has the biggest upside.  Check back tomorrow for part 2.

uTest: What do developers look for in their testing counterparts?
Jack: First off, most developers will ultimately hate their testing counterparts. The best developers have an almost g-d like complex where they think their code is always solid and their work infallable.  We know that is not the case.  A developer needs, in a good testing counterpart, a person who understands this and can reach the developer on a personal level.

Testers need to really keep reminding the developers that it’s not personal.  On the other hand, it’s important to state that not every functionality issue is a bug… I’ve had issues where a business spec was delivered, my team developed to spec, but the UI was just not right and the tester opened bugs against the developer.  It’s important to have a tester that can tell the difference between an enhancement and a true bug.  I know, it sounds so ridiculously trivial but finding quality testers who can also understand the nuances of business is key.

uTest: What’s the most overused buzzword in the lexicon of software apps?

Read more…

This is the second half of our recent interview with Andrew Muns (@amuns), the president of Software Test & Performance.  Today, we’ll cover his thoughts on how testers can get more respect, predicting STP’s future, and who would win in a fight between James and Jon Bach.  If you missed it, check out the first half of the interview.

uTest: Testing is often viewed as a behind-the-scenes profession. What can testers do to bring their craft to light and make sure others understand the value?

A: Upper management at most companies may never truly understand what a test department contributes, especially since a contribution by definition goes unnoticed (i.e., something worked as expected.)  To me this sounds like a cultural issue: how to translate the value of testing into manager-speak.  Managers like things they can measure, so speaking their language means associating a measurable value on something vital but difficult to observe.

Software Test & Performance magazine has written many features on this question, but as a manager more than a tester, here is one argument I like (that applies more to consumer-facing applications): explain QA as a marketing function.  How much does your company spend on marketing?  Why would testing merit less investment?  I bet your company would spend a lot to spread positive word-of-mouth from users.  Shouldn’t management be willing to spend the same amount or more to avoid negative word-of-mouth?  As United Airlines learned after breaking a customer’s guitar, negative word of mouth can be viral.

Critically, neither this argument, nor any other, will be made if testers themselves don’t make it!

uTest: Is James Bach really as smart as we think he is?  Who would win in a fight between him and his brother, Jon?

Read more…

In the latest installment of our “Testing the Limits” series, we sat down with Andrew Muns (@amuns) the President of Software Test & Performance (of STP Magazine and STPCon fame), to discuss how testers are perceived by execs and developers, the future of media companies, and the changes that are underway at STP.  This is the first half of our chat; check back Thursday for part two.

uTest:  STPCon is being held this October in Cambridge, MA… what do you have in store for the attendees this year?

Andrew: This is the first conference that will have been planned start to finish by Redwood Collaborative Media and was designed according to our very simple philosophy: “ask your audience what they want and give it to them.”  The show’s program was designed largely  based on a survey in which we asked two things, what topics are most important to you and who do you want to hear from.

The most requested topics among our readers were Test Automation, Performance Testing, Test Management and Agile.  We’ve built a five track program with specialized training and workshops for each of these four areas, plus a track we call “FutureTest.”  The concept of FutureTest is to take a look ahead to emerging tools, technologies and practices – to help our members stay on the cutting edge of the testing industry.

We’ve got only all-stars here (check out the full roster) plus a keynote by a NASA astronaut, Mike Mullane, who will talk about leadership and the organizational culture that led to one of the most tragic QA mistakes in history: the O-ring of the space shuttle Challenger.  Michael Bolton, will then use this story as a launching point (pardon the pun) to talk about test leadership.  It’s going to be a phenomenal event.

uTest:  You recently launched STPCollaborative.com. Tell us the purpose of this site and what’s so different about it.

Read more…

In the second installment of our “Testing the Limits” series, we sat down with Rosie Sherry, founder of the Software Testing Club, to discuss the state of the profession, advice for QA beginners, “flash mob” testing and much more.

uTest: Where did you get the idea for the Software Testing Club?

RS: It was fairly simple really: I wanted a place to meet like-minded software testing professionals. Everything else out there at the time (this was 2 years ago) didn’t meet this need at all. I was doing a fair amount of blogging on software testing then, and so starting a community seemed like the next logical step.

uTest: Considering the growth of the community over the past few years, have your expectations changed?

RS: My expectations have definitely changed – they had to. For instance, I have stopped new members from joining for free since that model is now completely unsustainable. Like I said, I never expected to have created a community this large. I’d prefer something much smaller where we can give members the attention they deserve, but we’re obviously not going to turn anyone away. If you’re interested in software testing, you’re welcome in our community.

uTest: Based on your observations, what is the most contested debate in the field of software testing today?

RS: The biggest ongoing debate right now is software testing certification. A lot of people maintain that some companies/people put too much emphasis on the certification process, and that much of what is learned deals with how to pass an exam, rather than learning true testing skills. It’s definitely a lively debate with many viewpoints, so I’m not going to get too deep into it right now.

Read more…