Software Testing Blog

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

Read more…

So far, our Guest Blogger series has demonstrated the incredible domain expertise of our community – and this post will be no exception. With us this month is uTester David Vydra. A resident of San Mateo, California, David is an Agile Tester for Guidewire Software (he’ll explain what they do in a bit). If his name sounds familiar, that’s because he is the co-author of testdriven.com – a very popular testing site covering developer testing, automated exploratory testing, model-based testing and more.

In this post, David offers us his “notes from the field”, where he’ll be addressing the role of  software testers; criteria for hiring testers; the experts he follows and more. We’re confident that you’ll like it. Enjoy!

I am thrilled to be invited to share my testing experiences with the uTest community. I hope my account will encourage more organizations to adopt the agile way and more testers to find fun and fulfilling jobs.

I joined Guidewire Software about seven months ago as an Agile Tester. We make core software for the Property and Casualty insurance carriers. If you have car or homeowners insurance, you may be serviced using our software. Right from the start, Guidewire has used agile development methods and credits a good part of its success to this philosophy.

Our applications are complex. It typically takes several months for a tester to become fully productive because there is so much domain knowledge and in-house tooling to learn. Our applications are enviably flexible, and each installation is customized to fit the specific needs of the insurance company. In order to empower the customization process, we provide a number of development tools including a custom java-compatible scripting language, an IDE, a GUI framework, and a screen painter.

Read more…

Earlier this week, we published our three-part interview with Michael Bolton.  This was the latest installment in our monthly Testing The Limits series, in which we sit down with luminaries from the worlds of testing, development, crowdsourcing or startup life.  As part of this discussion, we asked Michael for his take on the issue of testing certifications (as we’ve done with Matt Heusser and James Bach in previous months).

In response to what she felt was “cert-bashing,” Charity Stoner of ProtoTest has written a post defending test certifications.  Since we always encourage civil discourse and open-minded debate — and since the purpose of  the Testing The Limits series is to offer up different perspectives from around the world of software — I wanted to shine a light on this post.

What do you think about test certifications?  Do they provide testers with a toolkit that complements their experience and adds real value?  Are they a marketing mechanism that limits what it means to be a professional software tester?  Or is it somewhere in the middle?  I’d love to hear your thoughts.

We’re thrilled to have Michael Bolton as the latest victim of our Testing the Limits series. As the founder of DevelopSense, Michael has traveled the world teaching the craft of software testing to businesses and individuals alike. Since 2005, he has specialized in courses on Rapid Software Testing – which he co-authored with James Bach. Michael is also a prolific writer, and his publications include hundreds of articles, essays and columns. Aside from his blog, you can keep tabs on his latest work through Twitter.

In Part I of the “trilogy” we discuss the Weekend Testers, testing abroad, how numbers can enslave managers, and of course, his pop-star namesake.

uTest: You’ve been a thought leader in the testing space for a while now, but people still seem to get you confused with Michael Bolton (the singer) on Twitter. Ever thought about creating a tester alias? Or have you considered asking him to change his name since “he’s the one that sucks.” Assuming you (and our readers) have seen Office Space, I bet this joke never gets old.

MB: Yeah, it never gets old.  Try renting a car with this name.

A couple of things on that. First, Office Space captures very well what it’s like to have my name. Second, it’s not his real name; he changed it already. Way back when, before Office Space, I was working in tech support at Quarterdeck Canada.  American callers would occasionally turn north when there were long phone queues in Santa Monica. On one call, when I introduced myself to the customer, he laughed. “Really? That’s your real name?” “Yes, really,” I said, expecting one of the usual jokes. He said, “You know, it isn’t his real name. I used to be his bass player.”  The singer’s real name is Bolotin, but according to the bass player, there was no hope that radio DJs would ever pronounce “Bolotin” right, so he changed it.

uTest: We recently interviewed your friend and colleague James Bach, who had high praise for a group called the Weekend Testers. Can you give our readers a quick recap of what this group does, and whether or not you’re on board with their testing philosophy?

Read more…

After we re-launched our brand in May, we decided that the uTest blog needed to be more than just uTest employees talking about uTest events, uTest awards and the uTest community (see how repetitive that gets?).

Writing witty, thought-provoking content is really hard.  And we’re pretty lazy, but fortunately we know some extremely smart & funny people.  So we invented the Testing The Limits series, in which we interview leaders from the worlds of testing, software, entrepreneurship and crowdsourcing.

We’re immensely grateful to these talented, busy people, and we have much more planned for the Testing The Limits series in 2010.  But before we flip the calendar, these posts from this year are worth another look:

June: James Whittaker – Author, Professor and Testing Evangelist at Google

July: Rosie Sherry — Founder of the UK-based Software Testing Club

August: Andrew Muns — President of Software Test & Performance

September: Jack Margo — SVP of Internet Operations of Developer Shed

October: Jon Winsor — Author, Crowdsourcing Expert, and Founder of Victors & Spoils

November: Matt Heusser — Software Testing Author, Professor and Testing Manager

December: James Bach — Software Testing Author, Teacher and Speaker

We have some great guests and ideas lined up for 2010, including software execs, QA thought leaders, and famous journalists & authors.  As always, the goal of Testing The Limits will be to inform, to entertain, and above all else, to help our readers get to know these thought leaders who are worth following and listening to.

Have a suggestion for a future Testing The Limits guest?  Drop us a note or tell us in the comments section.

Yesterday we posted Part 1 of our interview with James Bach, where he discussed tester certifications, faking test projects, his latest book and wide range of other topics (including life as a freelance sentry in a parallel universe). Today, for Part 2, we discuss tips for automated checking, what makes a good tester a great tester, his flying lessons and much more. Enjoy!

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

Read more…

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

Read more…

Earlier this week, several members of the uTest team took in the opening reception of Software Test & Performance’s STPCon event.  We overheard some great conversations, and even jumped into the fray a few times ourselves.  It was a lively crowd and a great venue (the Hyatt Regency) right along the Charles River.  The event continues today, but even if you’re not attending you can still follow along, check out the STPCon Twitter stream here.

This event gave us a chance to reconnect with our friends from STP Collaborative, as well as sit in on sessions from top-shelf testing thinkers like James Bach, Jon Bach, Michael Bolton and Matt Heusser.  We also got to connect with about a half dozen local uTesters who took us up on our invitation to attend the reception.

We had a few people from uTest in attendance on Thursday and they gave the presenters high marks overall.  The two that I heard the most comments on, however, were from James Bach and Michael Bolton.  Bach tackled the provocative subject of “How to Fake a Test Project.”  Bolton took on the topic of “Rapid Software Testing.”  We’ll see if we can get either of their presentations and share them here. Also of great interest was Friday morning’s SpeedGeeking session with Matt Heusser, which put the speakers on the spot, giving them only five minutes to get right to their point. Talk about fighting your natural instincts!

For anyone who attended, chime in and share your most or least favorite moments from STPCon?  What surprised you, frightened you, entertained you or just generally pissed you off? Sound off in the comments or drop us a line.

Meanwhile, the uTest paparazzi was present and ready to capture some of the scenes from the evening.

Matt Heusser and James Bach take a timeout from their testing debate for a quick photo

Matt Johnston and James Bach deep in discussion about the art & science of testing (and the difference between the two)

Jennifer Moebius and STP chief, Andy Muns take a break from the exhibits to pose for a pic

Calling all New England QA and software testing professionals!

We will be co-hosting a free tester meetup with STP (Software Test & Performance) as part of the kickoff reception for their big event, STPCon 2009 at the Hyatt Regency Cambridge.  This meetup will be Wednesday, October 21 at 5:30pm.

Join us for a great evening of networking that will be held in the STPCon exhibits area. There, you’ll have the opportunity to connect with your peers, connect with execs from uTest and STP, discover new products and features and talk to the experts who created them.

Another great perk for attendees is that you’ll have the opportunity to discuss the latest and greatest trends with industry leaders such as James Bach and Michael Bolton.

If you’re around, it would be great to meet you in person!  To register, please visit: http://utest2009stpcon.eventbrite.com/.

This is the second half of our recent interview with Andrew Muns (@amuns), the president of Software Test & Performance.  Today, we’ll cover his thoughts on how testers can get more respect, predicting STP’s future, and who would win in a fight between James and Jon Bach.  If you missed it, check out the first half of the interview.

uTest: Testing is often viewed as a behind-the-scenes profession. What can testers do to bring their craft to light and make sure others understand the value?

A: Upper management at most companies may never truly understand what a test department contributes, especially since a contribution by definition goes unnoticed (i.e., something worked as expected.)  To me this sounds like a cultural issue: how to translate the value of testing into manager-speak.  Managers like things they can measure, so speaking their language means associating a measurable value on something vital but difficult to observe.

Software Test & Performance magazine has written many features on this question, but as a manager more than a tester, here is one argument I like (that applies more to consumer-facing applications): explain QA as a marketing function.  How much does your company spend on marketing?  Why would testing merit less investment?  I bet your company would spend a lot to spread positive word-of-mouth from users.  Shouldn’t management be willing to spend the same amount or more to avoid negative word-of-mouth?  As United Airlines learned after breaking a customer’s guitar, negative word of mouth can be viral.

Critically, neither this argument, nor any other, will be made if testers themselves don’t make it!

uTest: Is James Bach really as smart as we think he is?  Who would win in a fight between him and his brother, Jon?

Read more…