Software Testing Blog

In the third and final installment of our Testing the Limits interview with Ben Simo, we go back in time to the early 90s to find out how and why he entered the testing profession. We also rapid fire some questions on his browser of choice, his hardware preferences, hobbies and more. In case you missed them, here’s part I and part II.

*********

uTest: Let’s go back in time for a second: How did you get into the craft? What was the first application you tested? What was testing like back in the early 90s?

Simo: Providence. It was providence that got me into testing.

I was young, in love, and planning to get married.  I had been doing some part time database development work, but needed a full time job before the wedding.  I submitted letters and resumes to dozens of companies. I was willing to do almost anything that would pay the rent.  I lived in a city where the local job market was dominated by defense contractors. I quickly learned that many of them called nearly everyone who applied for anything in for an interview; so they could learn about people and add them to databases of potential hires for matching to work they did not yet have.  These companies would then present these people to the government as their available workforce when bidding on contracts. This made it frustrating for those of us looking for work. It often wasn’t clear, when going in for an interview, if it was for a real job or for a potential position that might come at some time in the future if that company were to be awarded a government contract.

I interviewed with the company for which my fiancé (now my wife of 19 years) Sophie worked. It appeared to be one of those information gathering interviews without an actual position to fill. I was asked a lot of questions but none seemed related to a specific opening.  At the end of the interview, the interviewer said he’d be calling me.  Time went by without any more contact. Nearly a month later, I got a call asking if I could start the next morning.

Read more…

In part II of our Testing the Limits interview with Ben Simo, we’ll discuss whether you should trust automated testing tools; the proliferation of testers on Twitter; the true meaning of “QA”; how testing evolves differently in each company; the long lost Bach brothers and much more. You can catch up on the conversation by reading part I. We’ll wrap things up tomorrow with part III.

********

uTest: Jon Bach mentioned that changing the meaning of “QA” to Quality Assistance would help outsiders (engineers, executives, et al) better understand the role of this discipline.  Agree or disagree?

Simo: I believe I first heard  “Quality Assistance”  from Cem Kaner.  I agree with Jon. When testers bear the title Quality Assurance, it often implies that they actually assure the quality of other people’s work. Testers are in a position to help assist quality; not assure it. Let’s not assist the setting of unrealistic expectations with inappropriate titles.

uTest: While we’re on the subject, are you anyway related to James and Jon Bach? The resemblance is uncanny.
Simo: I don’t think so. I’m available for adoption if the Bach family is interested.

uTest: You’ve said that you frequently use automated tools, but that you don’t trust them entirely (back to that whole defensive pessimist thing again). What advice do you have for testers and managers wanting to strike a healthy balance? And what’s currently in your arsenal of automated tools?

Simo: My mistrust in tools is based on the fact that tools can’t think for me. Automated checking can only process whatever decision rules someone thought to program when the checks were created. Automation will consistently do what it is programmed to do and consistently not do what it is not explicitly programmed to do. I find test automation to be useful. In fact, there are some things I’d not want to even try to do manually. I do, however, distrust the green bar. When automated checking passes, I ask myself what the automation does not tell me. I also try to keep aware that people who don’t understand what the automation does are likely to assume that it does more than it does.

Tools are much more than test automation. Tools are essential for testing. I don’t want to test without tools. I have some old programming books that promote testing in which a programmer manually executes code, step-by-step, with pencil and paper in order verify that the code works as expected. This is manual testing. This is a testing practice that came from a time when computer time was rare and cost more than people. We’d now laugh at someone proposing testing in this manner.

Read more…

Our Testing the Limits guest this month is Ben Simo. Known as the “Quality Frog” on Twitter, Ben is one of the most insightful and entertaining testers in the business. A proponent of the context-driven school, Ben has more than 19 years of experience testing software and developing testing tools. He currently lives in Colorado with his wife, two children, two dogs, five cats and fourteen – count ‘em – fourteen goldfish. For the full Ben Simo experience, go to his blog.

In part I of our interview, we get his thoughts on the Worst Bug Ever; his testing philosophy; what it means to be a defensive pessimist; testing certifications, the state of the industry and more. Be sure to check tomorrow for part II.

**************

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

Read more…

The good folks over at Software Test Professionals want to remind you about a very important election this Fall. No, we’re not talking about the U.S. Congress. And no, we’re not referring to American Idol either (at least not in this post).  Instead, we’re talking about something lasting and meaningful: the 1st Annual Luminary Award.

As described on their award page, this honor will “recognize a person in the software testing and quality community, who inspires others and dedicates their career to industry advancement.” The organizers were looking for someone who has dedicated their career to the betterment of software testing and quality; who has shown exceptional leadership and who has educated, promoted and published on behalf of the industry. In other words, a software testing luminary.

With that type of criteria in mind, we’re not surprised to see Cem Kaner, James Bach and Jerry Weinberg as this year’s finalists. You may know Kaner and Bach from our recent Testing the Limits interviews (Jerry, if you’re reading this, we’d love to have you as a guest as well). But in case you’re unfamiliar with these testing giants, here are clips from their award bios:

Read more…

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview.

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

Read more…

After almost a year of being told that we “have to interview Kaner” by previous Testing the Limits guests and readers, we exercised our listening skills and sought him out. With us this month to share his unique brand of wit and wisdom is Dr. Cem Kaner – author, lawyer, speaker, professor and one of the most respected minds in the testing world.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

Read more…

It was one year ago (June ’09) that James Whittaker helped us christen our ‘Testing The Limits’ interview series by being our first guest. And for much of the year, he held the distinction of generating the most page views… and then some guy named Patrick Copeland came along and took the lead a few months back.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

Read more…

Back in December, we interviewed the notable, quotable James Bach as part of our Testing the Limits series. At the time, James had just published Secrets of a Buccaneer-Scholar, where he advocates seeking out alternatives to traditional schooling. For software testers, this meant looking beyond industry certifications.

About a month ago, James was again interviewed on the subject of “buccaneer testing”, except this time it was caught on camera by Yvette Francino of TechTarget (while covering StarEast 2010).

Check out this inspiring, two-minute explanation of buccaneer testing:

Happy buccaneer testing, mateys!

In part II of our Testing the Limits interview with Lanette Creamer – aka “Testy Redhead” – we cover the need for Exploratory Testing; Matt Heusser and the “rebel alliance” of testing; how to create a popular testing blog; her stance on tester certifications and more from the wide world of QA. Catch up by reading part I, and when you’re done with this one, go check out part III.

uTest: In one of your recent blog posts, you mention Elisabeth Hendrickson’s STAREAST declaration that “Exploratory Testing Is Not Optional.” Why did this statement resonate so strongly for you? Do you think all testing managers should follow Hendrickson’s lead?

LC: As a frequent conference attendee and enthusiastic reader of testing blogs, I’ve seen many ideas about how to improve testing. I’ve been through countless industry trends, such as borrowing manufacturing ideas, extensive measuring schemes, and repeated attempts to automate all testing. The bottom line is exploratory testing works in practice. Not for a few months or years, but it works to find important bugs year after year no matter what other quality trends are happening. It works well side by side with automated checks, manufacturing ideas, and it can be used with session based test management to provide measures and metrics if needed. It is the one constant a tester can go back to and find bugs that impact the user experience. It is the meat in my testing sandwich. (My pun filled humor is the cheese.) To hear Elisabeth acknowledge the importance of exploratory testing in public shows me that agile testing is about more than just automation. Agile testing can be about a balanced approach to overall quality. It resonated with me strongly because it makes me hopeful for the future of testing on agile teams.

I think test managers should evangelize and defend what works well in practice on their teams. My experience has been that exploratory testing is generally undervalued considering how effective and practical it is.

uTest: What’s the deal with this “rebel alliance” thing we’ve been hearing so much about? It sounds subversive – we want in! Seriously though, what’s the mission of this group? Please explain it to our un-initiated readers.

Read more…

Next up in our Testing the Limits series is Lanette Creamer. Known to many in the QA blogosphere as “Testy Redhead”, Lanette has over ten years of experience in the software industry, including her current role as Quality Lead with Adobe. Like many of our guests, she writes a popular testing blog, publishes technical papers and has been known to speak at a conference or two. And yes, she’s on Twitter.

In part I of our interview, we get her thoughts on testers vs. hardware; the idea of “quality advocacy”; why unemployed testers should study The Price is Right;  life as a shift manager at a charity bingo parlor; and much more. When you’re done with one, be sure to check out part II.

uTest: What’s the biggest trend/challenge in testing that no one’s talking about yet?
LC: Testers are breaking out of the office like William Wallace, but with laptops, not swords. How much more affordable is it for a company to buy a great laptop every few years than all sorts of different hardware? Let someone else manage the machines so we can focus on the testing. Of course, this isn’t appropriate for every context, but I’m interested in going beyond multi-boot systems, local images, and to truly getting out of the business of managing hardware. I’m interested in cloud-based imaging. Part of my personal strategy of investing in one laptop that can run multiple operating systems is the temping ability to verify the scope of a bug on one machine. To do that without even rebooting with more built-in logging and debugging tools is really the next step to freedom from hardware and location-reliant testing.

uTest: In the last year, we’ve noticed you blog about the prospect of unemployment. What advice do you have for other testers who find themselves in this situation? Should they just wake up at noon, watch The Price is Right and eat nachos until a hiring manager comes knocking on the door? Or should they try to keep their skills sharp? If it’s the latter, then please elaborate on how to go about this.
LC: The Price is Right can teach you something amazing about interviewing. Have you ever noticed who they pick? It is the most enthusiastic people with the best stories. Come on down, job candidate! You’re the next contestant on The Job is Right. Can you imagine what The Price is Right would be like if they picked a contestant who was just above it all? Laughed at the fabulous prizes? Ignored the host? Win or lose, play the job interview game with style and be memorable. Also, I do like nachos. Layer the cheese and make them in the oven.

Well, rather than bouts of unemployment, I’ve been facing one very long pending layoff. I’ve not yet experienced the unemployment part, so maybe your readers can help me out with their advice when that happens. As a part of the CS5 team, my layoff isn’t effective until June, and it impacted every tester on my current team. The day I first became aware of my pending layoff, I felt a bit powerless. I realized that it was really up to me to decide what to do next. I decided that I wanted to end well and finish the project, and I really wanted to complete my 10 years at Adobe. I am proud of my work for my entire career at Adobe, and that hasn’t changed with my layoff notice.

Here’s what I recommend for those of you working in a job that you know is ending:

Read more…