Testing the Limits with Cem Kaner – Part II

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview.

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

Continue Reading

Testing the Limits With Cem Kaner – Part I

After almost a year of being told that we “have to interview Kaner” by previous Testing the Limits guests and readers, we exercised our listening skills and sought him out. With us this month to share his unique brand of wit and wisdom is Dr. Cem Kaner – author, lawyer, speaker, professor and one of the most respected minds in the testing world.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

Continue Reading

Testing the Limits with James Whittaker – Part I

It was one year ago (June ’09) that James Whittaker helped us christen our ‘Testing The Limits’ interview series by being our first guest. And for much of the year, he held the distinction of generating the most page views… and then some guy named Patrick Copeland came along and took the lead a few months back.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

Continue Reading

James Bach on “Buccaneer Testing”

Back in December, we interviewed the notable, quotable James Bach as part of our Testing the Limits series. At the time, James had just published Secrets of a Buccaneer-Scholar, where he advocates seeking out alternatives to traditional schooling. For software testers, this meant looking beyond industry certifications.

About a month ago, James was again interviewed on the subject of “buccaneer testing”, except this time it was caught on camera by Yvette Francino of TechTarget (while covering StarEast 2010).

Check out this inspiring, two-minute explanation of buccaneer testing:

Happy buccaneer testing, mateys!

Testing the Limits with Lanette Creamer – Part II

In part II of our Testing the Limits interview with Lanette Creamer – aka “Testy Redhead” – we cover the need for Exploratory Testing; Matt Heusser and the “rebel alliance” of testing; how to create a popular testing blog; her stance on tester certifications and more from the wide world of QA. Catch up by reading part I, and when you’re done with this one, go check out part III.

uTest: In one of your recent blog posts, you mention Elisabeth Hendrickson’s STAREAST declaration that “Exploratory Testing Is Not Optional.” Why did this statement resonate so strongly for you? Do you think all testing managers should follow Hendrickson’s lead?

LC: As a frequent conference attendee and enthusiastic reader of testing blogs, I’ve seen many ideas about how to improve testing. I’ve been through countless industry trends, such as borrowing manufacturing ideas, extensive measuring schemes, and repeated attempts to automate all testing. The bottom line is exploratory testing works in practice. Not for a few months or years, but it works to find important bugs year after year no matter what other quality trends are happening. It works well side by side with automated checks, manufacturing ideas, and it can be used with session based test management to provide measures and metrics if needed. It is the one constant a tester can go back to and find bugs that impact the user experience. It is the meat in my testing sandwich. (My pun filled humor is the cheese.) To hear Elisabeth acknowledge the importance of exploratory testing in public shows me that agile testing is about more than just automation. Agile testing can be about a balanced approach to overall quality. It resonated with me strongly because it makes me hopeful for the future of testing on agile teams.

I think test managers should evangelize and defend what works well in practice on their teams. My experience has been that exploratory testing is generally undervalued considering how effective and practical it is.

uTest: What’s the deal with this “rebel alliance” thing we’ve been hearing so much about? It sounds subversive – we want in! Seriously though, what’s the mission of this group? Please explain it to our un-initiated readers.

Continue Reading

Testing the Limits with Lanette Creamer – Part I

Next up in our Testing the Limits series is Lanette Creamer. Known to many in the QA blogosphere as “Testy Redhead”, Lanette has over ten years of experience in the software industry, including her current role as Quality Lead with Adobe. Like many of our guests, she writes a popular testing blog, publishes technical papers and has been known to speak at a conference or two. And yes, she’s on Twitter.

In part I of our interview, we get her thoughts on testers vs. hardware; the idea of “quality advocacy”; why unemployed testers should study The Price is Right;  life as a shift manager at a charity bingo parlor; and much more. When you’re done with one, be sure to check out part II.

uTest: What’s the biggest trend/challenge in testing that no one’s talking about yet?
LC: Testers are breaking out of the office like William Wallace, but with laptops, not swords. How much more affordable is it for a company to buy a great laptop every few years than all sorts of different hardware? Let someone else manage the machines so we can focus on the testing. Of course, this isn’t appropriate for every context, but I’m interested in going beyond multi-boot systems, local images, and to truly getting out of the business of managing hardware. I’m interested in cloud-based imaging. Part of my personal strategy of investing in one laptop that can run multiple operating systems is the temping ability to verify the scope of a bug on one machine. To do that without even rebooting with more built-in logging and debugging tools is really the next step to freedom from hardware and location-reliant testing.

uTest: In the last year, we’ve noticed you blog about the prospect of unemployment. What advice do you have for other testers who find themselves in this situation? Should they just wake up at noon, watch The Price is Right and eat nachos until a hiring manager comes knocking on the door? Or should they try to keep their skills sharp? If it’s the latter, then please elaborate on how to go about this.
LC: The Price is Right can teach you something amazing about interviewing. Have you ever noticed who they pick? It is the most enthusiastic people with the best stories. Come on down, job candidate! You’re the next contestant on The Job is Right. Can you imagine what The Price is Right would be like if they picked a contestant who was just above it all? Laughed at the fabulous prizes? Ignored the host? Win or lose, play the job interview game with style and be memorable. Also, I do like nachos. Layer the cheese and make them in the oven.

Well, rather than bouts of unemployment, I’ve been facing one very long pending layoff. I’ve not yet experienced the unemployment part, so maybe your readers can help me out with their advice when that happens. As a part of the CS5 team, my layoff isn’t effective until June, and it impacted every tester on my current team. The day I first became aware of my pending layoff, I felt a bit powerless. I realized that it was really up to me to decide what to do next. I decided that I wanted to end well and finish the project, and I really wanted to complete my 10 years at Adobe. I am proud of my work for my entire career at Adobe, and that hasn’t changed with my layoff notice.

Here’s what I recommend for those of you working in a job that you know is ending:

Continue Reading

Testing the Limits With Jon Bach – Part I

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

Continue Reading

Testing the Limits with Google’s Patrick Copeland – Part II

In Part II of our interview with Google’s Patrick Copeland, we discuss the challenges of managing a global engineering team; rewarding developers with food pellets; the difference between a good tester and a great tester; and why some companies will never launch a high-quality app. By the way, did you miss Part I of our interview?

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

Continue Reading

Testing the Limits with Google’s Patrick Copeland – Part I

In this month’s Testing the Limits interview, we’ll put Patrick Copeland on the hot seat. Patrick is the Senior Engineering Director for a promising young upstart named Google (we’re not familiar with them ourselves, but we’ve heard good things) where he oversees a global team of about 800 engineers. But this isn’t his first rodeo –  prior to Google, Patrick spent a decade at Microsoft, where he specialized in all things related to software engineering.

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

  • Strategic: Maybe a new feature, but in a market with existing competition (like Android).
  • Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
  • Customer usage: For example, popular high-traffic applications like GMail.
  • Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
  • Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

Continue Reading

On Tour With Michael Bolton

When we published our Michael Bolton interviews earlier this week, we forgot to mention that he’ll be speaking at a bunch of  testing events/seminars in the months ahead. So if you’re in these areas and want to see Michael speak in person (as opposed to Youtube) you owe it yourself to attend.

That said, here are a few upcoming testing events:

  • “Why is Testing Taking So Long?” – Michael will be giving an interactive talk on this topic at the Markham meeting for TASSQ, the Toronto Association of System and Software Quality, on Wednesday, February 10, 2010.
  • The Conference on Free Testing Tools: Michael will be giving a three-day public offering of his Rapid Software Testing course, and will deliver the keynote talk. The event will be sponsored by the Norwegian Computer Society, and will be held in Trondheim, Norway, March 22-26, 2010.

Continue Reading