Software Testing Blog

In part III of our of Testing the Limits interview with Cem Kaner, we discuss why “best practices” is merely a marketing tool; Silicon Valley of yesteryear; his upcoming CAST lecture on investment model and exploratory test automation; the blogs he reads and much more. To recap, here’s part I and part II.

uTest: What’s surprised you the most about the testing industry since you’ve been in the game?

Kaner: Context-dependence was a big surprise to me. It took me about ten years before I reluctantly accepted the idea that my favorite test techniques, attitudes, and life-cycle models were appropriate in situations similar to the ones where I developed my preferences, but not so appropriate in situations that were quite different.

It took me a long time to learn that developing software under a well-specified contract is a common and respectable activity that requires different tradeoffs from mass-market software sold to people who had no say in its design and buy only after it is ready to deliver and only if they like it. It took me a long time to sort out differences between scientific programming (where I started) and consumer software development. It took me even longer to see that testers working in an independent test lab operate under fundamentally different constraints from testers working inside the development company and provided different types of information. And it took well over a decade before I accepted the idea that two different development managers, running essentially the same project, could legitimately want different information from their testers and that it could make sense and be ethical for the two different test groups to structure their work differently, finding or being blind to or ignoring different classes of bugs, in order to satisfy the information needs of their key stakeholders.

The last major event in this chain happened 14 years and two books after I came to Silicon Valley. A client paid me to tour a lot of companies in California, Oregon and Washington. I gave a talk at each place, but I also talked with them about their business models, their testing challenges and methods.

Most of these were good companies with competent testing staff but they did things very differently from each other, often very differently from what I (in my ignorance) would have recommended, but in ways that addressed the risks they were trying to manage. It was already my practice to try to understand what worked at a client site, and why it worked, rather than to evaluate what they were doing against my prejudged ideas. But the diversity in this series of clients was overwhelming. It caused me to abandon many of my favorite ideas about development and testing—not as bad ideas, but as good ones only under suitable circumstances.

uTest: You’re shaping the minds of future testers… so how do you “future-proof” your teachings?  And what major changes do you think will impact software testing by the end of this decade?

Kaner: What I DON’T do is try to slow the field down.

I don’t pretend that there are One True Definitions for any testing terms, because different groups of testers see the craft differently and they use their language accordingly. If two testers have different ideas about the purpose and goals of testing, they are likely to have different meanings, or at least different nuances, in their definitions of “test case.” I don’t go to standards boards to try to legislate my favorite meaning as the official one. This effort to lock down a field that is still in motion, still finding itself, will primarily benefit people selling certification courses. In terms of helping their students prepare for the future, I think that this gives an illusion of certainty and uniformity to people who should be training to embrace questionability and diversity.

Read more…

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview.

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

Read more…

After almost a year of being told that we “have to interview Kaner” by previous Testing the Limits guests and readers, we exercised our listening skills and sought him out. With us this month to share his unique brand of wit and wisdom is Dr. Cem Kaner – author, lawyer, speaker, professor and one of the most respected minds in the testing world.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

Read more…

Our guest blogger this month is Travis Howell. A member of the uTest community, Travis has over 15 years of project management/analyst experience, including one year at NASA where he served as Deputy Manager of Human Spaceflight for the International Space Station. A father of three (or seven if you include dogs), Travis is an employee by day, a father by night and a uTester by early morning. Sleep would be nice, he said, but he’s adjusted to living on a couple hours.

In this post, Travis (a Gold Tester) outlines the mindset needed for success in the uTest community – including advice on logging bugs, preparation, knowing your limitations and more. Enjoy!

*************

How fitting to the time of year. It’s summer, it’s hot – for some of us it’s very hot – and a pool looks so inviting. Tell me this: Would you jump into a pool if you didn’t first know how to swim?  My guess is that your answer is NO. This world presents a lot of open invitations – invitations that appeal to our wants and desires and make it easy for us to just jump right in. I have to admit, sometimes it’s nice to be first. After all, we live in a generation that rewards the person that comes in first. 

Testing Is NOT About Being First
It’s not about the first person to find bugs or the first person to complete their work. To test is: to be informed, to understand, to question, to investigate, to identify, and to communicate. In the end, it’s about providing the customer with the ability to deliver a superior product. And once they bring that product to market, who is ultimately the winner?  You, the consumer.

What’s This — A New Test Cycle In My Inbox?
A new test cycle is an open invitation for everyone to jump in the pool. I know how inviting that test cycle invitation is, and I can guess what’s going through your mind. “What do I have here, a new test cycle. Looks like I’m the first participant, the world is my oyster. I’ll get in, look around, get a jump on everyone, score a few bugs, rack up some dollars, and out I go.”  Easy money right?  Maybe, maybe not.

Let me take you back to my first question, would you jump into a pool if you didn’t first know how to swim? Now, let me re-phrase it as it relates to testing. Would you enter a new test cycle without first knowing what the expectation of the customer is and what is expected of you to test? The correct answer is NO, but the lure of ‘easy money’ is there and provides the temptation to push a few in the pool. As an example, I have seen a large number of bugs logged to a test cycle where it is clearly identified within the objectives that testing is not to begin until a future date. We’re all guilty of moving too quickly sometimes. But don’t forget one thing, what you don’t know could come back to bite you.

Read more…

What’s the square root of Pi? Who led the NBA in rebounding in 1988? Are there really rattlesnakes in Vermont? Is Silly Putty edible?*

Here’s another question for you: Where do you go for answers online? If you’re not sure, we suggest that you ask Google, because that’s where almost everyone submits their online inquiries these days.

To make it a bit more scientific, we decided to make this question the topic of our weekly “What Do uThink?” poll. Here are the results from the uTest Forums:

• Google (search) – 91%
• Wikipedia – 4%
• Bing (search) – 2%
• Other – 2%

Clearly, Google pretty much owns the online answers space, which came as no surprise. However, when we posed the exact same question to our Facebook users the results were slightly different. Google still came in first place, but with “only” 53% of the total vote. Rounding out the totals were Wikipedia (18%); YahooAnswers (12%); Bing (6%) and WikiAnswers (6%).

So what makes Google the go-to place for answers? uTest Forums member “scornik” explains:

Read more…

According to tech analyst firm IDC, U.S. companies paid a record $14.2 billion for paid keyword-driven contextual ads in 2009, with Google dominating 55% of that revenue, Yahoo 9% and Microsoft 6%.

More dollars = More fraudsters. Period.

The company Click Forensics just released a report on the overall click fraud rates for the paid search industry. According to SearchEngineLand, the report said click fraud was up from 17.4% last quarter to 18.6% in Q2 of 2010. Traffic across 300+ ad networks is reflected in the data.

In addition, it was found that the countries outside North America with the greatest volume of click fraud were Singapore, Pakistan, Japan, Ukraine and China respectively.

Recent research by marketing intelligence company Visual IQ came out with similar numbers earlier this month. The company estimates marketers lose an average of 16.7 percent of their pay-per-click budgets to fraud.

So why is click fraud slowly trending higher and higher? The CEO of Click Forensics, Paul Pellman, stipulates that “the main reasons appear to be the continued sophistication of botnets and malware prevalent in the fast-growing search marketing space.”

According to Inc. Magazine, click scams use the following techniques:

• Manual clicking. Workers might be paid to click to run up totals.
• Software clicks. Automated clicks.
• Bot networks. Using malware to harness unsuspecting users’ computers, criminals can create large networks of computers employing programs that imitate clicks.

Despite detection innovations, click fraud rates show no signs of slowing. Attacks are becoming more sophisticated. Criminals are making more money. So what can we do? Any advice out there on how to mitigate it?



Creating new platforms like Android and iPhone is incredibly difficult, but it’s rare to hear stories about the challenges of building them unless you’re an insider.  There are probably dozens of good tales about developing these platforms that will take years to trickle out from behind closed doors.

So to hear stories like these, we must look back in time at the great development projects of the past.  Today the Computer History Museum announced that Apple has donated the source code for the original MacPaint application so that it can be downloaded by anyone.  MacPaint was a drawing application included with the first Macintosh that by today’s standards seems very simple, but in 1984 was completely revolutionary.  Many of the graphic design tools we take for granted, like the paint bucket and lasso select, were invented in MacPaint.

For developers and testers alike, there’s a lot to learn from the development of MacPaint.  Here are a few good stories:

Read more…

Not to beat a dead horse or anything, but I wanted to briefly revisit Apple’s  “Antenna-Gate” fiasco to drive home a very important lesson for companies of all shapes and sizes: Rely too heavily on “lab-testing” and you are virtually guaranteed to get burned.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

Read more…

Everyone wants to know what Apple’s going to say at their big press conference in a couple of hours. Will the iPhone 4 bugs prompt them to issue a recall? Will they send users a plastic case that supposedly solves the reception problems? Will they try to fix the defects with a software patch?  Will they say they’re sorry and that this will never happen again? Will they tell NY Senator Chuck Schumer to suck an egg?

We’ll have to wait and find out. But here’s one thing they’re NOT likely to say (but they should): “We should have listened to our testers!”

[Update: See this TechCrunch story for a round up of the press conference]

One of the biggest pet peeves among testers and engineers (or anyone in involved in quality assurance of technology) is not being taken seriously when a serious issue is uncovered. For most companies, it’s generally a cross-site scripting vulnerability, an SQL injection or a browser compatibility flaw in the UI.  For the iPhone 4, it was an antenna issue. As it turns out, many top executives – including Steve Jobs himself – were repeatedly warned about about the “death grip” well in advance of the product’s release. These warnings from respected internal resources were either ignored or not taken seriously. They should have listened to their testers.

But what should testers do when they find themselves in this situation? According to Bill Ricardi, they should report the bug and move on. A member of the uTest community, Bill gave his advice on this matter as part of our Guest Blogger series, writing:

You won’t always see eye to eye with the client. What you consider a critical bug, they might see as a non-issue (or worse, a ‘feature’). What you call a major security flaw, they might consider such a remote possibility that it doesn’t even deserve a mention.

You might ask how you bridge such a gap between your level of testing and the client’s level of acceptance and understanding of product integrity and the testing process in general. The answer is simple:

You don’t.

Read more…

The mobile wars are heating up! Microsoft is aggressively luring app developers for its Windows Phone 7 OS, while Android quietly gains market share. Blackberry expects big things out of OS 6, while The Big Apple deals with antenna issues, the yellow screen of death and the (remote) possibility of a recall. Interesting times indeed.

As part of our newly-launched “What Do uThink?” series (more on this shortly), we decided to ask our community which mobile OS they considered to be the best. Here are the results:

1. Android – 38%
2. RIM Blackberry – 28%
3. Apple – 16%
4. Symbian – 12%
5. Windows Mobile – 6%

“What do uThink?” is a weekly poll, where we’ll be asking the uTest community their preferences and feedback on various apps, operating systems and other technologies. To encourage voting, we’ll be awarding monthly and quarterly prizes to randomly selected participants. This quarter, for instance, we’re giving away an iPod Touch. The weekly polls open every Tuesday afternoon and voting takes place in the uTest Forums available to registered testers) as well as on our Facebook page. Got it?

Good. Now back to the mobile OS results…

Read more…