The results? More than half of respondents (51%) remain skeptical, choosing “No. Seems like a serious problem;” however, nearly 40% selected “Yes. I would still consider buying one,” with 10% still “Unsure.”

Our off-the-cuff survey jives with UK-based research group Opinium’s poll which questioned consumers about how they felt about the iPhone 4. A notable 57% said that they now have no intention of buying one.

While competitors have taken the opportunity to take serious jabs at the new iPhone — including the latest and greatest Droid X putting out full page ad’s with headlines reading, “No Jacket Required” (see Computerworld article) — it still looks like iPhone enthusiasts are holding strong.

According to a survey by The Yankee Group, most people who have an iPhone are very happy with the service they get from AT&T. The Yankee Group’s survey elicited positive responses from 73% of iPhone users.

So, while the iPhone 4’s antenna issues may have shaken up some consumers, Apple fanboys (and girls ) are not going to give up that easily. And, lastly, I have to ask: What Do uThink?

In part III of our of Testing the Limits interview with Cem Kaner, we discuss why “best practices” is merely a marketing tool; Silicon Valley of yesteryear; his upcoming CAST lecture on investment model and exploratory test automation; the blogs he reads and much more. To recap, here’s part I and part II.

uTest: What’s surprised you the most about the testing industry since you’ve been in the game?

Kaner: Context-dependence was a big surprise to me. It took me about ten years before I reluctantly accepted the idea that my favorite test techniques, attitudes, and life-cycle models were appropriate in situations similar to the ones where I developed my preferences, but not so appropriate in situations that were quite different.

It took me a long time to learn that developing software under a well-specified contract is a common and respectable activity that requires different tradeoffs from mass-market software sold to people who had no say in its design and buy only after it is ready to deliver and only if they like it. It took me a long time to sort out differences between scientific programming (where I started) and consumer software development. It took me even longer to see that testers working in an independent test lab operate under fundamentally different constraints from testers working inside the development company and provided different types of information. And it took well over a decade before I accepted the idea that two different development managers, running essentially the same project, could legitimately want different information from their testers and that it could make sense and be ethical for the two different test groups to structure their work differently, finding or being blind to or ignoring different classes of bugs, in order to satisfy the information needs of their key stakeholders.

The last major event in this chain happened 14 years and two books after I came to Silicon Valley. A client paid me to tour a lot of companies in California, Oregon and Washington. I gave a talk at each place, but I also talked with them about their business models, their testing challenges and methods.

Most of these were good companies with competent testing staff but they did things very differently from each other, often very differently from what I (in my ignorance) would have recommended, but in ways that addressed the risks they were trying to manage. It was already my practice to try to understand what worked at a client site, and why it worked, rather than to evaluate what they were doing against my prejudged ideas. But the diversity in this series of clients was overwhelming. It caused me to abandon many of my favorite ideas about development and testing—not as bad ideas, but as good ones only under suitable circumstances.

uTest: You’re shaping the minds of future testers… so how do you “future-proof” your teachings?  And what major changes do you think will impact software testing by the end of this decade?

Kaner: What I DON’T do is try to slow the field down.

I don’t pretend that there are One True Definitions for any testing terms, because different groups of testers see the craft differently and they use their language accordingly. If two testers have different ideas about the purpose and goals of testing, they are likely to have different meanings, or at least different nuances, in their definitions of “test case.” I don’t go to standards boards to try to legislate my favorite meaning as the official one. This effort to lock down a field that is still in motion, still finding itself, will primarily benefit people selling certification courses. In terms of helping their students prepare for the future, I think that this gives an illusion of certainty and uniformity to people who should be training to embrace questionability and diversity.

Similarly, I think the idea of “best practice” is a security blanket for babies or a marketing tool. In my experience, context matters a lot. A “practice” (a something that I use or do) can work very well under some circumstances and poorly under others. Teaching about “best practices” is another way to sabotage the critical thinking  of the students. As the context changes, people with this training have a poor foundation for adapting, or at least for adapting in any other way that paying a consultant to tell them what to think and do next.

As I see it, we test in order to find out useful information. We use technology (usually) to ask machines questions about how they work and humans questions about how the machines should work. We ask the questions in order to reveal information about the quality of what we are studying, and we usually do that because someone is paying us to get that information.

So, the “future-proof” skills and knowledge are the critical thinking skills, the questioning skills, and the understanding of full-lifecycle scientific method (including the qualitative research needed to develop ideas that lead to potentially-valuable hypotheses and the formal methods useful for focusing and assessing (typically refuting) those hypotheses.) The rest is going to change.

I can’t future-proof my ideas. That illusion is for the standardizers. Many of my ideas have changed enormously over the years. Take a look: http://www.kaner.com/pdfs/TheOngoingRevolution.pdf. Until I stop learning, my ideas will change even more.

uTest: Your latest presentation revolves around investment modeling and exploratory test automation. Give our readers a sneak preview of what to expect (also, where and when to expect it).

Kaner: This is a pair of talks at CAST (the conference of the Association for Software Testing). CAST is at Grand Rapids, Michigan from August 2-4. See http://conferences.associationforsoftwaretesting.org/CAST2010

The first talk is with Doug Hoffman on Exploratory Test Automation (ETA). Mainly, we will review a variety of examples of projects that used automated tests to learn new things about products rather than trying to confirm that old knowledge was still correct. Many of these involve automated creation of enormous sets of tests, scanning the program in ways that a human can’t, looking for potentially interesting results that would be worth a human’s time to investigate further. Several other people are having insights that a lot of (partially) automated testing that they do is not regression testing and is useful for learning new things. They’re presenting their ideas as ETA too, at CAST and at other conferences. We’re just adding our list to the mix and suggesting a framework for sorting out the differences and similarities among this large family of ideas.

My experience with teaching people about ETA in the past is that the examples require technical sophistication to understand (e.g., what does printer firmware really do and what are we trying to learn when we test it?). An hour-long talk is too short for some people. The good news is that at CAST (like at the LAWSTs), we are not limited by the time allotted for the talk. CAST has a breakout room. If people want to argue with us or ask detailed questions after the talk, we aren’t allowed to say, “I’m sorry, we only have 1 minute left to answer your question.” At CAST, we move to the breakout room and stay with the questioners until our discussion is done. Probably this hour-long presentation will take 3 to 4 hours (with questioning) before it is actually done. And hopefully, the people who stay with us to question and argue will walk away understanding what we’re trying to teach.

The second talk is my keynote, at the end of the conference. Building on what Doug and I presented, I present a case study, explaining how I use automated exploratory tests to assess software that implements investment models. Many people find discussions of investing easier to understand and more interesting. See http://adam.goucher.ca/?p=1492 for a reaction to a draft of the talk.

uTest: How do you effectively bridge the gap between academic theory and commercial practice?

Kaner: There are thousands of gaps, in all directions. We don’t ever bridge “the” gap. We bridge specific gaps, one at a time.

The first challenge is to think across paradigms, across ways of looking at the world. If I want to introduce my ideas on design to psychologists, I need to learn to think enough like the psychologists who I am trying to influence to make my presentation influential. I have to understand what questions they are trying to answer or what problems they are trying to solve, and then tie my ideas to their interests or needs. That takes work on my part.

Similarly, if I am trying to learn lessons from another field, I have to learn about that field. A new technique, a new idea, a new experimental method or a new insight (and accompanying wisdom) – these are rarely packaged in a self-contained way that lets me grab one and use it. What is this technique used for? Why? What does it replace? What makes it better? What are its limits?  To understand these things, I need to be able to understand the literature (books, articles, conference talks, demonstrations) that present the new knowledge. And all the time, I am asking, how does that apply to what I might be interested in? Again, that takes work on my part, a lot of it.

I host an annual meeting called the Workshop on Teaching Software Testing. This is one of the LAWST-style workshops (http://www.kaner.com/lawst.htm). We recently had our 9^th one (http://www.wtst.org/). The LAWSTs generally allow up to 20 people in the meeting. We schedule presentations, but we work through a presentation until we’re done with it. Thus, a 1-hour presentation (according to the schedule) might trigger one minute’s worth of discussion or a full day of discussion (we’ve gone to both extremes in the LAWSTs). The questions are often detailed. Sometimes there are debates and one participant will create slides to make a counter-presentation. In a meeting like this, 10 to 20 people work together to build bridges to each other’s ideas. At WTST, we bring together teachers—some academic, some commercial—and we share ideas on how to teach topics in testing.  This is my most effective vehicle for building bridges.

uTest: Name some other testing pundits, zealots, teachers and thought-leaders whom you respect. Who should we interview next month to share their wit and wisdom with the testing world?

Kaner: I’m surprised you haven’t talked with Elisabeth Hendrickson yet. And talk to Robert Austin.

Rapid fire Kaner Q&A:

• Last book read: Harry Dormash, Fire Your Stock Analyst. (This one was so good, I’m adopting it as a textbook in my investment modeling in software engineering course.)
• Last movie watched: Karate Kid
• Mobile device of choice: Amtrak
• Mac or PC: OK
• Techcrunch or Wired: Digg.com, Huffington Post, Automated Trader magazine, Bulletin of the ACM Special Interest Group on Computer Science Education, Law.com, Psychological Science, Bloomberg News, CNBC (especially Cramer) and Rachel Maddow. Ask me next year and the list will be different.
• Whiteboard or chalkboard: projector
• Best video game of all time: No idea. But my personal favorite was Wizardry.

Editor’s note: We hope you enjoyed our latest Testing the Limits interview. If you have a suggestion fir our next guest (or have a topic you want covered) just let us know.

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

• http://kaner.com/pdfs/FITcolloquiumBuildingFreeCoursewareCommunity.pdf

I describe the challenges in a little more graphic detail here:

• http://kaner.com/pdfs/nsfKanerFiedlerBarber.pdf

And I walk through our instructional model in a lot more detail here:

• http://kaner.com/pdfs/DevelopingInstructor-CoachedActivitiesNSF2008.pdf

At university and in practitioner training, my preferred teaching style is to leave the lectures to the videos, to use quizzes to drive students to watch the videos (and read the assigned readings) carefully,  so that they get the basic knowledge that I’m trying to convey, and then to use classroom time (in a live class or most of the course time in an online class) to do tasks based on the materials covered in the videos and readings. The students can apply and argue with the videos and readings in these tasks. I believe we foster a deeper level of learning this way.

As to “curriculum,” I think there are many different, valid, instructional objectives.

In an undergraduate university curriculum, the goal is not to train people for a specific type of future job. It’s to prepare people for the next 10 years of their career, with the expectation that they’ll want to switch jobs, and probably careers, a few times in that period. Undergraduate education has to be about teaching people learning skills as much as technical skills, and enough fundamental education that they are not blocked when they approach materials outside of their core domain. This drives us to a hard-to-optimize conflict between “general education” and technically-focused education in the field of the student’s choice. The broad curricular tradeoffs are too complex for me to understand how to resolve.

In terms of software testing, most universities offer little education directly focused on testing, and much of what is taught at most schools (that teach testing) is thoughtful code verification blended with applied mathematics, rather than skeptical assessment of product quality. Much of this contributes valuably to general education and to programming skill. But not so much to the craft of testing.

Mainly, our field relies on on-the-job experience and commercial training. The training courses rarely get much beyond the basics because the average student hasn’t gotten much past the basics. In addition, there is a critical difference between university education (done well) and commercial training: projects and homework. Students in my Testing 2 class (programmer testing) write take-home final exams of up to 5000 lines of code. Other courses at Florida Tech demand more than this. Most projects are much smaller, but each of them provides experience, not just vocabulary. Active engagement with course material is what fosters deeper learning, and we don’t have time for that in commercial training.

As a result, instead of starting from a well-built-up foundation, testing practitioners learn most of what they learn about testing on the job and in relatively simplistic training courses. They reinvent the field, each for herself or himself, learning the same lessons that we have learned 10, 20 or 30 years before. This is not a path to significant progress in the field.

This low level of common knowledge in the field is convenient for certification courses. They can sell a few days of training (at a high price) that is enough for students to pass a relatively trivial exam. Think of the absurdity of this. The current certifications are essentially advertisements that you (the certificate holder) know as much as someone with no experience and three days of training. Would you hire someone who was proud of that low a level of knowledge?

I keep meeting people who tell me that they send their staff to these kinds of courses because the course teaches everyone a common vocabulary, and our field needs a common vocabulary. I don’t personally think that a common vocabulary has much value. What is more valuable is learning to ask someone else what they mean when they say something.

Thirty years ago, a “big” program was 10,000 lines of code. Today, even cell phones have millions of lines of code. This enormous change in productivity comes from revolutions in the practice of programming and software design. But we aren’t seeing revolutions in testing, not like the change into structured programming or out of structured programming and into object-oriented. No one would think of teaching the programming methods of the 1970’s or 1980’s as state of the art today. But the current certifications in testing (and many of the “applied” university and college courses) seem to me to be based on “best practices” and attitudes that aren’t much different from what I started writing Testing Computer Software to rebel against back in 1983.

We need to make progress on the education problem, or we will dig deeper and deeper into the same old ruts.

uTest: How well are companies adhering to The Software Consumer Bill of Rights (which you wrote in 2007)? Have conditions improved for the average software consumer?

Kaner: The Principles of the Law of Software Contracts have finally been adopted by the American Law Institute. That will probably reflect itself in judicial opinions over the next few years.

In terms of legislation, the United States is in a dysfunctionally polarized period. Very little legislation of any kind is being passed. We might not see useful statutes (laws passed by legislatures) on computer law for several more years. Until then, companies will start acting more responsibly after some successful lawsuits or enforcement actions.

uTest: James Whittaker has made the jump from academia to corporate America.  Will we see you take your considerable talents “across the aisle” at some point (assuming that you aren’t doing so already)?

Kaner: For me, this draws a false dichotomy.

As I said in my last note, programming has been evolving in powerful ways. Testing: not so much. Developing better education/training for testers is one part of the way out of this hole. That’s what I’m currently working on. I’m working at a university because universities house the cutting edge for educational practice.

On an ongoing basis, I ask myself whether I am working on the projects that have the potential to make the greatest positive impact and whether I still think I can realistically make the kind of contribution that is needed. For now, I’m in the right place doing what I think are the right things for me to be doing. Later, that will certainly change. When? How? I don’t know.

Editor’s note: Check back tomorrow for part III of the interview.

In part I of our interview, we ask Cem to share his thoughts on the multi-disciplinary nature of software testing. His response includes thoughts on experimental psychology; law; testing metrics; arrogance in the field of testing and more. Check back for part II and part III in the next two days.

*******

uTest: In your online bio, you say the theme of your career has been to “enhance the overall safety and satisfaction of software”. To do so, you’ve studied (and worked in) areas like psychology, law, programming, testing, technical writing and sales. Explain how working in other disciplines has helped you better understand software. And on that note, what can testers learn from lawyers, writers and salespeople?

Kaner: Let me start this by saying that almost all of the best people I know in testing have significant experience in other fields. It’s common for people to move from testing to programming or writing or marketing and then back, bringing what they’ve learned with them, to test with a richer perspective and with a much more productive vision of where testing can fit within development/marketing/support cycles.

We write software to solve problems that people need solved, to do things that people want done, or to entertain people. To understand a piece of software, I need to understand why it was written, who it was written for, why they should want to use it, and what alternatives might serve them as well or better.

So I don’t see the “other disciplines” as “other.” They all contribute to this understanding in important ways.

You asked specifically about how multiple approaches fit into my own work and understanding. That’s a more personal story…

I came into software development with a doctorate in experimental psychology. I did a lot of programming (and writing about code) as a student and was deeply interested in what made products learnable and usable. A specific interest was what made a person more or less likely to make a user error. I wrote several data entry programs for large sets of scientific data. It was remarkable how much my design choices influenced the types of errors people made. What people call “user errors” are at least as much a feature of the program’s design as they are of the people who make the errors.

When people make a repeated error using my code, instead of asking why these people are idiots, I learned to ask what’s wrong with my software that causes the nice people to look like idiots.

Let me generalize this—the quality of a program extends far beyond its functionality. There is a huge gap between “works right” and “works well.” From my viewpoint, design choices that needlessly reduce the value of the program are defects, every bit as much as coding errors.

My dissertation focused on measurement of perceptual experiences (a field called psychophysics). We asked questions like “How loud is this sound?” Hearing is an experience. The physics of the sound can help us predict some aspects of that experience, but loudness is about what it sounds like to you, not the physics.

It’s pretty easy to think about how to measure the length of a table. Use a ruler. But what ruler do we use for sweetness? In order to think clearly about this, we need to clarify for ourselves the construct (the idea of) “sweetness.” What does someone mean when they say that X is sweeter (to them) than Y? Then we have to imagine a measuring instrument (like a ruler) that would give us low numbers for not-sweet, high numbers for very-sweet, and reasonable distances between different amounts of sweetness.

“How do we know that this (ruler) measures that (construct)?” is one of the fundamental questions of measurement. It even has a name, “construct validity.” In every field that I’ve studied other than computing (I’ve looked at measurement theory in a lot of fields), when people develop measurements or “metrics” and when they teach metrics, they talk seriously about construct validity. In contrast, construct validity is almost never mentioned in computing. (Do a search for “construct validity” in ACM’s Guide to the Digital Literature! Filter out the papers in business-oriented or medicine-oriented journals. What do you have left?). We have textbooks with dozens (I taught from one text with hundreds) of “metrics” and no practically-applicable system for assessing what these “metrics” measure.

Capers Jones sometimes talks disparagingly about the (claimed) fact that 95% of American software companies have no metrics program. On the surface, this sounds terrible. But what I saw as a consultant was that most software companies have tried a measurement program, or have executives with lots of experience with metrics programs in other companies. The problem is that their experiences were bad. The measurement programs failed. Robert Austin wrote a terrific book, Measuring & Managing Performance in Organizations. When you start measuring something that people do, people will change their behavior to make their scores better. People will change what they do to get better scores on the measurements—that’s what they’re supposed to do. But they don’t necessarily change in ways that improve what you want to improve. Often, the changes make things worse instead of better (a problem commonly called “measurement dysfunction.”) This problem happens more often, and worse, if you use weak, unvalidated metrics. I keep meeting software consultants, especially software process consultants, who say that it’s better to use bad measurements than no measurement at all. I think that’s’ a prescription for disaster, and that it’s no wonder that so many software executives refuse to harm their businesses in this way.

By the way, what if one person repeatedly says X is sweeter than Y and another person says Y is sweeter than X? Are these people wrong about their experiences? Probably not. So is X sweeter than Y? Yes. Is Y sweeter than X? Yes. Have fun putting THAT on your measuring stick.

Jerry Weinberg defines quality as “value to some person.” It’s subjective, like sweetness. Different for different people. Quality is not quantity. But that doesn’t make it any less real.

In terms of the contributions of multiple fields to software, it’s no surprise that Weinberg’s doctoral work was in psychology or that he is so popular a thinker in the testing community. When we evaluate the impact of a new piece of technology on humans, we are doing applied social science research. To say that testing is part of a fundamentally different from the social science is to miss the point of much of testing AND to turn a blind eye to tools developed over centuries that can help us do our work.

Let me switch disciplines–I started working in Silicon Valley in 1983. I watched some excellent companies fail and weak competitors succeed. A striking commonality was that companies would release products with missing features or features that didn’t work, often lying about what they offered. They would gain market share so quickly that they could drive out more honest companies. Most bad companies eventually failed but they cost the marketplace its best suppliers time and again. As a tester, and sometimes as a testing consultant, I was encouraging topnotch staff to hammer products very hard, to expose bugs and argue effectively to get them fixed – to do the right thing by their customers. But the effect of this was that companies were coming to market late with a better product but a worse competitive position. Doing the “right” thing was being punished in the market.

This REALLY troubled me.

I wrote Testing Computer Software over a period of 4 years (1983 to 1986). In many ways, it was an excuse for me to ask people probing questions and think with them about answers.

In terms of markets, I learned that in mature industries, North American society places expectations on companies. We expect companies to advertise honestly, to make products that do what they are supposed to do (maybe not perfectly, but pretty well), to fix their defects for free or give customers their money back, and to allow their products to be analyzed by journalists  who help customers distinguish good from bad products. As a society, we write those expectations into law. When we enforce the laws, we weed out the worst companies and create a safer market for honest businesses and their customers. These rules are less rigid for companies that sell services (e.g. custom programming), but in mature industries, service expectations (and rights) are pretty demanding too.

I came to believe that the core quality problems in software were (and still are) to a very large degree the result of loose enforcement of laws governing fraud, breach of contract, breach of warranty, and service negligence. I learn by doing. To learn more, I became an Investigator (part time, volunteer) in the County of Santa Clara’s Consumer Affairs Department. (Most of Silicon Valley is in Santa Clara County.) After a couple of years, I decided to go to law school and to focus on the law of software quality.

From 1984 to 1987, I worked in a telephone company in a user interface group. A telephone is a user interface to a richly featured, complex network of computers. Our particular system was a really early provider of integrated voice and data. You plugged your computer into the phone to be on the network, rather than into a modem or an Ethernet cable. We also delivered the first phone system that was actually for sale (rather than demonstrated in research laboratories) that had an LCD display.  The menus on our 2 line x 40 character display gave access to 108 voice features and 110 data features.

In our group, we could all program, we all knew a lot about usability. We evolved our system’s user interface over years, in an evolutionary manner (add one or a few features and get them working before adding the next). We moved fluidly between customer focus and code focus, and we modified each other’s code all the time, trying out new ideas for design, which we (and the rest of our company and sometimes our external customers) assessed at the systems level in weekly deliveries of new system builds. It was a great experience.

I don’t want to downgrade the technological challenges of this work. We had to pack a ton of capability into small spaces and have it run very quickly and extremely reliability. The programming side of my work was very difficult. But the programming would have been meaningless without a focus on who the programming was for and it would have been ineffective without an ongoing stream of information about what was working (design and code) and what was not.

At another company (Power Up Software) I was a Software Development Manager. When I started, my task was to design products for small businesses, help plan the marketing of these products, and manage programmers (typically remotely located programmers) who were to write these products. I wanted to turn my products into best sellers but I didn’t know how, so I took a job at Egghead Software, then a successful software retailer, and sold my best competitors’ products to Egghead’s customers. These people didn’t just buy once and disappear. They came back to the store every week or two, buying new things or complaining about old ones. You learn a lot about what makes a product great, and not-so-great, by selling it to people and then dealing with them, face-to-face, over and over. What is quality? Quality is what made my customers happy. What’s a defect? A defect is what made my customers shout at me.

Some people would tell me that quality is conformance to a specification and a defect is a nonconformity. But really, what’s a specification? A specification is a piece of paper, often written by someone who knows nothing about quality.

About those specifications. I did a lot of school learning about software process. I came to Silicon Valley expecting to see lots of well written specifications. What I actually saw were a lot of scribbles on napkins (from the local pub or restaurant), scribbles on flipcharts, and scribbles on whiteboards. They were informal and they were updated as we came to understand our customers, our competitors, and our products better. Many of the most tedious-to-use, most poorly-designed products that I worked on or consulted about were coded to meet a detailed specification. Famous consultants would come visit our companies or teach courses that I sent staff to, and they would tell us that The Right Way to build software was to write everything down at the start and then to build to that specification. But the start is when you know the least about whatever you are creating. As you create it, you learn. Why make your decisions when you don’t know as much as you’ll know tomorrow? Why set up change control processes to make it expensive to fix all the mistakes you made when you were more ignorant than you are today?

Of course, there are reasons to set up change control processes. And there are reasons to develop a clear idea of the product’s scope early on (try designing an appropriate architecture without this).

But as the Extreme Programming advocates came to say much more eloquently than I knew how, knowing the big picture is one thing, but there is rarely a good reason to lock down details until you need to code them. And you cannot call something a “quality” process if it has you refusing to improve a product because it makes the cost of testing too high or the paperwork too expensive.

While I was dev manager (and later, director) at Power Up, I went to law school (you don’t learn much humility there, but it’s good for other things), went to University of California Extension for several courses on technical publications management (to help me work with a tech pubs group that adopted me as their manager), and wrote the second edition of Testing Computer Software.

Hung Nguyen joined me in writing TCS 2.0, but on the condition that I take courses in traditional quality control. He had a B.Sc. in Quality and wouldn’t write with me until I got ASQ-certified in quality engineering. His point was that I had become an expert in informal processes that worked on (relatively) small projects, but in his view, I needed to understand what effective control processes were and why they were important on larger projects and formally-negotiated projects (outsourced development, especially when your customer is the government). That blending helped us make TCS 2 a much better book than the original.

The diversity of my situations taught me powerful lessons about how little I know, how much the other people on the project know that I don’t know, and how complex are the tradeoffs faced by the average project manager. When I work as a tester, I understand that I am playing a supporting role. It is an important role. It calls for skilled and thoughtful work. But as James Bach puts it, I am “the headlights of the project”, not the driver, not the brakes, not the engine, and not the door to the customer. I can change roles. Maybe I can work two or three roles at the same time. But IN MY ROLE AS A TESTER, I don’t understand as much about the rest of the project as many other testers (and test consultants/trainers) seem to think they understand. There is a lot of arrogance in our field. We need to learn more humility.

I graduated law school in 1994, passed the bar exam and was sworn in as an attorney. I went back to the Santa Clara County government, this time to work as a full-time volunteer in the District Attorney’s office. My ideas about fraudulent and unfair trade practices had deepened into a passion. As a DA, I prosecuted about 130 cases, including 5 trials. None of this was white collar crime (e.g., economic crime by corporations) but I learned how prosecutions were run, how crimes were investigated, how cases were organized, negotiated, and occasionally, brought all the way to trial. I didn’t become an expert, but I gained more courtroom experience than most American lawyers gain in their professional lifetime. It was foundational knowledge.

Over the next six years, I spent a huge portion of my time as an advocate for a higher legal standard for software quality. I helped write the Uniform Electronic Transactions Act (UETA), which was adopted by almost all American state governments. The federal government renamed it the ESIGN act (electronic signatures) and applied it to all the states so that we had nationwide uniformity in some critical aspects of electronic commerce. I also tried to help write, and then led the opposition to, what became the Uniform Computer Information Transactions Act (UCITA), a bill that in essence, rewrote contract and copyright rules to benefit large software companies and large companies that embedded software in their traditional products (e.g., cars). UCITA ultimately failed despite the millions spent on it. And one of the most important legal organizations in the United States, the American Law Institute, elected me as a member. The ALI has up to (a maximum of) 3000 members. Most of them are judges or tenured law professors. Some are senior partners in large law firms, a few are famous consumer advocates. I think I was the least experienced lawyer ever elected to ALI and every time I go to one of their meetings, I know (or remember quickly) that every person in that room knows more about the law than I do. But I know a little more about computers-and-law than some of them, so sometimes, my background is useful.

ALI writes books that judges pay careful attention to. In the United States, and in all other countries whose systems evolved from British Common Law, laws passed by legislatures are more like detailed statements of principles. They do not cover every situation. They can’t. So judges have to figure out how a body of laws applies to each particular dispute before them. Across different states, judges might apply the same laws differently. ALI helps organize and make sense of the diversity of decisions. They publish the Restatements of the Law (e.g. the Restatement of Contracts, the Restatement of Torts, etc.) and Principles (e.g. Principles of the Law of Software Contracts). After UCITA failed (only two states adopted it, dozens of states rejected it), ALI let the dust settle for a couple of years and then started writing the Principles of the Law of Software Contracts. One of its most important rules is one that I advocated: a seller of software who knows about a defect of the software but does not disclose the defect to the customer will be held liable for damages caused to that customer by that defect. Note that this does not apply to free software (not sold). And if the seller discloses the defect, it becomes part of the product’s specification (it’s a feature). And if the seller doesn’t know about the defect there is no liability (once customers tell you about a defect, you have knowledge, so you cannot avoid knowledge for long by not testing). ALI adopted it unanimously last year. This is not law, but until the legislatures pass statutes, the Principles will be an important guide for judges. Even though I am a minor contributor to this work, I think the defect-disclosure requirement might be my career’s most important contribution to software quality.

The Association for Computing Machinery recently honored this aspect of my work with its “Making a Difference Award” which is “presented to an individual who is widely recognized for work related to the interaction of computers and society. The recipient is a leader in promoting awareness of ethical and social issues in computing.”

Editor’s note: Read part II of the interview.

In this post, Travis (a Gold Tester) outlines the mindset needed for success in the uTest community – including advice on logging bugs, preparation, knowing your limitations and more. Enjoy!

*************

How fitting to the time of year. It’s summer, it’s hot – for some of us it’s very hot – and a pool looks so inviting. Tell me this: Would you jump into a pool if you didn’t first know how to swim?  My guess is that your answer is NO. This world presents a lot of open invitations – invitations that appeal to our wants and desires and make it easy for us to just jump right in. I have to admit, sometimes it’s nice to be first. After all, we live in a generation that rewards the person that comes in first. 

Testing Is NOT About Being First
It’s not about the first person to find bugs or the first person to complete their work. To test is: to be informed, to understand, to question, to investigate, to identify, and to communicate. In the end, it’s about providing the customer with the ability to deliver a superior product. And once they bring that product to market, who is ultimately the winner?  You, the consumer.

What’s This — A New Test Cycle In My Inbox?
A new test cycle is an open invitation for everyone to jump in the pool. I know how inviting that test cycle invitation is, and I can guess what’s going through your mind. “What do I have here, a new test cycle. Looks like I’m the first participant, the world is my oyster. I’ll get in, look around, get a jump on everyone, score a few bugs, rack up some dollars, and out I go.”  Easy money right?  Maybe, maybe not.

Let me take you back to my first question, would you jump into a pool if you didn’t first know how to swim? Now, let me re-phrase it as it relates to testing. Would you enter a new test cycle without first knowing what the expectation of the customer is and what is expected of you to test? The correct answer is NO, but the lure of ‘easy money’ is there and provides the temptation to push a few in the pool. As an example, I have seen a large number of bugs logged to a test cycle where it is clearly identified within the objectives that testing is not to begin until a future date. We’re all guilty of moving too quickly sometimes. But don’t forget one thing, what you don’t know could come back to bite you.

Preparation Is Key – Know Your Release!
Read the objectives of the test cycles, understand what the requirements are. It hurts when you enter a bug on a browser that was not listed as a requirement. Documentation attached to the test cycle is not just there to look pretty.  These are valuable resources provided to eliminate questions and eliminate entering bad bugs. Take advantage of the open invitation to ask questions of the project manager. If anything within the objective is unclear, ask the question. A question is learning experience, and if you have a question I guarantee that there are others who are in need of the same answer. Embrace the discussion thread, it’s there for a purpose and serves that purpose nicely.

Know Your Limitations
Before you commit to a test cycle, understand what is expected of you. There are all kinds test cycles, some large, some small, some public, some private. If you happen upon a test cycle where there are a limited number of participants, before you commit be sure you are able to meet the expectations. Do you have the time, can you meet the deadline, do you have the experience, and do you have the right technology? If you second guess any of these questions, the right approach may be to let this one go. Also, ensure that you can handle the workload. Never bite off more than you can chew.  We are the devoted few and that’s what makes us special.

The Database Is Your Friend
Defects are logged to provide a comprehensive picture of issues that have already been identified.  We are a community of testers working together, sharing information. As such, we should understand what other testers have already uncovered and embrace the discussion threads as a viable resource to communicate with each other and assist each other in overcoming hurdles. Before you enter that defect, ensure that it hasn’t already been discovered by another. If, within the objectives it is stated that tester feedback should be logged, take advantage of the opportunity.

Speaking from experience, a tester is not just someone who tests. They are someone who has become an expert on the product.  If something doesn’t fit the product and there’s an alternative which may add value, voice yourself.  The design phase can sometimes run down a ‘one-way street’.  On a one-way street there is no option other than going in the direction you are forced to drive.  Being familiar with the product, we have the ability to see a two way street.  You know the saying there’s more than one way to skin a cat.  Sometimes it takes another set of eyes to see that their may be a better direction to take than the one that was chosen.

Wow, That’s A Lot Of Bugs!
Sorry, I was just watching that Wow! That’s a low price commercial and got a little carried away.  Sometimes the number of bugs found deters us from working on a test cycle.  Nothing is more disheartening than entering a test cycle and seeing that 95 bugs have already been entered.  And what do we say to ourselves, obviously there can’t possibly be any more bugs left to find.  Let me tell you, where there’s one bug, he’s usually got a friend and where there’s a 100 bugs well, that’s a party. The goal of a test cycle is not to reach a particular number of bugs (xx) and call it a day. The goal is to ensure that we have done all we can, that we’ve tested each area of the application enough to feel good about the end product.  If there’s been 100 bugs found, great, make it a challenge to find those next five

Once You’re In The Pool….
We are the lucky few. We have the ability to see the future, to touch it, test it, learn it. We are ahead of the curve. We have all been given a great opportunity to explore technology that has yet to come to market. Embrace the future!

***********

[Editor's note: Want to get published?  If YOU would like to participate in our Guest Blogger series, send your posts or ideas to marketing@utest.com.]

Here’s another question for you: Where do you go for answers online? If you’re not sure, we suggest that you ask Google, because that’s where almost everyone submits their online inquiries these days.

To make it a bit more scientific, we decided to make this question the topic of our weekly “What Do uThink?” poll. Here are the results from the uTest Forums:

• Google (search) – 91%
• Wikipedia – 4%
• Bing (search) – 2%
• Other – 2%

Clearly, Google pretty much owns the online answers space, which came as no surprise. However, when we posed the exact same question to our Facebook users the results were slightly different. Google still came in first place, but with “only” 53% of the total vote. Rounding out the totals were Wikipedia (18%); YahooAnswers (12%); Bing (6%) and WikiAnswers (6%).

So what makes Google the go-to place for answers? uTest Forums member “scornik” explains:

- 90% of the time Google shows the expected result in the first page, so users don’t have to waste time looking at other search result pages.

- The Google search page design is very simple looking, yet very effective to all kinds of users. At first visiting the Google web page, users instantly know what this website is all about and what it’s main functionality is.

- Google is still avoiding showing advertisements at their home page. That’s a very tough thing to do. When I first started using Google, I was very satisfied by not seeing ads or other unrelated things before I performed a query. So thumbs up to Google.

Leave it to Google to cancel their “Google Answers” feature and still be the #1 place to find answers online. Make that two thumbs up.

So where do you go for answers online?

*Oh, and in case you were wondering:

• The square root of Pi is 1.772453850905516027298167483314
• Michael Cage of the LA Clippers
• Apparently yes, there are rattlesnakes in Vermont, but only in Rutland County
• Silly putty is NOT edible (trust me)

If you have suggestion for a future “What Do uThink?” poll, send them along.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

1. An insanely expensive and elaborate lab is still a lab
Notice how the lab contained rooms where there is “absolutely no interference when it comes to wireless testing.” Does this sound like an environment that exists in the real world? Yes, it’s absolutely helpful for testing some aspects of such a solution. But at some point, you actually want the wireless interference (and all the other stuff that real users must deal with). No matter how much a company spends to replicate reality, it will never become reality. A lab is still a lab.

2. You ALWAYS need a fresh set of eyes (plus ears and hands)
Apparently, testing was completed by those who were extremely close to the actual product – those who knew the device inside and out, top to bottom, front to back. As a result, many usability issues (like the death grip, for one) were never factored into the testing equation. Only through ‘in-the-wild’ testing (i.e. real devices in the hands of real users) can a company expect to uncover these type of unique issues.

3. Non-standard use cases are vital to acing the user test
Apple’s testing team should be applauded for driving around in a van, testing the wireless connection. But why stop there? Why not think of 50 more unusual scenarios – aka “non-standard use cases”? Of course, relying entirely on lab testing to accomplish these tasks would be extremely expensive, even for Apple. But by using elements of crowdsourcing, they would be able to extend their testing coverage relatively cheap.

4.Global users? Global testing!
As the iPhone 4 makes its way into other markets, it will be interesting to see whether users encounter issues specific to their geo-location. Judging from the centralized conditions of the Apple lab, it should not surprise any readers of this blog if such complications appear. To sum it all up: your product needs to work where your users are. For 99.997% of companies and apps, this is NOT in a lab.

Are there other lessons to be learned from Apple’s testing methods? Have you seen other companies pay the price for relying on lab testing, while discounting “in-the-wild” testing? Drop us a comment and tell us all about it.

Here’s a quick video “unveiling” of the Apple’s impressive test lab:

We’ll have to wait and find out. But here’s one thing they’re NOT likely to say (but they should): “We should have listened to our testers!”

[Update: See this TechCrunch story for a round up of the press conference]

One of the biggest pet peeves among testers and engineers (or anyone in involved in quality assurance of technology) is not being taken seriously when a serious issue is uncovered. For most companies, it’s generally a cross-site scripting vulnerability, an SQL injection or a browser compatibility flaw in the UI.  For the iPhone 4, it was an antenna issue. As it turns out, many top executives – including Steve Jobs himself – were repeatedly warned about about the “death grip” well in advance of the product’s release. These warnings from respected internal resources were either ignored or not taken seriously. They should have listened to their testers.

But what should testers do when they find themselves in this situation? According to Bill Ricardi, they should report the bug and move on. A member of the uTest community, Bill gave his advice on this matter as part of our Guest Blogger series, writing:

You won’t always see eye to eye with the client. What you consider a critical bug, they might see as a non-issue (or worse, a ‘feature’). What you call a major security flaw, they might consider such a remote possibility that it doesn’t even deserve a mention.

You might ask how you bridge such a gap between your level of testing and the client’s level of acceptance and understanding of product integrity and the testing process in general. The answer is simple:

You don’t.

It isn’t your job to convert the client to your way of thinking. Yes, you can contest a bug that they reject out of hand if you were technically correct to report it. Sometimes they’ll accept it as valuable feedback, but most of the time they’ll just ignore any contested bugs. This is something that you have to live with.

You cannot impose your standards upon a client. No matter how passionate you are about the quality of your work and your understanding of their product, no matter how much you study the test parameters and the client’s requirements… only they know what they really want out of the test cycle. They WILL usually refuse bugs that they don’t consider ‘real’ bugs, because they’ve probably had meetings about what their programming philosophy should be, and they probably have a preconceived notion of what they have time and budget to fix.

So you’ll get hit with the dreaded duo of rejection reasons: ‘not a bug’ or ‘out of scope’.

What do you think? Should testers and engineers do more than raise their hands to highlight risks and defects? Asked differently, who is ultimately responsible for quality?

There are plenty of other testing and QA lessons to be learned here (first and foremost, the cost of discovering defects after a product is launched). Let’s see what Apple announces today, and then how the public and media respond to it.  Something tells me this won’t be the last time we blog about this troublesome tale, and what it can teach us about launching high-quality products.

Yesterday, the security company WebRoot came out with a study discovering similar results. After surveying 1,500+ social network users with geolocation-ready mobile devices, WebRoot found that more than half (55%) of respondents fear the loss of security and privacy, and 45% are very concerned about letting potential burglars know when they’re away from home (ah yes, the now shut down PleaseRobMe experiment comes to mind).

What’s most interesting to us is that 39% of those surveyed by Webroot said they use geolocation services, but take a look at the number of people that have fallen prey to social network cyber-criminals:

• Nearly a quarter of respondents (22.4 percent) were victims of a phishing attempt to steal their social network password.
• About one in six (16 percent) reported a malware infection in the past year that originated from a social networking site.
• One in nine reported at least one of their social network accounts had been compromised or hijacked.

Even in the face of these risks, many consumers admitted to engaging in risky behaviors:

• Nearly one third (31 percent) accepted a friend request from a stranger.
• A majority (76 percent) clicked a link sent or posted by a friend on a social network site.
• Twenty-nine percent have shared their geolocation with people other than their friends.
• One in nine used a location-based tool to meet a stranger (e.g. check out Scout – the new dating LB app)

While we all get excited when new features (like geo-lo) are added to Twitter, Facebook and other social networks (I know I do!), it’s worth taking a step back and thinking about the potential dangers of giving away so much personal information. So, why do you check-in? What do you see as the primary motivators for doing so? Are they worth the risks?

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

• Since 2005, no significant up-, or downward trend in the total number of vulnerabilities in the more than 29,000 products covered by Secunia Vulnerability Intelligence was observed.
• A group of ten vendors, including Microsoft, Apple, Oracle, IBM, Adobe, and Cisco, account on average for 38 percent of all vulnerabilities disclosed per year.
• In the two years from 2007 to 2009, the number of vulnerabilities affecting a typical end-user PC almost doubled from 220 to 420, and based on the data of the first six months of 2010, the number is expected to almost double again in 2010 to 760.
• During the first six months of 2010, 380 vulnerabilities or 89% of the figures for all of 2009 has already been reached.

ZDNet’s Adrian Kingsley-Hughes made note of the products that are causing these vendors to rise or fall in the standings. The culprits, he says, are as follows:

• Apple - (iTunes, Quicktime)
• Microsoft – (Windows, Internet Explorer)
• Sun Microsystems – (Java, now part of Oracle)
• Adobe - (Acrobat Reader, Flash)

So testers, who do you see as the Bug Marathon winner (i.e. loser) at the end of the year? Not that we’re keeping score or anything.