Testing the Limits with Cem Kaner – Part II

In part II of our interview with Dr. Cem Kaner, we discuss advice for current and prospective testers; the future of software testing in higher education; tester certifications; the Software Consumer Bill of Rights and more. Catch up by reading part I and be sure to check back tomorrow for part III of the interview.

uTest: Even in the less than perfect economic conditions that we’re currently facing, software testing is one profession that seems to be bucking the trend. Why do think this is the case, and what single lesson or piece of advice would you give to someone who is considering a career as a tester?

Kaner: Develop skills that will be genuinely valuable to your (current or prospective) employer, and once you get the job, use them. Being able to demonstrate that you know how to DO something that is difficult is worth much more, with a competent interviewer, than being able to talk about that something or provide its definitions.

uTest: James Bach, among other past Testing the Limits guests, has been a harsh critic of the way testing is taught in higher education. However, when we interviewed him last year, he mentioned you as a great exception. What are you doing so differently? What’s the biggest thing that can be done to improve the curriculum?

Kaner: I did a lot of 3-day commercial-course teaching from 1993 through 2000. I taught on my own and in several different groups, chatting with other instructors about what we did, what our clients and students expected of us, and what kinds of learning we expected to help students achieve.

I was a popular teacher. I made very good money. But I gradually came to the conclusion that this was an inefficient way to teach and an ineffective way to improve the state of the practice in our field.

I’m not as disappointed with formalized education as James is, but most testing education (especially commercial training) is failing. People don’t learn much, and not much of what they learn is useful, and even less helps them develop the cognitive skills needed to learn more, faster, on their own.

To learn how to teach more effectively, I decided to teach at a university and do research on how to blend academic and practitioner-oriented teaching methods. I presented some ideas to the National Science Foundation, got research funding, and have been trying to develop better ideas for teaching testers for the last 10 years.

You can see my instructional videos at http://www.testingeducation.org/BBST. (Note that they are free.)

The Association for Software Testing and Rebecca L. Fiedler (my wife, the education professor) and I have been developing interactive courses for practitioners (courses with the same videos but with live teachers who give guided tasks and feedbacks). These are currently free to members, but the cost structure has been making this look impossible for the future.

I paint the contrast between academic and practitioner education, and lay out some of the challenges of developing a free education-ware community, here:

I describe the challenges in a little more graphic detail here:

And I walk through our instructional model in a lot more detail here:

At university and in practitioner training, my preferred teaching style is to leave the lectures to the videos, to use quizzes to drive students to watch the videos (and read the assigned readings) carefully,  so that they get the basic knowledge that I’m trying to convey, and then to use classroom time (in a live class or most of the course time in an online class) to do tasks based on the materials covered in the videos and readings. The students can apply and argue with the videos and readings in these tasks. I believe we foster a deeper level of learning this way.

As to “curriculum,” I think there are many different, valid, instructional objectives.

In an undergraduate university curriculum, the goal is not to train people for a specific type of future job. It’s to prepare people for the next 10 years of their career, with the expectation that they’ll want to switch jobs, and probably careers, a few times in that period. Undergraduate education has to be about teaching people learning skills as much as technical skills, and enough fundamental education that they are not blocked when they approach materials outside of their core domain. This drives us to a hard-to-optimize conflict between “general education” and technically-focused education in the field of the student’s choice. The broad curricular tradeoffs are too complex for me to understand how to resolve.

In terms of software testing, most universities offer little education directly focused on testing, and much of what is taught at most schools (that teach testing) is thoughtful code verification blended with applied mathematics, rather than skeptical assessment of product quality. Much of this contributes valuably to general education and to programming skill. But not so much to the craft of testing.

Mainly, our field relies on on-the-job experience and commercial training. The training courses rarely get much beyond the basics because the average student hasn’t gotten much past the basics. In addition, there is a critical difference between university education (done well) and commercial training: projects and homework. Students in my Testing 2 class (programmer testing) write take-home final exams of up to 5000 lines of code. Other courses at Florida Tech demand more than this. Most projects are much smaller, but each of them provides experience, not just vocabulary. Active engagement with course material is what fosters deeper learning, and we don’t have time for that in commercial training.

As a result, instead of starting from a well-built-up foundation, testing practitioners learn most of what they learn about testing on the job and in relatively simplistic training courses. They reinvent the field, each for herself or himself, learning the same lessons that we have learned 10, 20 or 30 years before. This is not a path to significant progress in the field.

This low level of common knowledge in the field is convenient for certification courses. They can sell a few days of training (at a high price) that is enough for students to pass a relatively trivial exam. Think of the absurdity of this. The current certifications are essentially advertisements that you (the certificate holder) know as much as someone with no experience and three days of training. Would you hire someone who was proud of that low a level of knowledge?

I keep meeting people who tell me that they send their staff to these kinds of courses because the course teaches everyone a common vocabulary, and our field needs a common vocabulary. I don’t personally think that a common vocabulary has much value. What is more valuable is learning to ask someone else what they mean when they say something.

Thirty years ago, a “big” program was 10,000 lines of code. Today, even cell phones have millions of lines of code. This enormous change in productivity comes from revolutions in the practice of programming and software design. But we aren’t seeing revolutions in testing, not like the change into structured programming or out of structured programming and into object-oriented. No one would think of teaching the programming methods of the 1970’s or 1980’s as state of the art today. But the current certifications in testing (and many of the “applied” university and college courses) seem to me to be based on “best practices” and attitudes that aren’t much different from what I started writing Testing Computer Software to rebel against back in 1983.

We need to make progress on the education problem, or we will dig deeper and deeper into the same old ruts.

uTest: How well are companies adhering to The Software Consumer Bill of Rights (which you wrote in 2007)? Have conditions improved for the average software consumer?

Kaner: The Principles of the Law of Software Contracts have finally been adopted by the American Law Institute. That will probably reflect itself in judicial opinions over the next few years.

In terms of legislation, the United States is in a dysfunctionally polarized period. Very little legislation of any kind is being passed. We might not see useful statutes (laws passed by legislatures) on computer law for several more years. Until then, companies will start acting more responsibly after some successful lawsuits or enforcement actions.

uTest: James Whittaker has made the jump from academia to corporate America.  Will we see you take your considerable talents “across the aisle” at some point (assuming that you aren’t doing so already)?

Kaner: For me, this draws a false dichotomy.

As I said in my last note, programming has been evolving in powerful ways. Testing: not so much. Developing better education/training for testers is one part of the way out of this hole. That’s what I’m currently working on. I’m working at a university because universities house the cutting edge for educational practice.

On an ongoing basis, I ask myself whether I am working on the projects that have the potential to make the greatest positive impact and whether I still think I can realistically make the kind of contribution that is needed. For now, I’m in the right place doing what I think are the right things for me to be doing. Later, that will certainly change. When? How? I don’t know.

Editor’s note: Check back tomorrow for part III of the interview.

Essential Guide to Mobile App Testing


  1. says

    Undeniably believe that which you said. Your favorite justification appeared to be on
    the web the simplest thing to be aware of. I say to you, I certainly get irked while people think about worries that they
    just don’t know about. You managed to hit the nail upon the top as well as defined out the whole thing without having side-effects , people could take a signal. Will probably be back to get more. Thanks

  2. says

    Hi, I do think this is a great blog. I stumbledupon it
    ;) I may return once again since I bookmarked it.
    Money and freedom is the greatest way to change, may you be rich and continue to guide others.

  3. says

    software testing is more of an “art” then a science, no matter how many tools you have it can never compensate the requirement of a great QA team. an experienced team is an asset


Leave a Reply

Your email address will not be published. Required fields are marked *