This month, we revisit Cem Kaner. Cem recently published The Domain Testing Workbook and is working on a collection of other workbooks and projects in addition to teaching several courses at Florida Tech.
In today’s interview, Cem explains his new workbook, discusses why it’s important for experienced testers to keep studying and improving, tells us what’s wrong with the testing culture today and hints at maybe having a solution for the QA credentials battle.
uTest: You’ve written quite a few books on software testing and you had a new book – The Domain Testing Workbook – come out in the past few months. Why do you enjoy writing these books and who are you trying to help?
Cem Kaner: My overall goal is to improve the state of the practice in software testing. How can we improve what working testers actually DO so that they are more effective and happier in their work?
The Domain Testing Workbook is the first of a series that focus on individual test-design techniques. Our intent is to help a tester who has some experience develop their skills so that they can apply the technique competently.
What’s wrong with the way domain testing is currently taught?
CK: There’s nothing wrong with the way domain testing is taught. Teachers introduce students to the two basic ideas: (a) subdivide a large set of possible values of a variable into a small number of equivalence classes and sample only one or a few values from each class. This reduces the number of tests to run. (b) When possible, select boundary values as your samples from each class because programs tend to fail more often at boundaries.
In general, students understand these introductions and can explain them to others.
This level of analysis works perfectly when you test Integer-valued variables one at a time. There are lots of Integer-valued variables, and it makes a lot of sense to test every variable on its own, if you can, before you design tests that vary several variables at the same time. So, I think many courses do a fine job of introducing a useful idea to students in a way that helps them use it.
However, there is much more depth to the technique than that. Here are four examples:
- It is common to look only at the input values and decide pass/fail based on whether the program accepts “good” inputs and rejects “bad” ones. A stronger approach goes past the input filter. For example, enter the largest valid value. The program should accept this. Suppose it does. Now continue testing by considering how the program uses this value. What calculations is it used in? Where is it displayed, stored, or compared to something else? Is this largest-acceptable value actually too large for some of these later uses?
- There are other types of variables, not just Integers. Different risks apply when you are testing floating-point numbers or strings. Dividing them into equivalence classes is a little trickier.
- We usually test variables together. Any test of a real use of the program will involve many variables. Even if you leave most of them at their default value, the program considers the values of lots of variables when you ask it to do something meaningful. We can manage the number of combinations to test using techniques like all-pairs. In all-pairs testing, the tester chooses a set of maybe 10 variables to test, then chooses a few values of each variable to test, then uses a tool like ACTS or PICT to create a relatively small set of tests (maybe as few as 30) that will combine these values of these 10 variables in an optimized way. (ACTS and PICT are free tools from the National Institute of Standards and Technology (ACTS) and from Microsoft (PICT). One of the challenges of this type of testing is picking the best values for the individual variables—and that brings us back to domain testing.
- We often test variables together that are related to each other. How do you choose boundary values when the boundary (such as largest valid value) of one variable depends on the value of the other? This particular issue appears often in university textbooks but there hasn’t been enough practical advice for working testers.
The Domain Testing Workbook goes beyond the perfectly-good introductory presentations that appear in many books and courses. We want to help testers apply the technique to situations that are a little more complex but still commonplace in day-to-day testing.