There is no right way to do testing – however, there is a wrong way. In order to test software to the best of your ability – you need to be able to think outside of the box and focus on quality. In this guest post , Joshua Grant shares some questions testers can ask themselves to stimulate good insights. Joshua is an automated tester currently living in Toronto, Ontario. He is very into software testing, development, management and big ideas.

******

I’ve been a software tester for almost two years now. In that time, I’ve primarily worked in environments that test using test case approaches. Each case is designed to test a specific aspect of an application, and is usually set up in a “pass/fail” manner. When executed, either manually or automatically, each test either passes (the software is “good” or doesn’t have issues) or fails (a bug is discovered and should be documented). This may be simplifying things a bit, but in some places, it wouldn’t be much. While there are benefits to test case pass/fail approaches, they can also fall short, sometimes quite obviously. In my experience, it can also give testers something of testing tunnel vision, where they stop thinking about testing creatively and think of things as more of formal processes with logical conclusions. This is a bit unfortunate, since testing can be quite non-linear and interesting. Despite this, some organizations insist on the pass/fail approach, and shoehorn testing into that format.

So, with what I feel I need to give a major hat tip to James Bach, here are some questions that a tester can ask themselves to stimulate some good insights into software when required to write and execute pass/fail test cases:

What would happen if….

•a test case had different results after 10 successive runs?

•a test case started passing when it was failing?

•a test case always passed/failed, no matter what you did?

•a test case never passed/failed, no matter what you did?

•there were certain circumstances under which the test could both pass and fail for identical setups?

•multiple test cases were always run in the same order?

•multiple test cases were run in a random order?

•the test case was executed a year from now?

•the test case was executed a year ago?

•the pass/fail result was accidentally reported as the opposite?

•no one ran the test for a week? a month? a year? ever again?

•you told your direct report that you ran the test, but you actually didn’t?

•you told your direct report that this test would never be ran again? or the test would be ran every day?

•the test was run on your home machine with no backups or testing environment set up?

•the test case was executed by someone other than yourself?

These are just some ideas I’ve been thinking of. Some of which seem somewhat controversial (“You’re telling me to contradict my boss?! Are you nuts??”) but if you stop and think about these questions from a testing or quality perspective, they could generate insights even if all test results are pass/fail. For example, these questions can help individuals think about test cases that don’t really do much, and just waste time and effort to run, like tests that always return the same result, or tests that no one pays any attention to either way.

Quality often depends on more than just a bunch of test case results, and this list tries to help testers realize that.