Over on the uTest Community Forums (the forums for our uTester community), a discussion has started listing some of the basic principles of software testing. There have been quite a few good ones, and I wanted to highlight some of them in a series of posts.
Principle #1: Exhaustive Testing is Impossible
Put simply, you can’t test it all. There’s just no way of validating that your software will run the way you expect on every variation of every system in the world. Software has become too complex, and validating every possible system permutation, input combination, and software feature is nearly impossible.
For example, if you are developing the next Flash video game, are you sure your code will run in every browser, with any recent version of Flash, on all platforms, all while making sure that your game still performs adequately? Most people can’t get this kind of coverage cost effectively.
Better Testing Options
Even though exhaustive testing is usually impossible, there are a few possible options that can help when applied either individually or in combination with each other:
- Support Less – Sometimes you can’t support every platform or combination. This option reduces the number of potential customers for your product, but it makes developing and testing easier. This is a good choice for products with narrower markets.
- Test More – You can’t test everything, but maybe you can test more than before. This option raises costs, but it’s a good choice when stability is crucial.
- Make it Simpler – Leave out complicated features or simplify the design. Leaving out features may mean that customers are unsatisfied, but it can also mean a product is ready for shipping sooner.
- Ship with Bugs – Maybe the bugs don’t matter. In this case, you accept that your product is buggy but you ship it anyway. Customers may not be happy when things crash, but it may not matter. This is a good choice if you know the bugs can be managed, or if your customers don’t really care about stability.
Obviously, software companies do all four of these every day, sometimes in the same product. There’s no one right answer, and it’s always best to be prepared for bugs in your product.
[Read more of our Principles of Testing series.]