Principles of Testing – Exhaustive Testing is Impossible

Over on the uTest Community Forums (the forums for our uTester community), a discussion has started listing some of the basic principles of software testing.  There have been quite a few good ones, and I wanted to highlight some of them in a series of posts.

Principle #1: Exhaustive Testing is Impossible

Put simply, you can’t test it all.  There’s just no way of validating that your software will run the way you expect on every variation of every system in the world.  Software has become too complex, and validating every possible system permutation, input combination, and software feature is nearly impossible.

For example, if you are developing the next Flash video game, are you sure your code will run in every browser, with any recent version of Flash, on all platforms, all while making sure that your game still performs adequately?  Most people can’t get this kind of coverage cost effectively.

Better Testing Options

Even though exhaustive testing is usually impossible, there are a few possible options that can help when applied either individually or in combination with each other:

  • Support Less – Sometimes you can’t support every platform or combination.  This option reduces the number of potential customers for your product, but it makes developing and testing easier.  This is a good choice for products with narrower markets.
  • Test More – You can’t test everything, but maybe you can test more than before.  This option raises costs, but it’s a good choice when stability is crucial.
  • Make it Simpler – Leave out complicated features or simplify the design.  Leaving out features may mean that customers are unsatisfied, but it can also mean a product is ready for shipping sooner.
  • Ship with Bugs – Maybe the bugs don’t matter.  In this case, you accept that your product is buggy but you ship it anyway.  Customers may not be happy when things crash, but it may not matter.  This is a good choice if you know the bugs can be managed, or if your customers don’t really care about stability.

Obviously, software companies do all four of these every day, sometimes in the same product.  There’s no one right answer, and it’s always best to be prepared for bugs in your product.

uTest can make it easy to test more in a very cost effective fashion.  Learn more about our software testing services, or sign-up for our upcoming interactive uTest demos.

[Read more of our Principles of Testing series.]

Essential Guide to Mobile App Testing

Comments

  1. says

    Another important option to keep in mind is being smart about how testing is done, and achieving as much coverage as possible in as few tests as possible. Realistically, the only way to do this efficiently is to use a test design tool (like Hexawise). Numerous studies have shown that testers find more than twice as many defects per tester hour when test design tools are used to develop the test cases. This is because test design tools can systematically cover thousands of combinations far more efficiently than humans can (e.g., with less unintentional repetition and with zero “forgotten” combinations).

    Justin Hunter
    Founder of Hexawise test design tool
    Blog: http://www.hexawise.wordpress.com
    Free test design tool: http://www.hexawise.com/users/new

  2. Mahmoud Al-Yasein says

    Hi Stanton
    This is good ideas,
    ISTQB certification add also that instead of doing an exhaustive testing you need to focus on risk analysis and PRIORITIES should be used to focus testing
    efforts

Leave a Reply

Your email address will not be published. Required fields are marked *