Software Testing Blog

We’re happy to have Brian Rock as our Guest Blogger this month. A resident of Austin, Texas, Brian has brought more than 10 years of software engineering and leadership experience to the uTest community (where he is among the most active members). Prior to his software days, Brian served as Sgt. for Combat Arms Forward Recon Team in the U.S Army – this after obtaining his Master’s degree in Engineering Management from the University of Texas. In this post, Brian incorporates each of these past experiences into an insightful commentary on the nature of exploratory testing. Enjoy!

In the U.S. Army the branch of the Cavalry Scouts are known as the “Eyes of the Battlefield.” Their mission is to provide forward reconnaissance and relay critical information about the enemy to the commanders. This information consists of enemy strength, threat level, location, and so on. The commanders can then base their decisions and strategy on this mission-critical information.

During software application development projects, like the Cavalry Scout, today’s Quality Engineers are the “Eyes of the Battlefield.” Through the techniques of exploratory testing these engineers are able to work in unfamiliar and sometime hostile environments, quickly diagnosing the situation and providing reports concerning the quality of the application and its mission readiness capability.

Instead of providing information about enemy strength, the quality engineers are more concerned with the number and location of defects, their severity level, and the impact to the application. With this information, the technical managers can base their decisions and strategy to ensure the product is delivered on time, in budget, and with acceptable quality.

Cem Kaner originally coined the term Exploratory Testing in 1983, defining it as:

…a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities run in parallel throughout the project.

This view of exploratory testing is aligned with our analogy of the Cavalry Scout, whose success depends on learning on his feet and reacting according to the input from his surroundings. In exploratory testing, the tester is using his or her immediate feedback from the interaction with the application on which to base the next test decision. This immediate feedback loop, when applied in a disciplined manner, can be as effective as a fully fleshed out test plan based on deep analysis of the requirements and the applications design.

In many of today’s software development projects, quality engineers are working on projects that are under specified or lacking specifications all together. Because of this, the quality engineers are severely handicapped when trying to create a fully specified test plan. A great deal of time is often spent trying to reverse engineer product specification documents to develop enough domain knowledge to develop these plans. A more effective and efficient use of this time would be to begin testing with what knowledge we do have and use the feedback from these activities to specify the testing plan as we go. This is the heart of Exploratory Testing.

This type of testing is also an excellent fit in agile environments where the specifications are being done as the product is being developed. The quality engineers begin with a basic understanding of the product and what acceptance criteria it will need to meet in order for it to be considered complete. With these loose specifications (objectives), the quality engineer must test the product and ensure that it meets the quality standards needed for the release to be successful. This is accomplished by integrating the feedback from each test activity back into a dynamic test plan that evolves as the application evolves.

This immediate feedback learning technique, which is so fundamental in exploratory testing, is the same method that is critical to mission success for the US Cavalry Scouts. The cavalry scout squad begins their mission with a loose understanding of environment, a set of objectives, and some mission parameters. Armed with this information they execute the mission, constantly readjusting their plan based on the current situation and the information gained thus far. The critical point to understand with this technique is that the mission objectives have not changed, only the route to meet those objectives is altered based on the information gained during the mission.

Exploratory testing is often confused as an ad-hoc test methodology lacking discipline and is frequently used as an excuse for not following a disciplined process. When exploratory testing is applied correctly, this couldn’t be further from the truth. Exploratory testing relies on skilled testers applying many different engineering methods to validate the correctness of a product. The only difference is that these engineers may be lacking a deep understanding of the product. It is through exploratory testing that they develop the understanding while providing value through test activities.

This testing approach is an ideal fit for a crowdsourced testing community such as uTest. The uTest engineers are mostly unfamiliar with the detailed specifications of the applications which they test. However, by using past experience and real-time feedback, a highly skilled squad of exploratory testers can effectively test an application which they are unfamiliar with and can provide valuable feedback.