Software Testing Blog

So far, our Guest Blogger series has demonstrated the incredible domain expertise of our community – and this post will be no exception. With us this month is uTester David Vydra. A resident of San Mateo, California, David is an Agile Tester for Guidewire Software (he’ll explain what they do in a bit). If his name sounds familiar, that’s because he is the co-author of testdriven.com – a very popular testing site covering developer testing, automated exploratory testing, model-based testing and more.

In this post, David offers us his “notes from the field”, where he’ll be addressing the role of  software testers; criteria for hiring testers; the experts he follows and more. We’re confident that you’ll like it. Enjoy!

I am thrilled to be invited to share my testing experiences with the uTest community. I hope my account will encourage more organizations to adopt the agile way and more testers to find fun and fulfilling jobs.

I joined Guidewire Software about seven months ago as an Agile Tester. We make core software for the Property and Casualty insurance carriers. If you have car or homeowners insurance, you may be serviced using our software. Right from the start, Guidewire has used agile development methods and credits a good part of its success to this philosophy.

Our applications are complex. It typically takes several months for a tester to become fully productive because there is so much domain knowledge and in-house tooling to learn. Our applications are enviably flexible, and each installation is customized to fit the specific needs of the insurance company. In order to empower the customization process, we provide a number of development tools including a custom java-compatible scripting language, an IDE, a GUI framework, and a screen painter.

For testing purposes, we ship a testing framework and have developed many internal tools such as a distributed continuous integration server consisting of several hundred hosts running various permutations of the operating systems, databases, and web servers that we support. We have an advanced UI testing framework that integrates with Selenium and a set of domain ‘builder’ classes that help with rapid test data creation.

Perhaps the biggest difference from other places where I have worked is the way testers are treated here. We sit together with developers grouped by functional areas. There is no dedicated QA space with the proverbial wall to throw releases over. We attend all product requirement meetings and participate in the product demos. Testers are truly first-class citizens of the development team.

Everyone does testing at Guidewire. Developer interviews include a large testing component, and the test-driven nature of our development is emphasized to make sure that new developers are on board with testing practices from the beginning. We have started a weekly Agile Testers brown bag lunch, which is well attended by testers and developers, as well as development and program managers. During this lunch, everyone is encouraged to demonstrate techniques that worked well or share war stories of challenging bugs. We discuss the latest books on testing and upcoming conferences, but no business decisions are made.

At this point you are probably wondering who writes the automated tests, developers or testers? On my team we have agreed to the following: both developers and testers write automated tests, but developers own and maintain the tests that run on the continuous integration server. Testers are focused on exploratory testing and use automation when appropriate to speed up the process. The advantage of this approach is that we recognize testers’ testing skills over their programming skills. It takes programming skills to maintain a large body of tests over a long period of time in an object-oriented language, and developers are typically better and quicker at this, whereas testers excel at domain knowledge, analytical skills, and the testing heuristics necessary for effective testing on tight schedules.

We work in two-week sprints and test continuously. Because developers practice test-driven development and the continuous integration server constantly reports test failures, the system is almost always runnable from the main branch in the source control system. We never have to wait for an official release to start testing. Both developers and testers work from story cards posted on the project board. If a requirement is unclear, the product managers sit next to us and provide instant clarification.

When we write test scripts that drive exploratory testing, we meet with developers from time to time to discuss which tests are good candidates for automated regression functional tests and turn them over to developers. Both developer-owned and tester-owned tests can coexist in the same source file by using the @ManualTest annotation. Because our application runs inside a web server, it’s important that our exploratory tests run inside a live application, not just in a testing harness. To accomplish this, one of our engineers wrote a tool called QuickTest that can load and run tests inside a live server. We recognize that tight engineer-tester collaboration is one of the most important aspects of our process.

But what if you are a tester with significant programming experience—is there a place for you at Guidewire? Absolutely! What I have described thus far is the testing approach we use on my application team. We also have testers on the platform team that possess significantly higher programming proficiency because they are testing our scripting language, IDE, and related frameworks. And last, but not least, we have our ‘testing labs’ projects such as the distributed, pseudo-random, model-based testing system I work on during my slack time.

If by now you are suspicious that all this sounds too good to be true, let me confess some of the challenges that remain with us. Due to the inherent complexity of our platform, unit tests incur a significant start-up time. We are fighting this by finding ways to tune our core code and by buying faster workstations with solid state drives. Over the years, we have accumulated a very large number of tests, and test maintenance is taking a noticeable chunk of engineering time. We are looking for ways to reduce test fragility and triage the regression test suites based on their relative value. Lastly, we have our share of non-deterministic tests—these are always ‘fun’ to diagnose and fix—they are often a symptom of concurrency or data locking problems and thus may be some of the more valuable tests in our arsenal. Unlike some smaller agile projects that boast their systems are always in a shippable state, we acknowledge the existence of “dark matter” – the stuff that is invariably found and must be fixed prior to a release – even as we are working hard to reduce its volume.

I would like to conclude with a short treatise on hiring testers. Too many development organizations are enamored with building out test automation at the expense of useful, efficient testing and have favored hiring programmers into testing jobs regardless of their true interest and talent for testing or their actual testing skills. This is a mistake. Some of the best testers I have worked with studied math, physics, philosophy, and psychology. Testing, like programming, is a performance art. When hiring testers, it’s important to have them test something during the interviews. It can also be very helpful to look at the testing they have done previously. This is typically impossible when someone has worked for a private company, but it is possible on the public projects on uTest.com. “Wow! How did you find that bug?” is a great question to get from the hiring manager during an interview. I value uTest as a fantastic resource for learning testing skills from peers all around the world, and I visit the site often.

I owe a debt of gratitude to the following individuals who have nurtured and supported me in my studies of testing. I encourage you to seek out their writing on the topic:

• Cem Kaner
• Elisabeth Hendrickson
• James Bach
• Paul Carvalho
• Michael Bolton
• Brian Marick
• Bret Pettichord

Special thanks to my peer and mentor at Guidewire, Dave Liebreich.