Testing the Limits with Noah Sussman – Part I

Our Testing the Limits guest this month is Noah Sussman, test architect at Etsy. If you recall, we chatted with Noah last month as part of our STPCon video interview series, which you can watch here. Noah’s professional background includes a stint at Barnes & Noble, where he was the Tools and Automation Specialist, in addition to being a technical adviser to several startups and organizations. To learn more about Noah, you can follow him on Twitter, or check out his posts on github or Quora.

In part one of our interview with Noah, we get his thoughts on how Etsy tests software; why engineers should take more responsibility for testing their code; testing at startups and much more.

uTest: When it comes to testing, there’s certainly no shortage of collective knowledge to draw from. Who are some of your software testing role models – in terms of both companies and individuals – that you strive to emulate?

NS: As far as people, I work with a great team at Etsy. I feel really lucky to work with this many talented and knowledgeable people. Check out our engineering blog, http://codeascraft.etsy.com, as well as http://github.com/Etsy. You’ll find a large number of people with a wide range of ideas, all of which I believe are interesting. Once you’ve read their blog posts on Code As Craft, you should check out their personal blogs too, because those are fascinating as well.

Sebastian Bergmann and Jason Huggins have also been very helpful to me as I tried to figure out the best way to make CI work with Etsy’s unique deployment process.

As to companies, Blizzard, Amazon.com and Facebook all are using the Internet to build new kinds of communities and businesses. All three are essentially collections of Web services. But they’re able to manage their enormous communities and are constantly adding new features that their users pretty much love.

uTest: You’re an established figure in the testing space now, but you’ve spent much of your career on the development side of things. What (if any) were some of the biggest challenges you faced transitioning from dev to testing? And conversely, what advantages has your prior dev experience given you in your testing career?

NS: When I began making Web sites in 1999, I manually tested each site I built. Sometimes I worked with a QA team and sometimes not, but I have always checked my own work before delivering it, regardless.

As the Web grew, my projects got more complex. Tasks like making sure there were no broken links on a site quickly became infeasible to do manually. Soon even manually viewing every page on a site became unworkable as well.

So I learned some Perl and suddenly I could do static analysis. Then I learned to parse and alert on the output from tools like HTMLTidy, the W3 CSS Validator and later JSLint. After that, the number of bugs in my code dropped dramatically.

At first most people thought I was a little nuts for taking this approach. In 2001 almost no one cared if a site was robust. Web sites were cheap, throwaway things — “brochureware.” But a few years later along came Google Maps and GMail and now Web sites were first-class applications, expected to perform well and last a long time.

After that I just found myself more and more involved in helping teams produce quality Web sites. I think once everyone realized testing Web sites was important, I just naturally got drawn into that because I was interested in the domain. There aren’t a whole lot of Web developers who are really deeply passionate about testing. Or if there are, I haven’t met most of them yet!

uTest: What’s the biggest difference between testing at a startup versus testing at a larger company like Etsy?

NS: That’s a funny question to me because I still think of Etsy as a startup. Most of the companies I’ve worked for were much larger and more bureaucratic.

Anyway, I think that for a very early stage startup without a (good) product in production, time to market is the only thing that matters.

That said, it is worthwhile to take the 20 minutes to set up Jenkins and have it run your tests and some static analysis every time anyone checks in code. Beyond that I would not spend any time or money on infrastructure until the first (successful) version of the product had shipped.

For a startup with a successful product, paying down technical debt becomes paramount. Especially the first release tends to be done with very little testing and a lot of last-minute hacked-together solutions. Emergent problem solving is good, really good, when all that matters is getting to market. Without expedient solutions you might never get to market at all. But that’s only the first phase in a Web site’s life cycle, and it’s important not to hold on to those initial strategies once their usefulness has been outgrown. Improving test coverage and refactoring for comprehensibility are tightly coupled activities, and both are very important for the long-term survivability of a complex piece of software.

uTest: We find that a lot of people in the software business – particularly those in testing – are reluctant to adopt the agile methodology. Was this the case for yourself and your peers at Etsy? And what advice do you have for testing teams that might be resisting the move to agile?

NS: I like to move fast and deliver quality and I have always tried to work for companies that share that philosophy. At Etsy they hired a team who believe that the right thing is to experiment a lot in the name of resiliency and adaptability. I think that hiring the right people was key.

I think when people are happy and respect each other, they’re likely to want to do what’s best for the team. So assuming you have a culture like that and that you’ve identified rapid iterations as the right workflow for your team, it should be just a matter of trial and error and hard work to get that kind of process in place.

I do think all those elements have to be present for success: the right team, the right values and a willingness to repeatedly fail in the interest of progress. If an organization doesn’t have all those attributes, that’s a problem that needs to be addressed before it will be possible to move forward with any kind of process improvement.

uTest: A big part of your STPCon presentation dealt with the idea of asking engineers to assume more responsibility for testing their code. First off, explain why this is a good idea. And second, which group would be more reluctant to accept this idea: testers or engineers?

NS: Tests are no more and no less than executable documentation. When writing new code, it makes sense that it should be documented by the person closest to it: the author. That’s my rationale for developer testing.

Comment “doc blocks” and wiki pages, are always found to be inaccurate to some extent. By contrast automated tests fail noisily whenever they go out of date. So with regard to documentation, automated tests are uniquely advantageous in that (given a CI server) you at least have the option of keeping your documentation up to date whenever it starts to drift away from reality

Etsy was a comparatively small company when the decision to embrace developer testing was made. So by the time I got there, they had a great product and their code was looking pretty clean. Given this great environment, they had no problem recruiting more engineers who also felt strongly that testing was a good idea.

Editor’s note: Continue reading Part II.

Essential Guide to Mobile App Testing


Leave a Reply

Your email address will not be published. Required fields are marked *