Our Testing the Limits guest this month is Seth Eliot, the Senior Knowledge Engineer of Test Excellence at Microsoft. In this role, he focuses on driving best practices for development testing across the entire company. Prior to Microsoft, Seth had a successful stint at Amazon in addition to several startups. Apart from his professional background, Seth is one of the industry’s very best bloggers, writers and presenters. For proof, check out his blog or follow him on Twitter.
In this must-read interview, we ask him about testing challenges at Microsoft, including those of Bing and the new Surface tablet; the notion of testing in production (TiP); the most rewarding testing project he’s ever worked on; big data and more. Enjoy!
uTest: You’ve spent the bulk of your testing career with two of the most successful companies of all-time: Amazon and Microsoft. Unfortunately, most testers spend their careers with companies that – how shall we put this – aren’t so successful. In your opinion, is testing easier or more rewarding when the company is doing well? And what advice do you have for testers who might be working at a dysfunctional company?
Seth: The most satisfying testing job I ever had was a small startup in Pittsburgh called CoManage. It ultimately fizzled, but at the time we thought we were all going to be millionaires and I was consistently surprised to walk out of the test lab to see it was dark outside and I didn’t even know where the day had gone. If your company is dysfunctional, ask yourself if there is something you can do to turn it around and turn it into one of those dream successes. Learn new strategies and approaches for software engineering, change the direction, and bring new life to the company. At best you will be the hero, at worst you will have learned some valuable skills and lessons for finding that next job.
uTest: Prior to your current role at Microsoft, you were the Senior Test Manager for the team working on Bing, where you were primarily tasked with exabyte storage and data processing challenges. What were some of the specific testing challenges here and how were you and your team able to overcome them?
Seth: Yes, this is an internal system called Cosmos – a massively scalable, distributed data processing system. The technical challenge to put it simply is how do you test something so big and complex? I was fortunate to have a really talented team of testers who built out tools and monitors that enabled us to evaluate end-to-end test cases leveraging actual jobs being run in the production system. This led to us to finding the bugs that really mattered – those that affect real users. We were even able to prioritize our test scenarios based on the revenue impact of the user workflows and on the current pain points experienced by those users. This is an advantage of having an internal customer, but with good monitoring you can also approach this level of insight with external customers too.
uTest: We’d be remiss if we didn’t ask you about the recent launch of Microsoft Surface. First off, did you know about the project beforehand? If so, good job keeping it a secret. Secondly, what are some of the big testing challenges you’d expect to be associated with this project? The hardware? The Windows 8 OS? The touch functionality? We’d love to hear your thoughts on this HUGE Microsoft endeavor.