Testing the Limits With Microsoft’s Seth Eliot

Our Testing the Limits guest this month is Seth Eliot, the Senior Knowledge Engineer of Test Excellence at Microsoft. In this role, he focuses on driving best practices for development testing across the entire company. Prior to Microsoft, Seth had a successful stint at Amazon in addition to several startups. Apart from his professional background, Seth is one of the industry’s very best bloggers, writers and presenters. For proof, check out his blog or follow him on Twitter.

In this must-read interview, we ask him about testing challenges at Microsoft, including those of Bing and the new Surface tablet; the notion of testing in production (TiP); the most rewarding testing project he’s ever worked on; big data and more. Enjoy!


uTest: You’ve spent the bulk of your testing career with two of the most successful companies of all-time: Amazon and Microsoft. Unfortunately, most testers spend their careers with companies that – how shall we put this – aren’t so successful. In your opinion, is testing easier or more rewarding when the company is doing well? And what advice do you have for testers who might be working at a dysfunctional company?

Seth: The most satisfying testing job I ever had was a small startup in Pittsburgh called CoManage. It ultimately fizzled, but at the time we thought we were all going to be millionaires and I was consistently surprised to walk out of the test lab to see it was dark outside and I didn’t even know where the day had gone. If your company is dysfunctional, ask yourself if there is something you can do to turn it around and turn it into one of those dream successes. Learn new strategies and approaches for software engineering, change the direction, and bring new life to the company. At best you will be the hero, at worst you will have learned some valuable skills and lessons for finding that next job.

uTest: Prior to your current role at Microsoft, you were the Senior Test Manager for the team working on Bing, where you were primarily tasked with exabyte storage and data processing challenges. What were some of the specific testing challenges here and how were you and your team able to overcome them?

Seth: Yes, this is an internal system called Cosmos – a massively scalable, distributed data processing system. The technical challenge to put it simply is how do you test something so big and complex? I was fortunate to have a really talented team of testers who built out tools and monitors that enabled us to evaluate end-to-end test cases leveraging actual jobs being run in the production system. This led to us to finding the bugs that really mattered – those that affect real users. We were even able to prioritize our test scenarios based on the revenue impact of the user workflows and on the current pain points experienced by those users. This is an advantage of having an internal customer, but with good monitoring you can also approach this level of insight with external customers too.

uTest: We’d be remiss if we didn’t ask you about the recent launch of Microsoft Surface. First off, did you know about the project beforehand? If so, good job keeping it a secret. Secondly, what are some of the big testing challenges you’d expect to be associated with this project? The hardware? The Windows 8 OS? The touch functionality? We’d love to hear your thoughts on this HUGE Microsoft endeavor.

Seth: I found out about Surface when you found out about it, so obvious kudos to that team for keeping this secret so tightly wrapped.  It seems to me that re-using an existing product name was quite clever – it arouses no suspicion when you see it next to someone’s name in the internal address list.

Honestly hardware testing is almost the antithesis of the services testing where I have concentrated my career. In the latter you have rapid cadence and an ability to get things out there and experiment. With hardware you better get it right before launch time. I remember when I was QA Manager at Amazon and had an original Kindle before they were released. I got an email saying that Bill and Ted from the lab were going to come around to install new chips in all the test Kindles we had and thought to myself, that is just not going to scale after launch!

Getting back to the Surface, what excites me about this new product is also what poses the greatest challenges for testers… it’s all new. New hardware, utilizing new manufacturing; new operating system with a whole new interface; plus as you said the touch and gestures functionality. I have no doubts the teams involved will knock it out of the park, which will be a true testament to them considering those hurdles to overcome.

uTest: You’re a big proponent of Testing in Production (TiP) – an idea that surely scares the crap out of most test teams. How is TiP leveraged at Microsoft and what are the main advantages of this approach? And while you’re at it, what (if any) are the big disadvantages?

Seth: TiP is a methodology suitable for services where we (the engineers) can see what is happening with the service via monitoring, and we control the deployment. Testers hear “Testing in Production” and remember their worst nightmare dev bypassing test to throw cruddy code to users, but to be TiP it has to be a conscientious strategy aware of risks and utilizing mitigations. It’s not a secret that products like Bing, Amazon, Google, and Facebook launch experiments all the time exposing new code and features to a small number of users. This is a powerful way to get information on your product in real-world conditions that at the scale of these services simply cannot be reproduced in a test lab. TiP techniques often leverage the massive flow of data constantly emitted from our services. Hotmail measures actual performance experienced by users (anonymously of course) across all operating systems, browsers, and locations around the world to pinpoint trouble spots and optimize performance. In other cases we might be testing production itself. Our Office web services run validations after each deployment to check things like certificate validity, network health, and that all the bits got where they were supposed to be.

The advantage of TiP is you find things you just cannot find (or are prohibitively expensive to find) pre-production. The corollary to this is that you should not be finding bugs in production that you could have easily found earlier by unit testing or simple functional testing. It’s not necessarily a disadvantage, but based on your technical and business needs, some TiP methodologies might not be right for you.

uTest: In your current role, you focus on driving best practices in QA (among other areas) across the entire company. Is testing viewed drastically different in some parts of the company than others? And how difficult is it to get a company as diverse as Microsoft to adopt similar best practices?

Seth: In Engineering Excellence (my group) we never try to get all teams to adopt similar patterns. Instead we identify what works for teams at Microsoft and in the industry, and make sure other teams here have access to that information. I have worked with many, many great teams across Microsoft who are hungry for this knowledge. Being able to help those teams makes this a very satisfying role for me. Microsoft values data-driven decision making and I expect individuals on teams to be skeptical, so I better have good examples of what has worked and why.

Testing across Microsoft definitely has some distinctions from group to group. Some are cultural, but most often I think the product drives the test strategy. What you need to do for a service, boxed product, or hardware are necessarily different. Even among services how you treat a consumer product like Bing will be different than what you do for an enterprise product like Office 365.

uTest: What’s the best part about testing at Microsoft?

Seth: Testing is a respected profession at Microsoft.  We have many principal level individual contributor testers (for comparison at Amazon when I left there were none).  We have even more high level test managers including three VPs in test.  We have several test communities with their own events and speakers.  At Microsoft testers can really grow their careers and learn the latest technologies. Career-wise every tester who can, should do a gig at Microsoft.

uTest: As one of the more prolific testing writers out there, we’re interested to hear your thoughts on the next test trend. In your opinion, what’s the biggest testing trend that no one is talking about yet?

Seth: It’s more of an evolution of what I have already been talking about around testing services and Testing in Production. I call it TestOps.

Testers need to move out of the mindset of writing tests – running tests – evaluating results. Instead of using test results from a daily run as your quality signal, use the big data pipe coming off of your product (generally this refers to services) as your quality signal. This includes system data like CPU, API requests, system response time as well as (properly anonymized) user data. It also includes data emitted from synthetic transactions that you can run constantly in production. These are indeed test cases, but instead of getting just a daily red/green, you get constant availability and performance. This is the technology, but it also necessarily will change our software engineering organizations. The questions of roles and specialization versus generalization will need to be answered to meet each organization’s needs. And the emergence of the Data Scientist as part of the engineering team is an exciting outcome of the TestOps approach.

uTest: Which QA method do you prefer and why: Agile, Waterfall or other?

Seth: Unsurprisingly my response is… first show me examples of teams that have done each of these well and not-so-well, then show me the system under consideratio, and then I can give you an answer.

uTest: True or false: It’s better to have testers who are evangelists for their product, than testers who are evangelists for their department.

Seth: It’s best to connect with what you are working on and evangelize that. If you feel more connected to the product, then go for it… same for your department.

uTest: Fill in the blank: The most rewarding testing project I ever worked on was _____.

Seth: Version 1.0 of the first product from that aforementioned startup. I can’t even remember its name now! But as well as a stint at Microsoft, I also recommend one at a small, exciting startup for all testers that can do it.

uTest: You’re one of the more active participants on the test speaker circuit. Care to give our readers a head-ups on your next appearance, as well as what you plan to cover?

Seth: STPCon has been kind enough to ask me back to give an encore of A-Z Testing in Production for the October 2012 conference. If you missed it the first time come on by. If you already saw it and are still hungry for TiP, I will be adding some new things and putting more emphasis on big-data, so come by and let me know what questions you may have come up with since seeing the first TiP talk in March.

uTest: What’s Seth Eliot doing when he’s not improving the state of software testing?

Seth: Since I live the Pacific Northwest my answer should be scaling mountains, biking trails, and kayaking Lake Washington, but nah…I prefer to spend Saturday morning taking my girls (4 years old and 18 months old) to Starbucks for vanilla milk and cake pops.  This also has the advantage of giving my wonderful wife some break time as thanks for letting me jet all over the country to teach and give talks.

Editor’s note: We hope you enjoyed our latest Testing the Limits interview. If you have a suggestion for future interview guests, send them along to blog@utest.com.


1 Comment
  • No Fee Loans says:

    When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I get three e-mails with the same comment.
    Is there any way you can remove me from that service?
    Thank you!

Write Your Comment