Testing the Limits With Microsoft’s Seth Eliot

Our Testing the Limits guest this month is Seth Eliot, the Senior Knowledge Engineer of Test Excellence at Microsoft. In this role, he focuses on driving best practices for development testing across the entire company. Prior to Microsoft, Seth had a successful stint at Amazon in addition to several startups. Apart from his professional background, Seth is one of the industry’s very best bloggers, writers and presenters. For proof, check out his blog or follow him on Twitter.

In this must-read interview, we ask him about testing challenges at Microsoft, including those of Bing and the new Surface tablet; the notion of testing in production (TiP); the most rewarding testing project he’s ever worked on; big data and more. Enjoy!

********

uTest: You’ve spent the bulk of your testing career with two of the most successful companies of all-time: Amazon and Microsoft. Unfortunately, most testers spend their careers with companies that – how shall we put this – aren’t so successful. In your opinion, is testing easier or more rewarding when the company is doing well? And what advice do you have for testers who might be working at a dysfunctional company?

Seth: The most satisfying testing job I ever had was a small startup in Pittsburgh called CoManage. It ultimately fizzled, but at the time we thought we were all going to be millionaires and I was consistently surprised to walk out of the test lab to see it was dark outside and I didn’t even know where the day had gone. If your company is dysfunctional, ask yourself if there is something you can do to turn it around and turn it into one of those dream successes. Learn new strategies and approaches for software engineering, change the direction, and bring new life to the company. At best you will be the hero, at worst you will have learned some valuable skills and lessons for finding that next job.

uTest: Prior to your current role at Microsoft, you were the Senior Test Manager for the team working on Bing, where you were primarily tasked with exabyte storage and data processing challenges. What were some of the specific testing challenges here and how were you and your team able to overcome them?

Seth: Yes, this is an internal system called Cosmos – a massively scalable, distributed data processing system. The technical challenge to put it simply is how do you test something so big and complex? I was fortunate to have a really talented team of testers who built out tools and monitors that enabled us to evaluate end-to-end test cases leveraging actual jobs being run in the production system. This led to us to finding the bugs that really mattered – those that affect real users. We were even able to prioritize our test scenarios based on the revenue impact of the user workflows and on the current pain points experienced by those users. This is an advantage of having an internal customer, but with good monitoring you can also approach this level of insight with external customers too.

uTest: We’d be remiss if we didn’t ask you about the recent launch of Microsoft Surface. First off, did you know about the project beforehand? If so, good job keeping it a secret. Secondly, what are some of the big testing challenges you’d expect to be associated with this project? The hardware? The Windows 8 OS? The touch functionality? We’d love to hear your thoughts on this HUGE Microsoft endeavor.

Seth: I found out about Surface when you found out about it, so obvious kudos to that team for keeping this secret so tightly wrapped.  It seems to me that re-using an existing product name was quite clever – it arouses no suspicion when you see it next to someone’s name in the internal address list.

Honestly hardware testing is almost the antithesis of the services testing where I have concentrated my career. In the latter you have rapid cadence and an ability to get things out there and experiment. With hardware you better get it right before launch time. I remember when I was QA Manager at Amazon and had an original Kindle before they were released. I got an email saying that Bill and Ted from the lab were going to come around to install new chips in all the test Kindles we had and thought to myself, that is just not going to scale after launch!

Getting back to the Surface, what excites me about this new product is also what poses the greatest challenges for testers… it’s all new. New hardware, utilizing new manufacturing; new operating system with a whole new interface; plus as you said the touch and gestures functionality. I have no doubts the teams involved will knock it out of the park, which will be a true testament to them considering those hurdles to overcome.

uTest: You’re a big proponent of Testing in Production (TiP) – an idea that surely scares the crap out of most test teams. How is TiP leveraged at Microsoft and what are the main advantages of this approach? And while you’re at it, what (if any) are the big disadvantages?

Seth: TiP is a methodology suitable for services where we (the engineers) can see what is happening with the service via monitoring, and we control the deployment. Testers hear “Testing in Production” and remember their worst nightmare dev bypassing test to throw cruddy code to users, but to be TiP it has to be a conscientious strategy aware of risks and utilizing mitigations. It’s not a secret that products like Bing, Amazon, Google, and Facebook launch experiments all the time exposing new code and features to a small number of users. This is a powerful way to get information on your product in real-world conditions that at the scale of these services simply cannot be reproduced in a test lab. TiP techniques often leverage the massive flow of data constantly emitted from our services. Hotmail measures actual performance experienced by users (anonymously of course) across all operating systems, browsers, and locations around the world to pinpoint trouble spots and optimize performance. In other cases we might be testing production itself. Our Office web services run validations after each deployment to check things like certificate validity, network health, and that all the bits got where they were supposed to be.

The advantage of TiP is you find things you just cannot find (or are prohibitively expensive to find) pre-production. The corollary to this is that you should not be finding bugs in production that you could have easily found earlier by unit testing or simple functional testing. It’s not necessarily a disadvantage, but based on your technical and business needs, some TiP methodologies might not be right for you.

uTest: In your current role, you focus on driving best practices in QA (among other areas) across the entire company. Is testing viewed drastically different in some parts of the company than others? And how difficult is it to get a company as diverse as Microsoft to adopt similar best practices?

Seth: In Engineering Excellence (my group) we never try to get all teams to adopt similar patterns. Instead we identify what works for teams at Microsoft and in the industry, and make sure other teams here have access to that information. I have worked with many, many great teams across Microsoft who are hungry for this knowledge. Being able to help those teams makes this a very satisfying role for me. Microsoft values data-driven decision making and I expect individuals on teams to be skeptical, so I better have good examples of what has worked and why.

Testing across Microsoft definitely has some distinctions from group to group. Some are cultural, but most often I think the product drives the test strategy. What you need to do for a service, boxed product, or hardware are necessarily different. Even among services how you treat a consumer product like Bing will be different than what you do for an enterprise product like Office 365.

uTest: What’s the best part about testing at Microsoft?

Seth: Testing is a respected profession at Microsoft.  We have many principal level individual contributor testers (for comparison at Amazon when I left there were none).  We have even more high level test managers including three VPs in test.  We have several test communities with their own events and speakers.  At Microsoft testers can really grow their careers and learn the latest technologies. Career-wise every tester who can, should do a gig at Microsoft.

uTest: As one of the more prolific testing writers out there, we’re interested to hear your thoughts on the next test trend. In your opinion, what’s the biggest testing trend that no one is talking about yet?

Seth: It’s more of an evolution of what I have already been talking about around testing services and Testing in Production. I call it TestOps.

Testers need to move out of the mindset of writing tests – running tests – evaluating results. Instead of using test results from a daily run as your quality signal, use the big data pipe coming off of your product (generally this refers to services) as your quality signal. This includes system data like CPU, API requests, system response time as well as (properly anonymized) user data. It also includes data emitted from synthetic transactions that you can run constantly in production. These are indeed test cases, but instead of getting just a daily red/green, you get constant availability and performance. This is the technology, but it also necessarily will change our software engineering organizations. The questions of roles and specialization versus generalization will need to be answered to meet each organization’s needs. And the emergence of the Data Scientist as part of the engineering team is an exciting outcome of the TestOps approach.

uTest: Which QA method do you prefer and why: Agile, Waterfall or other?

Seth: Unsurprisingly my response is… first show me examples of teams that have done each of these well and not-so-well, then show me the system under consideratio, and then I can give you an answer.

uTest: True or false: It’s better to have testers who are evangelists for their product, than testers who are evangelists for their department.

Seth: It’s best to connect with what you are working on and evangelize that. If you feel more connected to the product, then go for it… same for your department.

uTest: Fill in the blank: The most rewarding testing project I ever worked on was _____.

Seth: Version 1.0 of the first product from that aforementioned startup. I can’t even remember its name now! But as well as a stint at Microsoft, I also recommend one at a small, exciting startup for all testers that can do it.

uTest: You’re one of the more active participants on the test speaker circuit. Care to give our readers a head-ups on your next appearance, as well as what you plan to cover?

Seth: STPCon has been kind enough to ask me back to give an encore of A-Z Testing in Production for the October 2012 conference. If you missed it the first time come on by. If you already saw it and are still hungry for TiP, I will be adding some new things and putting more emphasis on big-data, so come by and let me know what questions you may have come up with since seeing the first TiP talk in March.

uTest: What’s Seth Eliot doing when he’s not improving the state of software testing?

Seth: Since I live the Pacific Northwest my answer should be scaling mountains, biking trails, and kayaking Lake Washington, but nah…I prefer to spend Saturday morning taking my girls (4 years old and 18 months old) to Starbucks for vanilla milk and cake pops.  This also has the advantage of giving my wonderful wife some break time as thanks for letting me jet all over the country to teach and give talks.

Editor’s note: We hope you enjoyed our latest Testing the Limits interview. If you have a suggestion for future interview guests, send them along to blog@utest.com.

Essential Guide to Mobile App Testing

Comments

  1. says

    This is really interesting, You’re a very skilled blogger.
    I’ve joined your feed and look forward to seeking more
    of your fantastic post. Also, I have shared your site in my social networks!

  2. says

    Hey there just wanted to give you a brief heads up and let you know
    a few of the images aren’t loading correctly. I’m not sure why but I think its a
    linking issue. I’ve tried it in two different browsers and both show the same outcome.

  3. says

    adsense_ad_unit += ” +. Basically the salads are healthy with low calories most
    commonly found on every dining table in US. Sprinkle white pepper, curry and a touch spicy seasoning onto the paper plate.

  4. says

    Hi there I am so glad I found your blog page, I really found
    you by mistake, while I was looking on Aol for something else,
    Anyways I am here now and would just like to say many thanks for
    a tremendous post and a all round interesting blog (I also love the theme/design), I don’t have time to look over it all at the moment but I have saved it and also added your RSS feeds, so when I have time I will be back to read much more, Please do keep up the superb work.

  5. says

    I believe that is one of the so much important information for me.
    And i am satisfied reading your article. However want to remark on some basic things, The web site style is perfect, the articles is truly great : D.
    Excellent process, cheers

  6. says

    Howdy just wanted to give you a quick heads up. The words in your content seem to be running off the screen in Internet explorer.
    I’m not sure if this is a formatting issue or something to do with browser compatibility but I thought I’d post to let you know.
    The design look great though! Hope you get the problem solved
    soon. Kudos

  7. says

    Hello there! This is kind of off topic but I need some
    advice from an established blog. Is it difficult to set up your own blog?
    I’m not very techincal but I can figure things out pretty fast. I’m
    thinking about setting up my own but I’m not sure where to begin. Do you have any tips or suggestions? Thanks

  8. says

    When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I get three e-mails with the same comment.
    Is there any way you can remove me from that service?
    Thank you!

Trackbacks

  1. [...] Our Testing in Production Expert, Seth Eliot, believes that TestOps is one of the future paths for our tests.  You look at this link for related information. I think Testing in Production can really change the way of how we do testing and the future of tester’s career.    Here is one section in  the interview about TestOps [...]

Leave a Reply

Your email address will not be published. Required fields are marked *