Testing the Limits With Jim Sivak – Part II

In part II of our interview with testing expert Jim Sivak, we get his thoughts on hiring testers; waterfall vs. agile; a flexible SDLC; pointless automation testing; his time spent working on the Space Shuttle, landscaping at Disneyland and more. Enjoy! If you missed part I, you can find it here.


uTest: You’ve hired your fair share of software testers over the last 30+ years. What sorts of skills and attributes do you look for when filling out your roster?

JS: The best people I hire are the ones who have three key attributes—they love to learn, they constantly look to do things better (not satisfied with the status quo) and they love puzzles.  To be a great QA engineer, you have to continuously learn to make yourself and the team better.  The best teams continually try new approaches and processes.  And testing is solving puzzles—how do you make sure it works, how do you determine what can break and when it does, doing initial looks at what actually happened. I feel that the other skills—programming, product expertise, etc. can be learned on the job.

uTest: You’ve worked with dev teams that adhered to all sorts of different methodologies (agile, waterfall, etc.). From a testing perspective, which one of these is the most effective? Are there certain approaches that don’t mesh better with testing objectives than others?

JS: You can find success or failure with any of these methodologies and I don’t think that one is more effective than the others, if they are done correctly.  Each offers challenges to test teams and there is no single approach that works everywhere.  The key is to work within the software life cycle that you are presented with.  You can’t succeed in an Agile process using the techniques you learned on waterfall-based projects.  You have to be flexible and adapt to the SDLC.

From a testing perspective, I think the easiest is an SDLC that is more waterfall than anything else.  If the business supports it, having traceable requirements up front, lots of accurate documentation and formal test cycles is much easier than Scrums.  Agile is less forgiving as your status is given on a daily basis and there is little time to take corrective actions.

Effectiveness is really determined by customer satisfaction, regardless of the SDLC.

uTest: What’s the biggest mistake being made by test teams today?

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With eBay’s Jon Bach – Part II

In part II of our Testing the Limits interview with Jon Bach, we get his thoughts on responding to change in the testing world; what his brother James Bach has been up to; his criteria for hiring testers at eBay; mobile challenges; searching for defective pocket change and more. If you missed the first session, you can read it here.

uTest: It looks like eBay wasn’t able to keep you off the testing speaker circuit (woohoo!). In fact, you were at STPcon earlier this month – care to give our readers who couldn’t make it a summary of what you covered?

JB: Two things: A workshop with Dan Downing of Mentora, who approached me at the last WOPR (held at eBay in November) and had a cool idea to bring a little slice of WOPR to STP.  It’s for anyone who needs to build a game plan for performance testing.  He called it “Arming Yourself for Performance Testing: War Stories from the Trenches” — http://www.stpcon.com/Item/1032/.

I also spoke about an idea that I’ve been experimenting with after James came back from a business trip and talked about how to respond to project change and chaos: http://www.stpcon.com/Session/13/My-Crazy-Plan-for-Responding-to-Change

uTest: Speaking of James, he has been doing some interesting things the past year as well. What’s the latest testing topic of conversation among the Bach brothers? And did he have any words of advice for you in starting your new job?

JB: He came to eBay and spent a week with me.  I sat him in the cube next to mine and he did some testing from outside the firewall on the guest wireless. I gave him a charter and he executed it beautifully.  The secret about James is, he’s really friendly and service-minded if you’ve managed to win his respect.

We talked CAST 2011 (I’m conference president, he’s my program chair); we talked about new tester games; we shot a new CAST promo video; we talked about Egyptian democracy and systems thinking (how it affects the price of gas).  But just when we were in the thick of testing eBay site page compatibility with IE 9.0, the Japan quake hit and we took time to watch the footage with the rest of the world.  Then we did impromptu research and found out more on nuclear plant  meltdowns, which led to being curious about microseiverts, which led to an article about Byzantine failures.

About eBay, he gave me no advice per se, just ideas for tactics.  He offered some free consulting, which he gave, then said, “I’m proud of you, man. Kick ass.”

uTest: Part of your new role at eBay will be to hire and recruit a top-flight team of testers (in addition to the ones already there). What sort of traits/skills/attributes will you be looking for in particular?

JB: The ability to come up with ideas – either old or new – and execute them in a way that helps us improve notions of Search.  For years, I used the triangle program in test auditions.  Now I use something more simple.  I draw a long horizontal rectangle on the whiteboard with a little “Submit” button below that.  I say “this is a text input field for Search, just like the one you see on the eBay site. Help me create a test plan for it.”  I’m hoping that instead of an interview, it comes across more like an invitation to a real collaboration.

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With eBay’s Jon Bach – Part I

Sometimes, we interview a testing expert with so much wisdom, so much experience and so much to say, that one appearance simply won’t do. Jon Bach is one of those experts – and he’s back for his second Testing the Limits interview (Bach to the Future…anyone?). Jon recently joined eBay as the new QA Director for their Buyer Experience department. Prior to his new role, he had stints at Quardev Laboratories, Microsoft, Rational, LexisNexis, and Hewlett-Packard. Impressive track record, eh?

In part I of the interview, Jon discusses his new role at eBay; knowledge transfer in testing; war stories in software testing; how to test with no requirements and more. When you’re done here, go read Part II.


uTest: First off, congrats on your new job as the QA Director for eBay. I know you’re only a few weeks in, but what are you most looking forward to in this new role? And how is the transition going so far?

JB: I work for an aptly-named eBay department: “Buyer Experience: Search and Discovery”.  My team handles anything that has to do with what we call “Search Front End” – the UI elements rendered and functions that support searches for merchandise on eBay. eBay does more searches per day than Google, and I’m looking forward to finding search issues that affect millions of people every day.

As for “transition”, I have my laptop, badge, and a name tag for my office; bought myself an eBay hat, but the most important “transition” is one of mindset.  I used to think eBay was kind of like a pawn shop or a swap meet, but it’s got millions of new items from major manufacturers.  It’s a platform for worldwide ecommerce, whether it’s items sold by individuals or major retail outlets.

The most important transition for me is changing my own ecommerce habits.  Instead of going to Amazon to buy books, for example, it’s more useful for me as a tester and as an eBay employee to think “can I get it from eBay first?” Recently, I needed to order Valentine’s Day flowers for my wife.  I started going to proflowers.com when I thought, “can I get them through eBay?” Turns out that eBay options for flowers are not displayed very prominently on the site.  I finally found text ads for florists on the bottom of one of eBay’s search results pages, but I had to dig to find them.  This caused me to email my PMs – “why don’t we have a more prominent upsell for flowers, especially on the home page for V-day?” That mindset shift inspired me to actually test the main mission of my department in terms of my “buying experience.”

uTest: It seems that knowledge transfer must be especially difficult in the testing space. As someone going through the “on-ramping” process at a big company, with lots of moving parts, what advice you have for other testers and managers in the same situation?

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Elisabeth Hendrickson – Part I

We’re very excited to have Elisabeth Hendrickson as this month’s guest for Testing the Limits. Known throughout the blogosphere as Test Obsessed, Hendrickson is also the founder of Quality Tree Software, a California-based software consulting and training company. Her years of industry experience include stints as tester, developer, manager, and quality engineering director for a variety of companies. She’s also very active in the agile community, having served on the board of the Agile Alliance in 2007 and 2008.

In the first part of our interview, we get her thoughts on the dangers of fake agile; the progress of open-source test automation; the general strengths and weaknesses of today’s younger testers; career advice and more. Be sure to check back tomorrow for Part II. Enjoy!

uTest: You wrote a great blog post in which you referenced “the victims of fake Agile.” First off, have you considered starting an anonymous support group? If so, what are steps 1 and 2 in the 12-step program? Seriously though, what are some things testers and developers can do to avoid being victimized by fake agile at their companies?

EH: Grin. No, I don’t plan to start a support group. But I do intend to continue helping teams understand the difference between fake agile and real agile. Fake agile is dangerous: it removes all the external controls that made traditional practices work without instilling any of the team-centered disciplines that make agile work. The end result is usually worse quality, slower, and an organization that blames “agile” for the mess.

When I meet victims of fake agile, my first advice is to stop accepting the role of victim. No matter what our role in an organization, we have the power of influence. We can educate our colleagues and managers about the difference between fake and real agile. And we can help the key business decision makers understand that if they want the potential benefits of agile — capabilities delivered sooner at higher quality — they have to support the practices that provide those results and not just support “compress the schedule” and “change everything up to the last minute”.

uTest: You recently opened up Agilistry Studio, described as a practice space for agile software development. What’s the biggest benefit of learning agile practices on location with co-workers, as opposed to reading books, articles and websites by oneself? Is it a case of “seeing is believing”?

EH: As I write this, Rob Myers is leading a class in TDD for Java here at Agilistry. I’m eavesdropping on participants as they run through the exercises together. The amount of learning going on in the room is amazing. They’re learning at least as much from each other as from Rob. They’re having great conversations. And they’re able to apply the ideas right away. This is exactly what I intended when I opened Agilistry Studio. “Seeing is believing” is part of it. Collaborative learning and the opportunity to practice putting new ideas to use right away are other big pieces.

uTest: One last question on Agile: As a former member of the Agile alliance and other organizations, is there a risk of the term “agile” falling victim to groupthink? Or has that already happened?

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Matt Evans from @Mozilla – Part II

In part II of our Testing the Limits interview with Mozilla QA Director Matt Evans, we get his thoughts on mobile immaturity; the worst bug ever submitted by a Mozilla community member; the so-called “skills shortage” in Silicon Valley; skepticism for all things open-source; the next great browser innovation and more.

If you missed Part I, do yourself a favor and catch up here.


uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

Continue Reading

Essential Guide to Mobile App Testing