Testing the Limits With eBay’s Jon Bach – Part II

In part II of our Testing the Limits interview with Jon Bach, we get his thoughts on responding to change in the testing world; what his brother James Bach has been up to; his criteria for hiring testers at eBay; mobile challenges; searching for defective pocket change and more. If you missed the first session, you can read it here.

uTest: It looks like eBay wasn’t able to keep you off the testing speaker circuit (woohoo!). In fact, you were at STPcon earlier this month – care to give our readers who couldn’t make it a summary of what you covered?

JB: Two things: A workshop with Dan Downing of Mentora, who approached me at the last WOPR (held at eBay in November) and had a cool idea to bring a little slice of WOPR to STP.  It’s for anyone who needs to build a game plan for performance testing.  He called it “Arming Yourself for Performance Testing: War Stories from the Trenches” — http://www.stpcon.com/Item/1032/.

I also spoke about an idea that I’ve been experimenting with after James came back from a business trip and talked about how to respond to project change and chaos: http://www.stpcon.com/Session/13/My-Crazy-Plan-for-Responding-to-Change

uTest: Speaking of James, he has been doing some interesting things the past year as well. What’s the latest testing topic of conversation among the Bach brothers? And did he have any words of advice for you in starting your new job?

JB: He came to eBay and spent a week with me.  I sat him in the cube next to mine and he did some testing from outside the firewall on the guest wireless. I gave him a charter and he executed it beautifully.  The secret about James is, he’s really friendly and service-minded if you’ve managed to win his respect.

We talked CAST 2011 (I’m conference president, he’s my program chair); we talked about new tester games; we shot a new CAST promo video; we talked about Egyptian democracy and systems thinking (how it affects the price of gas).  But just when we were in the thick of testing eBay site page compatibility with IE 9.0, the Japan quake hit and we took time to watch the footage with the rest of the world.  Then we did impromptu research and found out more on nuclear plant  meltdowns, which led to being curious about microseiverts, which led to an article about Byzantine failures.

About eBay, he gave me no advice per se, just ideas for tactics.  He offered some free consulting, which he gave, then said, “I’m proud of you, man. Kick ass.”

uTest: Part of your new role at eBay will be to hire and recruit a top-flight team of testers (in addition to the ones already there). What sort of traits/skills/attributes will you be looking for in particular?

JB: The ability to come up with ideas – either old or new – and execute them in a way that helps us improve notions of Search.  For years, I used the triangle program in test auditions.  Now I use something more simple.  I draw a long horizontal rectangle on the whiteboard with a little “Submit” button below that.  I say “this is a text input field for Search, just like the one you see on the eBay site. Help me create a test plan for it.”  I’m hoping that instead of an interview, it comes across more like an invitation to a real collaboration.

Continue Reading

Essential Guide to Mobile App Testing

Google’s Whirlwind Start To 2011

Google’s off to a pretty intense start in 2011 – from a change in CEO to launching new products that compete directly with some of the biggest tech companies including Microsoft, Amazon and of course, Apple.

It’s no secret that web and mobile apps represent a lot of money to businesses and app markets are in a race to keep up. Google is using this as an opportunity to greatly expand their presence — and the early returns are impressive. In fact, the Android app market is growing 3x faster than Apple’s iOS market (although, as its marketshare grows, it become a more attractive target to black hat malware apps).

Google isn’t stopping there, though. They’ve recently launched their Shopper app on iOS – an alternative to Amazon’s really nice native apps – and the “One Pass” a publisher subscription alternative to Apple.

Continue Reading

Essential Guide to Mobile App Testing

Testing the Limits With Matt Evans from @Mozilla – Part I

What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.


uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.
Continue Reading

Essential Guide to Mobile App Testing

2010 Word of The Year: Privacy

I recently attended a marketing conference that discussed emerging technology trends.  When the panel was asked what was the single-word topic of 2010 they almost all said, “mobile”.  I didn’t think of it at the time but I’d argue that the word of the year is “privacy”.  That thought, coupled with a current email-based discussion I’m having with a luddite friend (he’s not on Facebook or LinkedIn), got me thinking about some of the privacy issues that we — as a global population of netizens — will face in 2011 and beyond.

Concern about privacy is hardly a new topic.  Back in 1999 Scott McNealy, then the CEO of Sun Microsystems notoriously said, “you have zero privacy.  Get over it.”  I love the brevity, Scott, but that is not going to get you on a Hallmark card anytime soon.  Yes, the web brought on a change in the level of privacy that users may expect, but the role of marketing has always been to predict the intent of potential customers by tracking user behavior.  Computers and the internet, however, have yielded a seismic shift in the cost, speed, availability and sheer amount of data – perhaps changing at a rate faster than humans can conceptually deal with, and thus creating debates about how to strike a balance in this brave new world.

In 2010, however, we’ve seen more information about the reconciliation of online and offline data. From cars, to finances, to the recent announcements about the TSA’s new full-body scanners, it’s no longer just our web browsing history that’s available to evil marketers like myself.  Here’s a quick rundown of a few privacy issues, how they can be exploited, and what you should know about protecting yourself:

Continue Reading

Essential Guide to Mobile App Testing

Five Server Load Tips

Last week, a company called Skyfire launched an app that made it possible to watch Flash movies on an iPhone or iPad. Their app uses servers hosted in the cloud to convert Flash videos to HTML5, making them viewable on iOS devices where Flash is not available.

Now we’ve heard before that nobody wants Flash on their phones, so you would understand that Skyfire had a slow first day after their launch. And by slow first day I mean their servers were so overloaded by the demand that they couldn’t provision them fast enough. Eventually they had to pull their app from the app store and declare it “sold out.”

We here at uTest think that among the things in this world that could be called “problems,” having so much demand that your servers can’t keep up is definitely a good problem to have.  Skyfire did, after all, make over $1 million in their first weekend on the app store.

On the other hand, nobody wants their servers to crash because of excessive load. The trouble is that properly planning for heavy load is often something that can vary from one application to another. (The load planning for Skyfire is probably very different from Facebook.) Still, there are some very simple load planning tips that almost everyone should know:

Continue Reading

Essential Guide to Mobile App Testing