Users Use; And Testers Test
VentureBeat has an interesting article about eBay’s announcement that they’re going to tap into their user base to test new features — a kind of opt-in, ongoing beta program for new features. The title for this article:
eBay to Use Crowdsourcing to Test New Features, Starting with Streamlined Search
Those who know me well know that defending the purity of the term”crowdsourcing” against misuse is a pet cause of mine (e.g. – Meet-ups are not crowdsourcing; online polls are not crowdsourcing; asking your Twitter followers a question is not crowdsourcing). But don’t worry… this won’t be another rant about the importance of definitions and how critical labels are. Well, at least not about the word “crowdsourcing”.
My problem with this piece is the flagrant misuse of the word “test”. Don’t get me wrong — I think what eBay is doing is great. Beta programs like this can be highly useful, as proven by Google Labs. But it is not testing. As the author states about eBay’s program:
It allows the company to see how users respond to new ideas, and gives intrepid users new toys to play with.
Does this sound like the kind of testing that you do? “Playing with” an app… doesn’t sound like testing to me. Users attempt to get an app to work and then maybe provide like/dislike feedback. Testers structure their use to isolate variables and systemically probe for weaknesses and defects. I guarantee that the author or the eBay spokesperson didn’t mean any offense or slight to professional testers, but this illustrates how far we, as an industry, have to go in clearly defining and upholding the discipline of testing.
At the end of the day, users use; and testers test. What do you think — how do we as an industry differentiate between beta user programs and professional software testing?
A large difference to me in terms of having users ‘use’ a beta product and having testers test it is the quality of reports.
My feeling is that with users reporting issues, you’ll generally get “This doesnt work” or maybe “I got an error doing this” or even the ever helpful “This site sucks!”.
With an experienced tester you will get details. You will get detailed steps for reproducing the bug. You will get someone who actually tries more than once to reproduce the issue. You will get someone who likely does additional analysis to further ‘fence in’ the issue like: trying other browsers, other OS’s, trying steps that are slightly different. If you have a tester with a technical background you will likely get a clearer, more specific, detailed bug report in a language developers can understand and you’ll get a tester who has further insight into the ‘fencing’ process I mentioned (e.g. maybe they’ll test back end network connectivity or ping a server or test connectivity with another app).
There’s a world of difference between the average user and a professional tester.
And dont get me started on the definition of crowdsourcing. Crowdsourcing goes far beyond simply opening the flood gates and having a large number of people use a product.
If anyone wants a proven definition of crowdsourcing, just ask someone at uTest because they know how to do it.
- Brad
Well said, Brad. The quality of the bug reports (and their ability to be reproduced) is one major difference between users and testers.
And thanks for the kind words re: crowdsourcing. We’re certainly trying and we’re getting better at it every day.
-Matt