Software Testing Blog

Whether you’re a crowdsourcing critic or devotee, it’s worth hearing all the angles from the experts and from those who have built crowdsourcing business models. Check out the Top 20 Crowdsourcing Tweeps (experts and companies) to follow on Twitter below (in no particular order). Also, here are a few recent articles to hopefully spark even more debate — The Huffington Post’s Does Crowdsourcing Threaten Your Job (or Offer New Opportunity)?, Entrepreneur Magazine’s Crowdsourcing: Opportunity or Time Suck?, and Network World’s Could designer bras be a natural fit for crowdsourcing?.

1. Jeff Howe, @crowdsourcing (coined term crowdsourcing, quiet this summer)
2. LiveOps, @liveops
3. Neil Robertson, @nielr1 (co: @Trada)
4. John Winsor, @jtwinsor (co: @VictorsnSpoils)
5. Ross Kimbarovsky, @rosskimbarovsky (co: @crowdspring)
6. Dwayne Spradlin, @InnoCentiveCEO (co: @innocentive)
7. Tim Thomas, @imstarboard (co: @localmotors)
8. Community Roundtable, @jimstorer & @rhappe (co: @TheCR)
9. 99 Designs, @99designs
10. CrowdFlower, @crowdflower
11. uTest, @uTest (shameless self plug )
12. Chokha, @chokha
13. Mike Martoccia, @mmartoccia
14. Top Coder, @TCJim (co: @_TopCoder_)
15. Threadless, @threadless
16. Chaordix, @chaordix
17. Peter Lamotte, @peterlamotte
18. Genius Rocket, @GeniusRocket
19. SmartSheet, @crowdwork & @mcolacurcio
20. Tongal, @tongal

Update! Apologies for those I may have missed. Here are more recommendations straight from the community: @crowdsourcecap, @ArticleOne, @crowdsourcerisk, @fergusdyersmith, @OpenRunway, @crowdsourcery…

More can be found on our actual Crowdsourcing Twitter list. So, who do you follow on Twitter to find the best crowdsourcing advice and breaking news? Did I miss anyone? Please let us know in the comments and I’ll add them to the list.

Not to beat a dead horse or anything, but I wanted to briefly revisit Apple’s  “Antenna-Gate” fiasco to drive home a very important lesson for companies of all shapes and sizes: Rely too heavily on “lab-testing” and you are virtually guaranteed to get burned.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

Read more…

Last week, we posted a pretty powerful piece about the loyalty of crowds for all those crowdsourcing veterans out there — both skeptics and champions alike — to start demystifying the “human nature” of online communities. We received some really great feedback (thank you!), and realized that many people still haven’t heard the basics of the crowdsourcing model. For those of you who just need the quick nuts and bolts, here’s a pretty good interview on CBS NEWS that may be useful: What Is Crowdsourcing?

Watch CBS News Videos Online

Depending upon who you ask, crowdsourcing is either evil, revolutionary, or a next gen of internships.

But one thing that ALL crowdsourcing companies like to preach is how loyal and trustworthy and professional their community is. I know because I’ve read it in 100 different sites. Hell, I’ve written it a 100 different times here at uTest. So why do crowdsourcing companies insist upon telling the world how loyal and earnest their community is?  Maybe it’s to assuage the fears of prospective customers about entrusting their logo design, app development, content production or marketing to a community of strangers. Maybe it’s because if marketers say it enough times, we hope it’ll come true.

The more pessimistic view is that people — cloaked in the anonymity of the web — often act in greedy, selfish, mean-spirited ways (this perspective didn’t make it into the crowdsourcing brochure, by the way). Such dark behavior is well-documented and takes the form of flame wars on message boards, bullying via social media and online fraud.

So which is it — are people good-natured and honest?  Or are they money-hungry malcontents who will do anything to get ahead, as long as they don’t get caught?  Obviously it depends on the people, but I learned the truth about our community this week — and it was a lesson we learned the hard way.

A little background:  At uTest, we pay our testers twice per month via PayPal or Payoneer. And at this point in our growth, each pay cycle involves a non-trivial amount of cash — pretty deep into the five-figure range.  Now, it’s not easy or flattering to admit this, but in our most recent pay cycle, we experienced a glitch that caused us to pay our testers twice. That’s right folks, it was double payday here at uTest!

Read more…

Did you ever have that teacher who would give up on Friday afternoon and let the class watch a movie? We didn’t, hence your weekend reading assignment. What else are you gonna do while you wait in line for the iPad? That’s what I thought.

So, here are some testing/crowdsourcing articles to help pass the time. There will be a multiple choice exam on Monday. Or maybe we’ll just watch Ferris Bueller’s Day Off instead. Be prepared either way.

Serve Your Stakeholder
“Dr. Cem Kaner describes stakeholder as someone who has a vested interest in the success of the testing effort and/or in the success of the product. Stakeholders could be project managers, marketing managers, product managers, programmers, technical support representative, sales representatives, customers and many others in many different roles. Testing is done on behalf of the stakeholders. Here is a question for you. Have you ever asked who your stakeholder is before you start to test?” (From Parimala Shankaraiah’s latest post on curioustester.blogspot.com)

CNN Believes Whatever Computers Say
“Note to CNN: Information retrieved from an onboard computer is not reliable in cases where the computer itself is a suspect in the crime. No one is claiming that ghosts are causing the cars to go out of control. And there is no evidence yet that a mechanical failure is the culprit, despite the NHTSA looking hard for that evidence. Alcohol is also not a factor in these specific cases (or else we wouldn’t even be talking about them). That leaves two big possibilities: sober experienced people are suddenly forgetting how to drive OR something is wrong with the software or electronics that controls the cars.” (From James Bach’s blog on Satisfice.com)

Read more…

It’s back to the basics for the US Census. After several failed attempts to “modernize” the national headcount with hand held computers, officials are now blaming the resulting confusion, delays and costs on – wait for it – software bugs. You can read more about it here. To save you some time, I’ve summarized the progress made along the way with some telling quotations from those in the know:

Phase I: “Critical software errors are increasing; system performance is still lagging and testing continues to be compressed. A shortfall in testing portends potentially significant technical problems in the field.” – John Thompson, former associate director of the Census Bureau

Phase II: “The testing of the system is continuing to reveal critical defects.” – Todd Zinser, Commerce Department Inspector General

Phase III: “The performance of this system is not taking the load we’d like. It’s not going to accept the load that we will need to get it in about a month or so.” – Robert Groves, Census Director

Results: “They have prepared to do the follow-up using pencil and paper, and they’re very good at that.” – John Thompson (again)

Read more…

Well, today he’s sticking close to home in Boston. Tomorrow he’ll land in London… and before the week is out, he’ll hit Tel Aviv.

Doron starts Wednesday morning off (after his usual 10-mile run, of course!) in London with some tea and networking with friend and colleague, James Whittaker and UK partner, TCL.

Then he’s off to QCon London, an excellent conference for the enterprise software community. On Friday, 3/12 @ 2pm, he’ll be presenting at QCon re: The Mobile App Quality Challenge & How Crowdsourcing Can Help.

Doron is one of five software testing leaders chosen to present in the “How Do You Test That?” track. This track explores unique solutions created to address situations in which automated testing does not suffice.

And on the last leg of his marathon journey, Doron will present at Garage Geeks in Israel on Monday, 3/15 @ 8pm. There, Doron will be taking a deep dive into the topic of Crowdsourcing, and how smart recruiting, training and incentives can turn an unstructured, loosely assembled mob into a unified, professional community.

So, where in the world is Doron this week?  Catch him if you can!

VentureBeat has an interesting article about eBay’s announcement that they’re going to tap into their user base to test new features — a kind of opt-in, ongoing beta program for new features.  The title for this article:

eBay to Use Crowdsourcing to Test New Features, Starting with Streamlined Search

Those who know me well know that defending the purity of the term”crowdsourcing”  against misuse is a pet cause of mine (e.g. – Meet-ups are not crowdsourcing; online polls are not crowdsourcing; asking your Twitter followers a question is not crowdsourcing). But don’t worry… this won’t be another rant about the importance of definitions and how critical labels are.  Well, at least not about the word “crowdsourcing”.

Read more…

There’s no shame in being a couch potato (every now and again). After all,the competition between the major TV networks has been extremely entertaining. But now is the time to get up off the couch and onto your desktops, laptops and smartphones – it’s time for the very first Bug Battle of 2010!

That’s right, we’re pitting the four major TV networks against each other in this quarter’s bug-hunting competition. And while there are multiple sources to find out which network has the highest rated shows, we’re going to compare their web and mobile apps to see who distributes that content best.  Here are the networks, in alphabetical order: ABC, CBS, FOX, and NBC.

As an added twist, we’re introducing a mobile testing component to the Bug Battle. For more details on this – as well as info on rules and prizes – check out our Battle of the TV Networks page.

It’s now time to pull back the curtain to determine which network sites have the most bugs, the most critical bugs, and the best usability and feature set. And so uTesters, best of luck to you all as you begin another exciting bug hunting mission. May the best network win and may the top testers prevail. Good luck!

Special note: This competition is available only to members of the uTest community.  uTesters can get complete details and discuss the Bug Battle with their peers in the uTest Forums.

The major networks have been engaged in a decades-long struggle to win the hearts and minds (and eyes) of viewers. Whether it’s news, sports or sitcoms, these battles are now being fought on multiple fronts – including the distribution of their content through their web and mobile applications.

Which brings me to our latest bug-hunting competition. This quarter’s uTest Bug Battle will challenge testers to find bugs on the web AND mobile apps of the four major TV networks: CBS, NBC, ABC and FOX.

The competition gets started this Friday, February 5 at Noon (ET). At that time, testers from our community will be given one week to search these sites for the most compelling bugs, and to report them through our online platform. We’ll be dishing out nearly $4,000 in prize money for categories like Top Tester, Best Bug and Best Survey Feedback. As an added twist, we’re also including prizes for the Best Mobile Bugs.

More details on this Bug Battle – as well the rules, prizes and deadlines – after this commercial break. Just kidding, you can find them in the “Bug Battle” thread in the uTest Forums and we’ll be sharing more details here in the next few days.

So testers, you’ve got a few days to prepare for our most challenging competition to date. Until then, stay tuned.