To Crowdsourcing Friends, Foes & Fanatics: Just How Loyal Is Your Community?

Depending upon who you ask, crowdsourcing is either evil, revolutionary, or a next gen of internships.

But one thing that ALL crowdsourcing companies like to preach is how loyal and trustworthy and professional their community is. I know because I’ve read it in 100 different sites. Hell, I’ve written it a 100 different times here at uTest. So why do crowdsourcing companies insist upon telling the world how loyal and earnest their community is?  Maybe it’s to assuage the fears of prospective customers about entrusting their logo design, app development, content production or marketing to a community of strangers. Maybe it’s because if marketers say it enough times, we hope it’ll come true.

The more pessimistic view is that people — cloaked in the anonymity of the web — often act in greedy, selfish, mean-spirited ways (this perspective didn’t make it into the crowdsourcing brochure, by the way). Such dark behavior is well-documented and takes the form of flame wars on message boards, bullying via social media and online fraud.

So which is it — are people good-natured and honest?  Or are they money-hungry malcontents who will do anything to get ahead, as long as they don’t get caught?  Obviously it depends on the people, but I learned the truth about our community this week — and it was a lesson we learned the hard way.

A little background:  At uTest, we pay our testers twice per month via PayPal or Payoneer. And at this point in our growth, each pay cycle involves a non-trivial amount of cash — pretty deep into the five-figure range.  Now, it’s not easy or flattering to admit this, but in our most recent pay cycle, we experienced a glitch that caused us to pay our testers twice. That’s right folks, it was double payday here at uTest!

We discovered the error moments later when several testers emailed us to point it out to us. We immediately split our efforts up into two camps:  one group trying to identify the issue and make sure it never happened again; the other trying to figure out how to get our money back.  I was in the second group.

After exhausting our options with our payment vendors (who were responsive and sympathetic, but unable to help), we came to the realization that we would have to simply explain to our testers what happened and ask them to refund the duplicated payments via a not-so-simple five-step process. Now, I love our community and I think the world of them, but let’s just say I was skeptical. I mean, c’mon, it’s our mistake and it’s free money.

So what happened?

  • At 2am on Sunday morning, we sent out an email to all the affected testers… and hoped for the best
  • Within 24 hours, more than two-thirds of the money had been refunded
  • Within 72 hours, more than 90% was refunded
  • And within 96 hours, more than 98% had made it’s way home
  • I expect that within a week, less than 0.5% will remain outstanding
  • We’re not talking about a few dollars here and there… one U.S. tester refunded nearly $2,000. Several others from the U.S., Canada and Western Europe issued refunds in the $1,500 range. We had testers in developing nations who refunded nearly $1,000 — in places where $1,000 is more than one month’s salary for a QA engineer.

So in addition to proactively bringing this gaffe to our attention, our community went out of its way to refund this money promptly and in full. Sure, I’m still pissed and embarrassed that it happened. But it did, and the takeaways have been surprising and worthwhile:

  1. Communities beat crowds. Crowdsourcing companies should stop trying to build crowds or mobs. Instead, invest in building a community. We pour a lot of time, energy and money into our community. From our tester forums to free training programs to Bug Battle competitions to town hall meetings to our uTester of the year awards.  And we don’t do it because it’s always fun or easy or even public. We do it because we believe that communities are better than crowds or unruly mobs.
  2. Surprise, surprise, surprise. Even after nearly 7 years of building community-driven businesses, I can still be genuinely surprised by the loyalty and integrity of a large group of people who have never met one another in person.  I wonder how else our community would surprise me if I keep a more open mind.
  3. Deep, trusting relationships are a necessity.  We treat the men and women in our community like respected professionals. We work hard to give them a lot of different ways to engage with us on their own terms. And they have always treated us well in return. But when push came to shove and there were real dollars on the line, the true impact of the relationships between us and our community became abundantly clear.
  4. My new favorite story. The next time a prospective customer asks me “who are these 25,000 testers and how do I know I can trust them with my super-secret app?”, I know exactly what story I’ll tell them in response.

Now, my fellow crowdsourcing enthusiasts (on both sides of the aisle), let me ask you this:  when push comes to shove, just how loyal is your community?  While I don’t recommend this particular path as a way of proving it, I would love to hear other stories about what happens when a community is transparent and accountable to one another. Or, if you have them, tell us your nightmarish tales of community deception and betrayal.

Essential Guide to Mobile App Testing


  1. says

    Glad you liked it, Ellie. We were pretty inspired by it internally as well, and felt that others deserved to hear this story — even though it began with egg on our face. :-)

    We hear enough negatives of online (mis)behavior, so it was nice to share some positive.


  2. says

    THAT is an amazing story. First that you were willing to tell it publically, second to read of the loyalty and integrity of such a large mass of people.

    It is really a testament of how much your people believe and trust in uTest.

    It did my heart good to read such an amazing story.
    Thank you for sharing it!

  3. says

    Fantastic story Matt – it sure feels good when you have the support of your community.

    Well Done.


  4. says

    Thanks, Andrea & Mike. Appreciate hearing from pioneers in the crowdsourcing space. As you know well, building a community is like a marathon (long, painful and ultimately, rewarding)! :-)

    Thanks, Niel. We always enjoy chatting with you and the Trada crew.

    And I agree with you that the bar is being raised quickly and dramatically. When building a crowdsourcing / community-driven business, simply having the good idea is no longer enough. Short-term thinking will no longer suffice… mobs need not apply!

  5. says

    Awesome post (very self effacing) and great proof that:

    a) You’ve done a great job building loyalty in your community

    b) Crowdsourcing is not just a mob of people out for a buck

    c) The whole industry is growing up. Professionals are in the crowd now and when they need to, they act like professionals.

    We’re learning lessons from you guys every day at Trada. Keep it up!

    CEO, Trada (, @nielr1)

  6. says

    Hey Matt,

    Nice story! We had a very similar incident when we relaunched last December and our payment reports blew up on us. We accidently paid out a rather large sum to a sizable number of Creatives on the site and were able to recover it in a similar time frame. The combination of community loyalty, and their “fear” of not being able to continue earning on the site proved to be a powerful combo.



  7. says

    Thanks for sharing this gripping story, Matt! It’s gratifying to hear the happy ending and the fact that your community is ready to work with you for the long haul, not a one-off quick-win. That’s a huge testament to the value of community-building. And I look forward to sharing your new favorite story when the occasion arises!


Leave a Reply

Your email address will not be published. Required fields are marked *