Crowdsource Testing your Application

Check out our guest post on 47 Hats, a great blog devoted to Micro ISVs.  We talk about some of the reasons you might want to do crowdsourced software testing:

Thousands of people want to help you make your product better, and they’re in places around the world both near and far. Some of them may live down the street and some of them in places on the other side of the planet. They’re all part of a global crowd, ready to pitch in and contribute their knowledge and skills. The Internet makes it possible for you to connect with them, share your product, and learn from their experiences.

Crowdsourced software testing gives you access to a number of benefits:

  • Access to diverse platforms, languages, and people
  • Real insights from the real world, not just made up test case results
  • Testing done by hundreds of people at the same time
  • Rapid feedback right away

Keep reading our post about Crowdsource Testing your Application.

Essential Guide to Mobile App Testing


  1. Joy Young says

    Thanks Matt, I’d love to find out some more info on this. Is there a number I can reach you at, as I have a bad wrist making typing harder some days? Really appreciate all the candor and disclosure. So many people out there getting taken advantage of by hackers and others. However, it’s amazing to read all the great things you all do.

    Also, thanks Derrick. A friend I really respect named Uri, from Hadera, who recently sold his “IaaS” cloud company said pretty much the same. He said he had a lot of cloud sourced testing experience. Explained that traditional apps from big companies that can afford to protect and litigate generally have no issue beyond their normal exposure. Said they fit well with it and many other traditional co’s. But, explained that smaller firms, especially Web/mobile that have a lot riding on one or two things have to be very careful. I only know that it sounds like we should break-up and protect things first. We plan to file more legal protection, but that does not seem to be the value it use to. It’s so sad that there are so many hackers and other unscrupulous people out there.

  2. Derrick Fludd says

    Joy, I agree with Matt said. Who you use for this type of testing is critically important to reduce risks. Also, from the experience of people I work with uTest is the best at what they do. We have pushed clients to work with them. With that said, you still have to be VERY careful to not expose sensitive unprotected or exposed workflow processes and applications. This means anything of value that can be readily reverse engineered. There are ways to reduce the risk though.
    I am especially paranoid since I have worked as what some people call a white hat computer security expert. I work with a group who focus mostly on government sec. I have also worked with a wide range of corporation. That includes a few of the companies that Matt mentioned. I try to expose specific vulnerabilities before others. I also make best practices recommendations to reduce general risk exposure. Related to open internet testing, crowd source or others, many things are safely testable that way. That is, if you use a top testing company like uTest and take the right precautions. There are also many things that should not be tested in the wild, at least without extra planning and effort. That includes breaking-up exposed workflows and adhering to the appropriate security protection practices.
    This is even more important for a big exposed highly targeted company. This includes testing in high visibility environments.
    I have pointed many people to uTest and other types of testing companies, who have had great experiences. I have also tracked security and other factors for testing in progress and have seen significantly more activity around high value testing targets and companies.
    To summarize: do not throw things that you don’t have well protected out into the wild. Protection means patents and copyrights on top of solid code and workflow protection on every level. From our tracking and other experiences we learned to break high value exposed software up into pieces for testing wherever possible. In fact, government clients typically mandate this for development and for testing. This is important for several reasons. First, countries like China do not care at all about patents or any other ‘paper protection’. Second, you should always try to minimize your exposure to hackers, who have learned to target such environments. Third, if you need to expose your application for testing or other reasons you should always minimize the perceived value of your ‘honey pot’, because the likelihood that hackers will take the time to acquire and reverse engineer your application or workflow for sale goes down based on how much time and effort it takes to do this. Over years of risk mitigation work we found that the reduction in risk is over 80 percent by these methods.
    So use common sense and realize that the amount of protection should always be related to the value of the software to you, and the estimated perceived value to hackers and others. Finally, when you are ready, only use the best like uTest. I do not work for uTest. i have just seen the tragic result of others who do not use the right companies and security practices and I like to see others avoid those problems.

  3. says

    Hi Joy,

    Full disclosure: I’m the CMO of uTest and you can reach me at matt [at] utest [dot]com if you’d like to discuss further.

    The short answer is that you need to work with better, more accomplished crowdsourcing companies (not all firms are created equal).

    At uTest, we require every tester to sign an NDA with us when they join our community, which states that they cannot talk about our customers, their products or specific defects on Facebook, Twitter, personal blogs or anywhere outside the confines of the private uTest platform. And they certainly cannot reverse engineer our customers’ IP. Beyond that, our customers can upload an NDA which testers must sign before viewing your testing project (this puts legal paperwork between you and the individual tester).

    Also, for our security testing projects, we use a very carefully screened list of white hat engineers with whom we have a long professional relationship. Until we get to know someone well, they are not invited to security-related projects.

    Ultimately, the proof is in the results. uTest has conducted well over 10,000 test cycles for companies like Google, Microsoft, Netflix and BBC (sample list is available at — most of them testing pre-production releases. And we’ve never had any type of IP or security breach. Those companies trust uTest because we preach and practice professional, white hat testing ethics.

    Those results aren’t dumb luck; it’s what comes when a company knows how to build a professional crowdsourcing model and serves its community as well as it serves its customers. We respect and value our testers (and pay them well for their performance). And in turn, they respect what uTest is doing and how we’re trying to help testers and customers alike. In fact, they take a great deal of ownership for the professionalism of the uTest community.

    I hope that helps answer your question. I’d be happy to discuss further and prove to you that crowdsourcing — when executed properly — is a powerful tool for security testing (or functional, load or usability testing, for that matter).


  4. Joy Young says

    We had a couple big security problems in the past that we wanted to get your input on. We have done crowd source testing several times over the past few years, both on our own and using a company that specialized in it open crowd source testing.
    We found a coupe of our web technology reverse engineered on two occasions. So, we employed a research firm that found the trail and source of the initial thefts. They found that our technologies got reverse engineered as a result of hackers watching some of the crowd source testing groups. In each instance these hackers were not part of the testing company. They were just hackers doing what hackers do, and that sold the result to Chinese companies. So, what can be done to stop this from happening.

  5. Robert says

    Man you guys are stubborn, I check you out back then and didn’t feel like being nagged by sales to get pricing. I’m pricing QA again and was trying to find out what the price ranges are by googling it. All I can find are people complaining about how you don’t publish pricing.

    Here is to hoping that the next time I am pricing QA services you have a more informative website. I am not willing to give you my I don’t like to have to go through a salesman to get pricing.

  6. says

    Jaan – you’re right that we don’t publish our pricing. it’s very difficult to price all of the details about a testing release without knowing more about your needs or product.

    What we do offer is a way to get a quick pricing quote here.

  7. Jagan says

    You can use different pricing models and publish the brief information of each model. You can ask users to contact contact sales team for more details and for customized pricing.


  8. says

    Your service looks promising, but without pricing information on your web site I cannot really make the decision whether to even apply for your demo or would it be a complete waste of time (why waste time analyzing the demo if in the end it turns out I can’t afford your service?). Please consider making available your price list online. If your service is really affordable, why hide that? Thanks.

  9. says


    I wrote a reply to your post at 47Hats (currently awaiting moderation), however I’ll reply here as well:

    You’re right, we don’t have pricing information on our website. We’re discussing this now and trying to decide if we want to have more details online, especially given our variable pay-per-bug model where prices reflect the number of bugs actually discovered and your budget. However, I really do appreciate your feedback. This is definitely an open issue for us.

  10. Peter says

    Sounds like great service (assuming you do your job right, of course). Too bad you insist on keeping your prices secret, I really have no patience for having to beg you to tell me your prices and then be pestered by your sales guys (who I was just forced to give my email to) forever.

Leave a Reply

Your email address will not be published. Required fields are marked *