Software Testing Blog

House MD, one of the most popular television shows in the US and globally, is a weekly medical mystery where a patient with a rare and unsolvable disease is diagnosed by Dr. Gregory House – the title character.  Dr. House is a brilliant diagnostician who can solve almost any medical puzzle, but this past Monday the show featured an entirely different way of diagnosing a medical problem: crowdsourcing.  Over the course of the episode, House’s team dealt with many  advantages and disadvantages of crowdourcing while trying to diagnose a patient with a tricky disease.

Even though House is fiction, there’s a lot we can learn about crowdsourcing from the characters’ experiences.  Here are four of my favorite crowdsourcing lessons from House MD.  Be warned, MAJOR SPOILERS BELOW!  Stop reading now if you don’t want the episode spoiled for you.

Update: The full episode is now available for US viewers on Hulu.

First, here’s a quick plot summary (full episode summary here).  Dr. House resigns from the hospital because of his drug addiction problems. Dr. Foreman takes over the team, and their first patient is a wealthy video game developer with the usual weird symptoms.  Their new patient decides to post his symptoms to the Internet in an effort to gather second opinions and alternative diagnoses – eventually offering $25,000 for the right diagnosis.  Foreman dismisses all of the Internet medical opinions as incorrect, but not before the patient takes a “wisdom of the crowd” vote. The crowd favors amyloidosis, a diagnosis that actually seems reasonable.  Before long, however, the patient exhibits new symptoms that suggest another cause. Dr. Foreman finally realizes that it could be Fabry’s Disease, but not before Dr. Hadley takes a second look at the crowdsourced opinions and sees a convincing submission for Fabry’s.  In an ironic closing, the audience discovers that Dr. House himself anonymously submitted the correct opinion because he was bored and needed a challenge.

Lesson 1 – All opinions are not created equal
Throughout the episode, Foreman’s team must constantly dismiss terrible opinions submitted by the crowd.  While the crowd is often a good way to get new perspectives on a problem, some ideas are better than others.  Good crowds must cultivate experienced subject matter experts whose ideas and contributions can rise to the top.

Lesson 2 – Reputation matters
When Foreman’s team started receiving hundreds of faxed crowdsourced opinions, they were simply overwhelmed by all of the ideas. What if Dr. House’s anonymous opinion had said that the submitter had a 98% approval rating and 95% patient satisfaction rating? Chances are his submission would have been read and considered long before the others.  Good crowdsourcing systems must have reputation management features to help separate the good ideas from the bad.

Lesson 3 – In-house experts are always critical
Crowdsourcing can’t really replace the value of in-house expertise. Dr. Foreman’s team worked to evaluate many of the crowd’s opinions, helping to rule out bad ideas and test good ones.  Crowdsourcing is not a replacement for in-house experts, but rather an exciting new tool to help them gain greater perspective.

Lesson 4 – Money is a paradox
When the patient offered a reward of $25,000, he unleashed a torrent of opinions from around the world. By creating such a huge bounty, he definitely increased participation.  On the other hand, Dr. House submitted his correct opinion because he was bored – not for the reward.  Money is a blunt weapon for incentivizing crowds that can cut both ways. Sometimes more money is necessary to maximize valuable participation, but sometimes people in the crowd are looking for other things of value like a community, an intellectual challenge, or the promise of recurring work in the future.  Good crowdsourcing systems must walk the fine line between offering the right and wrong kinds of rewards to get the best results from the community.

Closing Thoughts
First, props to Innocentive for their House blog post earlier this week. Check it out!

Second, learn more by checking out our great whitepaper with eight awesome and essential crowdsourcing tips.