On the Wisdom of (very small) crowds - more Debunking 2.0

You suspected this all along, didn't you? You looked at those extreme power laws in social network usage (0.9% creators, 9% commentators, 90% consumers), the shenanigans at Digg and Wikipedia, and the massive disparity in Twitter followers between oiks and slebs - and the attention paid by the PR Ho's to the latter in this most non-hierarchical of all mediums - and began to suspect that not all in The Crowd were equal. And, after the Long Tail was debunked last year you suspected the Wisdom of Crowds was probably also not quite what it once was (see footnote below on what it really is). Well, now research from Carnegie Mellon proves you were right - from RWW:

Recent research by Carnegie Mellon University (CMU) professor Vassilis Kostakos pokes a big hole in the prevailing wisdom that the "wisdom of crowds" is a trustworthy force on today's web. His research focused on studying the voting patterns across several sites featuring user-generated reviews including Amazon, IMDb, and BookCrossing. The findings showed that a small group of users accounted for a large number of ratings. In other words, as many have already begun to suspect, small but powerful groups can easily distort what the "crowd" really thinks, leading online reviews to often end up appearing extremely positive or extremely negative.

Yup, turns out that 1% are running (ruining?) all the crowdsourcing schemes too:

It's not surprising then to discover that, when it comes to review sites, it's again small groups that are in control there too. Some sites, including Amazon, attempt to address this discrepancy by allowing users to vote on the helpfulness of reviews - a much easier process than having to write a review yourself. Also, local business finder and recommendations site Yelp implemented ways for business owners to respond to what they feel are inaccurate reviews by way of an owner comments feature. Unfortunately, despite these efforts, the small groups still remain in control of these so-called "popular opinion" features.

And never mind the flacking! As you can imagine, there are schemes abounding about how to solve this - reviewer activity transparency, vetting by social network, censorship of overly positive or negative views etc, but the best is just to be aware:

Perhaps it's time we give up the idea that the "wisdom of the crowds" was ever a driving force behind any socialized, user-generated anything and realize that, just like in life, there will always be active participants as well as the passive passerbys.

Another thought I have is how to "nudge" people into a more equitable form, some way of reducing your influence the more posts you make for example. One could have a declining rate of gaining social capital based on volume over time, modified by voted perceived value from "small cap" low volume users. The options are fascinating.

But fundamentally, the theoretical basis on which a thousand crowdsourcing startups have been launched is flawed, and all the sticking plaster in the world won't solve that.

Update - very good reminders by Taylor Davidson and David Jennings in the comments that in the original book "The Wisdom of Crowds" it made it clear that some very specific conditions were necessary for it to operate effectively. These are (Wikipedia):

Diversity of opinion - Each person should have private information even if it's just an eccentric interpretation of the known facts.
Independence - People's opinions aren't determined by the opinions of those around them.
Decentralization - People are able to specialize and draw on local knowledge.
Aggregation - Some mechanism exists for turning private judgments into a collective decision.

Most of the commercial "wisdom of crowds" commercial applications blithely ignore these, and use unfiltered social network input, so are prone to all the power law errors, madness of crowd problems etc. I forgot to note this difference between the original theory and commercial execution in the post, so apologies. The prime point remains, however - most commercial "crowdsourced" applications are flawed.