The Limits to Social Media and the opening of Pandora's Box.

This post comes from something thats been rattling around in my head awhile, which is this - if the wisdom of crowds is so great, how come Pandora, with its analysis based "music genome" algorithms, gave me a more consistent selection of music I like compared to the crowd-sourced Last.fm?

(By the way - huge kudos to Improbulus for showing how us UK based musophiles can get Pandora again via her article on Geospoofing - we luvs the Internetz )

If I look at my experience of the key differences between the two, it is this:

(i) Output quality - LastFm, like Amazon, selects based on what others who like X also like. So for example, if I like the Sex Pistols and Abba, and you love Abba - guess what - at some point a crowdsourcing system is going to serve you Anarchy in the UK. Last.fm now does have a "music like X..." option, and I've been playing with it over the last 3 months, but it just does not optimise like Pandora can. Both systems allow you to like / not like, but I suspect it is just harder for a system where the base data is more flawed to optimise as well.

(ii) Input Quality - crowdsourcing systems depend on people putting in what they like - which, in my experience, means a shorter tail, its not just me who has noticed that Pandora serves a greater variety of acts that accord with your taste as its not as bothered with popularity of any kind

Now I did notice when I started last year that Last.fm got up and running to converge around an optimum faster, but once Pandora was on the hunt it quickly overtook Last.fm in quality of the experience. Also, I suspect that its harder to "game" Pandora wth SEO link love etc than any crowdsource based system, which is prone to various schemes to artifically boost perceived popularity.

The impetus for blogging all this now is that I have recently finished a rather interesting book, Super Crunchers, which is in essence the emerging story of data crunching in a high data web 2.0 world with very cheap megaflops. The implication of this book is that its is now both operationally and economically possible to crunch extremely large datasets and derive subtle links that no "wisdom of crowds" system could achieve (as I'm sure Google et al know only too well).

Also, as anyone who has studied social networks knows, there are some quite difficult precondions that have to be in place to ensure you get a "wisdom of crowds" rather than a madness or groupthink of crowds going on. The minute we see feedback, we stop thinking for ourselves and drop into socialthink mode. It is extremely doubtful if these preconditions exist in any current social nets, or even if they are possible - and if they are, whether they are stable as they are prone to gaming.

Thus, if there is a claim for a "Web 3.0", where "Web 2.0" is the interactive web, its the emergence of the "supercrunched" Web (which also plays in to another trend, the rise of the m2m web I think). And there is a lot of storage and crunching capacity being built ( And just who is building many of those datacentres - ah - Google. Funny that )

There are going to be downsides - privacy for example. I don't know what Pandora could deduce from me by knowing my taste in music, but I suspect a psychologist could have a field day - and if it were linked to my activity elsewhere I suspect - as behavioural psychology and economics is showing - that it may know more about me than I do. Thus there will be a very strong argument to keep various niches of our data separate, for us to optimise our experience in vertical.

By the way, I suspect that the endgame will be hybrid systems, and that a Social Media system may (may!) give a higher level of serendipity simply by the nature of how its architected

But this Pandora's box, just like the first one, once opened, can't be shut again.

Now, I'm sure the criticism will be levelled that I have just succumbed to the "groupthink" mode by allowing myself to be seduced by this book, and that in fact no algorithms can reproduce the subtlety of a million, a billion human minds. My counter - that's possible, but the evidence, in my view, sings for itself