On the increasing uselessness of Google.....

The lead up to the Christmas and New Year holidays required researching a number of consumer goods to buy, which of course meant using Google to search for them and ratings reviews thereof. But this year it really hit home just how badly Google's systems have been spammed, as typically anything on Page 1 of the search results was some form of SEO spam - most typically a site that doesn't actually sell you anything, just points to other sites (often doing the same thing) while slipping you some Ads (no doubt sold as "relevant"). The other main scamsite type is one that copies part of the relevant Wikipedia entry and throws lots of Ads at you. It wasn't just me who found this - Paul Kedrosky found the same:

Google has become a snake that too readily consumes its own keyword tail. Identify some words that show up in profitable searches -- from appliances, to mesothelioma suits, to kayak lessons -- churn out content cheaply and regularly, and you're done. On the web, no-one knows you're a content-grinder.

The result, however, is awful. Pages and pages of Google results that are just, for practical purposes, advertisements in the loose guise of articles, original or re-purposed. It hearkens back to the dark days of 1999, before Google arrived, when search had become largely useless, with results completely overwhelmed by spam and info-clutter.

And I can't believe Google doesn't know this - nor does Paul:

Google has to know this. The problem is too big and too obvious to miss. But it's hard to know what you can do algorithmically to solve the problem. Content creators are simply using Google against itself, feeding its hungry crawlers the sort of thing that Google loves to consume, to the detriment of search results and utility. For my part it has had a number of side-effects. One, I avoid searching for things that are likely to score high in Google keyword searches. Appliances are an example, but there are many more, most of which I use mechanisms other than broad search. Second, it has made me more willing to pay for things. In this case I ended up paying for a Consumer Reports review of dishwashers -- the opportunity cost of continuing to try to sort through the info-crap in Google results was simply too high.

Reading the comment's on Paul's blog post was interesting - you can parse the responses into 3 broad groups:

- Yes, we agree with you, and here are some tips on how to deal with it
- Yes, but its not poor Google's fault, its those evil spammers (ie Google has no way of changing their systems and is at the mercy of SEO)
- No, there is no problem, this is the best of all possible solutions (complete bollocks IMHO, it was definitely better a few years ago)

(Ignoring the ones trying to pimp their own products or agendas of course, and the end posts comparing the economics of online vs library copies of Consumer Reports.....)

Ignoring these comments, I have found my behaviour is exactly the same as Paul's , i.e. increasingly reaching for paid-for, edited research (Which? in the UK) as Google and some of the "comparison" sights (clearly flooded with Spam, Sock Puppets and Sleazeoids) become less and less credible. (Another aside - I had a gift voucher from Amazon, and searching for a book I wanted I found Page 1 was totally full of results for the book on Kindle, which was very irritating - they need to allow one to select e-book and/or book).

The interesting question to me is what happens if this gets worse, as Google risks attacks on 2 fronts:

(i) Other search engines decide to eschew Ads for accuracy and cut down the spamming, to gain market share. There is an article on Techcrunch today about Blekko, which appears to promise this.

(ii) The market for paid-for search and research grows - how much would you pay per month for a neutral search engine? Which? costs about £7 a month, would you pay that for a neutral engine?

Frankly, I don't believe that it is not possible to reduce this sort of spam, I think Google's problem is more that it is trying to navigate a line between income (systemically the more spam there is, the more Ad money it makes) and usefulness (how much spam can you run before the user walks away) and has veered too far to the spamside.

Update - this piece was ReTwt'd by Tim O'Reilly, and had the equivalent of a slashdotting so down went our server - fortunately those nice people over at Hacker News pointed to the proxies pronto. I liked two of the themes in the comments there:

(i) a lot of these sites must be known, ditto their pattern, so just a few weeks judicious "mechanical turk" work should have a large 80/20 impact

(ii) Google is like a monoculture, and thus parasites have a major impact once they have adapted to it - especially if Google has "lost the war". If search was more heterogenous, spamsites would find it more costly to scam every site. That is a very interesting argument against the level of Google market dominance

Update 2 - Anil Dash and Coding Horror have picked up on the same issue today.

Update 3 - Bruse Sterling has picked up on the story - his comment:

"we may be approaching a period where the machines will feed you an infinite amount of cunningly-engineered gibberish"