New Search

Practice is starting to look like theory. When Social Networking started, it was clear that people would use it to find things that way as well as using search engines. Its just that the volumes were small and Social Networks, by and large, had crap comms to share things by. Twitter changed all that, and now Facebook - having implemented far better and simpler comms - is finding the same:

According to Web measurement firm Compete Inc., Facebook has passed search-engine giant Google to become the top source for traffic to major portals like Yahoo and MSN, and is among the leaders for other types of sites.

For what its worth this blog has had more referrals from Twitter than Google for some time, which has made us far less impressed with all the arcania of Google SEO).

But now that its happening in scale, it puts Google's model under pressure as it is simply cruder for Advertising:

David Berkowitz, director of emerging media and client strategy for the digital marketing firm 360i of New York, said the importance of search engines isn't going away.

"But there's always been one downside to search," he said. "Consumers only spend about 5 percent of their time online searching and the other 95 percent of the time at the destination. Social media is quickly accounting for a large percentage of that 95 percent. Google's biggest acquisitions, DoubleClick and YouTube, have been all about playing a big role in the rest of consumers' Web usage."

But Social search just isn't scalable without the Signal to Noise ratio getting unmanageable - so we expect to see a whole raft of Social Algorithms emerge to help automate, speed up and filter the process. How do we know this - because we, and a whole lot of other people we know, are fiddling with them, and they show a lot of early promise.

Steve Rubel hypothesized Buzz was launched half baked and half cocked to get out there before Facebook launched its FBMail service, and that may be so - but to our mind the real Buzz is in the algorithms dealing with the blend of social graph and social data.

Not only can one tell a lot about you by the searches you make, seems a lot can also be deduced from the search engine we use, as a WPP survey of 17,000 US internet users shows - Ad Age:

What does your search engine say about you? Well, if it's Bing, you're probably an early adopter, but you also visit, shop and ultimately make purchases from Walmart more than other search-engine users. Google searchers, on the other hand, are partial to Target and Amazon, and Yahoo searchers have a strong preference for wireless service from AT&T and Sprint.

Google users are more likely to book a flight online at JetBlue or make a reservation on Hotwire. They are also more likely to do research on a Lexus, while Bing users tend more toward Toyota. Aside from uncovering the research and purchasing habits of consumers, the study claims to have identified a host of demographic and psychographic information on search-engine users.

For instance, AOL customers feel less intellectual than their peers, are 55 and older, spend their money more responsibly, want to blend in to the crowd, feel like they've gotten a raw deal out of life, expect less from their future and, believe it or not, still use dial-up modems. Bing users are middle-aged, highly educated tech-savvy individuals who consider themselves to be average and spend more than 10 hours a week online.

Googlers tend to be the average internet Joe, according to the study. The search leader's loyalists are conventional people yet open to trying new things, believe in following rules and don't consider themselves any smarter or less intelligent than the person next to them. Yahoo users tend to be 55-plus, reserved and a less-independent group with little faith in imagination. They feel they have little control over their future and are skeptical and cautious of new or untried ideas.

Fascinating. I've used Dogpile for nigh on 10 years, it searches all the other engines and aggregates their result for you. I love the "parasite search engine" design (I wonder what that says about me ). Interesting that it hasn't really taken off though.

Update - realised there is a bigger story here. In an age where we see Google moving to mainstream advertising at the Superbowl (while Pepsi eschews it), clearly which search engine you use is becoming as much a branding battle as for any other commodity product - ie there is little inherent technical differentiation left. Increasingly search engines will try and identify demographics they want for Advertising purposes, and set out their brands to recruit them. Next step is diversofoed brands for different segments - Google Lite, Bing Premier etc - and celebrity sponsorship. Oh joy.....

I wondered how long it would take someone to build a system that obfuscates your data from Google. We looked at how to do it a few years ago, but the best short term default seemed to use Dogpile, a "parasite" search engine, or similar - yes, they have your data but its a far smaller operation. But that was just search. . However, a smart hacker has come up with a way of doing it. Forbes:

Now an independent security researcher who goes by the name Moxie Marlinspike is making Web users a counter-offer: Take Google's ( GOOG - news - people ) giveaways and keep your privacy too.

On Tuesday, Marlinspike launched a service he calls Googlesharing, a plug-in for Firefox designed to give users access to Google's online offerings while cloaking their identity from the company's data collection tools. By hosting a proxy server with a collection of Google "identities," the privacy software, which can be accessed at Googlesharing.net, will allow users to temporarily route their traffic through another computer that masks their identity by mixing their online actions with those of other users.

"Each identity looks like a normal user, but everything is mixed up between identities so Google can't track any individual," says Marlinspike. That means users can exploit any of Google's offerings that don't require logins, such as search, maps or news, without allowing Google to assemble a profile of their activities that can be used for advertising targeting--or, as some users might fear, information that could be subpoenaed by government investigators.

You can get the Firefox addition here. Marlinspike is offering up his code so you can build your own, so well worth a look methinks. I predict a large increase in Google services that have to be signed in for. I wonder if the Open ID people may have looked at this, but they have a different gig right now - but maybe we have a bigger need for Private rather than Open ID.

You all know by now that Google is pulling out of China (actually, its "May pull out if non censorship request is not met", but that is very unlikely). The reason given is a massive attack on the accounts of Chinese dissidents in Google (and other) companies' servers. But, as was pointed out on the BBC Today program this morning, everyone told Google that would happen when they went in 4 years ago. The Google response this morning was that they hoped that by engaging, Chinese behaviour would change

Another reading - the RealEkonomik reading - is that Google knew this would happen, but that it could not afford to ignore such a huge emerging market so went in with eyes wide open, albeit possibly with rose tinted glasses. As Shefaly Yogendra points out, strategy in China is a new Great Game.

So what's changed? We suspect that as well as an increase in attacks on the servers (Why is China doing it only a problem now, though?*), three other factors have driven them:

- Their reputation in the far more valuable US and European Union market is diving, and both areas are looking at regulation to curb their activity.

- Nothing builds confidence in your new Cloud strategy like knowing its massively hackable by anyone with bit of will and nous

- Penetration in China has been disappointing for Google, its Chinese management has been turbulent, ARPU is still low but costs are not, so its not a great economic benefit at this point

Let's be clear - a publicly owned corporation does not do things on points of ideological principal, Google is doing this as it believes it serves its own best economic interests here and now. In other words, in the short term it would hopefully do Google a lot of PR good (and allay some legal activity) to give China short shrift. In the long term this may prove to be short sighted, but its a reversible decision and in the long term we are all dead - and the current management will long since have taken their pension pots into retirement. Besides, doors are still open....

All that said, it is still thrilling - regardless of the reasons why, they are doing the ethical thing.... Interesting Times are surely ahead.

* It will be interesting to see if other companies like Microsoft and Yahoo report increasing attacks. Also, surely this is a government to government issue if as widespread as claimed? (Update to this - a statement from Hilary Clinton)

Techmeme comes up with their top 10 stories of the year - and 7 out of 10 are Apple or Google stories. The other 3 are TechCrunch (with an Apple substory), Friendfeed (ex Googlers) and eBay/Skype. How was such a monochrome palette selected - sez TM:

Techmeme has distilled its very own Top 10 story list, utilizing headline ranking data archived throughout the year. While we employ a terrific team of news editors to guide our story selection, this list draws on many factors outside our direct influence, making it a … nearly objective sampling of what was important in 2009.

Hmmm... to an extent what you get out of it depends on what you put into it. I've noticed that TM is biassing more and more towards mainstream sources, so maybe that is an explanation.

But even so - seriously - no Twitter, Hulu etc? Or the AT&T/4Chan and PirateBay stories that hit top spots on Digg? If I look at the traffic on our blog, we covered all the above stories and they were not the biggest runners (see here). Be interesting to see what the top runners on TechCrunch, Mashable etc. were.

One wonders about the exact selection criteria! (Update - Techmeme's Gabe Rivera has commented below, explaining its based on link count. That makes more sense as I suspect Google and Apple have more fanboi 2.0's than say Microsoft, but also - in my humble opinion - shows the risks of pure link based ratings)

Now I really like TechMeme, and am a great fan of its model, I've always seen them as a leading example of how media may work in the future - but the risk they run here is becoming Yet Another Mainstream Tech News Organ..... so why?

Does becoming a me-too storyboard system create sufficient value from the (hypothesized) increased volumes? One also therefore starts to wonder more deeply about how they are funded, and the implications thereof!

Benjamin Ellis is trying a fascinating experiment - trying to live a week online without Google. Why? Well, those who are watching the runes are getting increasingly worried about Google's view of privacy (aka None, Get over it - see our comments here ) and wondering what steps can be taken. As he notes:

I’m just curious as to how pervasive a part of my Internet experience Google has become and how much data they really are amassing. The only way to really know is to pay some attention to it for a few days, so why not a week? At the end of day 1 I have to say I’m quite shocked. I’ll post a full summary at the end of the week, but here are some of my observations from today:

He goes on to list some things he's noted about how pervasive Google is:

- Firstly, I didn’t realise that Safari uses a “Google Safe Browsing” service, so that had to be switched off in settings before I could do anything else on my Mac.

- I couldn’t change the default search engine in Sarafi – the setting that I’m SURE used to be there is GONE! The site with instructions on switching Google off in the Safari web browers, returned by the Bing.com search engine, had both Google Friend Connect AND Google analytics, as well as google ads and google blog search on it. Doh! Foiled there as each of those services passes data back to Google, and so is off limits this week.

- Finally I thought, ok, I’ll just take “search” off of the safari tool bar. Would you believe I can’t remove the search box with out removing the address bar too? Apple, what where you thinking? Do they really love Google that much?

I've had a policy for about a year now of not using any Google products knowingly, including search, with one exception - YouTube. But what is interesting about Benjamin's findings to date is you may be unknowingly passing data back to Google via pre-sets in the architecture such as in Safari.

He also later noted later that he sees Doubleclick (aka Google) based Ads popping up on Amazon for example. This puts Google seemingly into increasing conflict with what users want, ie search results for the most relevant content. Be interesting to see what he concludes after a week.

Calling the Zenith of any growth curve is always hard, and calling the Zenith of a smart company like Google is one of those "gulp" moments, but I think we are now in sight of it (and by in sight I don't mean right now, but in a 1 - 2 year horizon) - for 4 reasons:

Firstly, the Content Owners are starting to organise themselves.

In the recent brouhaha over instream Ads on Twitter, the Ad.ly CEO wrote a defence of the practice. Overall its a seductive hymn for Twitter to turn its assets over to commercial rapine by a 3rd party (if Twitter do this they are daft in my humble opinion, but that is another story - see my worry that they have Missed The Cluetrain), but one of his points was well made:

3. In-Stream rewards the content producer, Contextual just the technology provider – The funny thing about contextual search like that in Google is that it benefits the tech company and NOT the content producer at all. Think about this – bands and stars helped bring people by the millions to MySpace. Having amassed a following they realized that MySpace was able to put ads everywhere and make all of the money. Same goes with Google. All of us bloggers and journalists create content that gets indexed and allows Google to serve up ads alongside us that we don’t benefit financially from. When a content producer promotes an ad in-stream the revenue flows mostly to the person who published the content.

My Italics, but that is really what has happened for the last 6 years or so - Google has managed to pinch a lot of the surplus from other parts of the value chain. But the others are starting to fight back. The push by News International to get Mainstream media to de-list from Google is the latest and most public salvo in this war but it won't be the last. The Mainstream media is hurting badly, and - given the alternatives are pretty bleak - one option is to force Google to hand some of the surplus back by taking away their bat and ball elsewhere, reducing Google to search the "Long Tail of Crap" that is the rest of the Web (as far as the mainstream market is concerned, anyway). In fact, the latest twist in the tale, that Microsoft and News International are reaching a pact to delist from Google and go exclusively elsewhere, also illustrates the second trend:

Secondly - Better Competition is emerging

Microsoft's Bing search engine, and other niche search applications like Real Time search are starting to chip away at Google's stranglehold on search - and more importantly, Ad revenues. The attempt by Microsoft to de-list News International from Google and pay it to be on Bing is the latest - and by far not the last - foray by other people's tanks onto Google's well manicured lawns. This matters for 2 reasons:

- Google's share of the Ad market starts to decline as others take a greater % of it - this may even add up to a real revenue decline as Google has no other source of income

- The margins of Advertising fall as competition increases

This will hit revenue, increase costs and reduce earnings - so Google stock price falls, market sentiment shifts and a whole load of positive reinforcing loops fall away. It also puts more pressure on Google's main strategic weakness, ie its inability to find revenue elsewhere.

Thirdly - So Far, Google has been unable to make money elsewhere

Google has used its huge surplusses to build out businesses in a large number of other areas, but as yet none have been successful except maybe YouTube ( though it will have to start making money soon as the world moves to far more expensive-to-serve HD formats). Every time Gmail goes down another nail is driven into its Cloud services. Its architecture, though formidable, is optimised for text search so its acquisitions struggle to get integrated and the Google culture seems too strong to allow them to succeed once acquired anyway. (And in fact YouTube highlights something else - most home grown services in other areas are poor - that they now acquire most of their Innovation speaks volumes about their ability to innovate a way out of the Ad Biz.)

Strategically the risk they now face is that they have opened up wars on multiple competitive fronts but funding to carry them out will start to drain away. The problem is, the people they have attacked are now all mustering their tanks and will probably rather relish carving up bits of Google's lawns for themselves.

Fourthly - Consumers are starting to understand the privacy implications of Google's Datamining

Due mainly to the avaricious behaviours of other johnny-come-later companies trying to mine the consumer datastream, a better user understanding is emerging of what user data Google has, and what it does with it. From a slow start, this has started to worry users and they are reacting in 3 ways:

- Using other services that do not seem to compromise them so much, and tools to help (the VRM movement being typical of this counter-cycle)

- Demanding legislation to protect their interests - so far more prevalent in Europe than the US, but this is a trend that won't go away soon

- Asking for open-ness of the data so they can control its use better

All these activities curtail the value-add that Google can put on its Ads.

These trends do not mean that Google withers on the vine of course, just that its 5 year hegemony is coming to an end as the rest of the value chain works out strategies to reclaim some of the surplus, as competitors work out the chinks in its armour, and as customers start to work out what they do want. By and large Google has used its Ad money to subsidize sub-priced entry elsewhere, now others are using the same tactic back to hits its core revenues in advertising.

Strategically, this puts them on a back foot. Here are some predictions for the next 3-5 years:

- Increased spend on Marketing and Lobbying - every embattled incumbent reaches for legislation, Google will do so too. The Net Neutrality loony fringe will go into overdrive, as will the "MSM are dumb sh*ts" chorus and the "Ads are Good and Natural" apologists, but the real work will be on limiting the impact on data storage and mining.

- Exiting some of the areas they have been in - so far they have tended to cull them, but we may see them spinning stuff off or out, or even joint venturing in attempts to create value from sunk investments.

- Start buying "real" businesses with non Net-Ad revenue streams while its share price is still very highly valued (When this happens, that is the main sign that this analysis is accepted by teh company itself)

- Company culture shifts from Geeks to Commercial people, from innovation to sales, and operational excellence (aka cost cutting) takes a bigger role. The bright young things leave for newer, shinier things, but every blue-blazered MBA wants to join like they once wanted to be in P&G.

Its all part of the inevitable circle of (economic) life, the difference is that when it happens to Google, then a few Internet Economy myths will hit the floor too - Free(Con)omics, Network Effects, Do NO Evil Corporations - will all be shown to be things that happen in times of early rapid growth, before Darwinian competition for a profitable niche emerges.

We could be wrong, of course....but I think we've called it about right, give or take a year it will seem clear.

Update - nice rebuttal by the very clever Graeme Pietersz, essentially arguing the competitors won't be good enough and consumers will always be just dumb sheep. May be right, but in my view the competition sucks a lot less than just a year ago and consumers seem to be slowly waking up. As I said, we'll know in a year or so - the Ad market will be the driving factor,

Watching a fascinating story unfold in real time on the blogosphere (I saw it on Twitter first). There has been a hack on one of the main Pro Global Warming group's files and it turns out that they maybe were being a little economical with the truth - NYT:

Hundreds of private e-mails and documents hacked from a computer server at a British university are causing a stir among global warming skeptics, who say they show that climate scientists conspired to overstate the case for a human influence on climate change.

The e-mails, attributed to prominent American and British climate researchers, include discussions of scientific data and whether it should be released, exchanges about how best to combat the arguments of skeptics, and casual comments — in some cases derisive — about specific people known for their skeptical views. Drafts of scientific papers and a photo collage that portrays climate skeptics on an ice floe were also among the hacked data, some of which dates back 13 years.

In one e-mail exchange, a scientist writes of using a statistical “trick” in a chart illustrating a recent sharp warming trend. In another, a scientist refers to climate skeptics as “idiots.”

Not entirely surprised, it's been clear that the AGW - Anthropomorphic (Human Caused) Global Warming - industry has been an "industry" - ie has a lot of money is riding on it - for quite a few years now, so there is now a huge investment by vested interests. To be honest its always irritated me somewhat as I know enough maths* to know that most of the allegations of man made warming activity are just noise in the standard deviation of Good Old Gaia, which is in addition quite a complex system and can flip from state to state quiet fast. Also a mere smidgen of historical research makes you realise the world has been warmer than today even in recent millenia (grapes were grown in Yorkshire in Roman times and cattle grazed in a green Greenland in early medieval times for example).

But I am in favour of reducing reliance on fossil fuels and ensuring our energy consumption is optimised on purely rational "what happens when its gone" basis, so its been an "enemy of my enemy is my friend" thing as the sceptic lobbies are heavily populated by the Status Quo and do need quite a big push to do the right thing - and they are certainly no angels in this either.

I've long felt the risk that the populist AGW argument was too over the top and that it would eventually crash and burn and take all the good pro Global Warming science down with it. In that respect this is quite worrying.

There are also three useful internet based lessons from this:

- The truth will out, eventually - and its usually not a techy issue, its good old human intervention at the heart of the system. (i.e. when you write an email, think - would I like this read out in public?)

- Once out, its gone - this stuff has been mirrored on servers across the world.

- Once its out, hordes of people will start to spend brain cycles looking at it, and start to talk about it, and put together a more accurate analysis - and with modern social media tools they will do it fast, and people like me will read it and comment, and write about it......

The resulting broahaha should be very entertaining, at any rate. Should make the Copenhagen conference quite a bunfight.

*being a qualified engineer I've not only studied (a lot of) maths, but physics, chemistry and thermodynamics as well.... not that many pundits feel the need to be constrained by these things in this debate!

Scarcely a day goes by without yet another "real time search service" developer on the Twitter API declaring that they can do data analysis on the Twitter ecosystem, thus hopefully chasing the unbelievable riches that brand and influencer aggregators believe is there.

Unfortunately, Twitter has spotted it too and has decided if anybody is going to make money, its them. They are "Talking Separately to Microsoft and Google About Big Data-Mining Deals" sez BoomTown.

Sources said a number of scenarios are being discussed to compensate Twitter for its huge and potentially valuable trove of real-time and content-sharing information, generated from the data stream of billions of tweets from its 54 million monthly users.

These include a number of structures, including a payment of several million dollars to Twitter, along with various revenue-sharing proposals that would give Twitter a piece of the revenue made from search results.

The deals, stressed sources close to the situation, are nonexclusive, especially because Twitter’s management is keen to remain independent and also nonpartisan in the growing search battle between Google (GOOG) and Microsoft (MSFT).

This means Yahoo (YHOO)–which recently struck a search-technology and online-advertising partnership with Microsoft–could also license Twitter’s feed to make its search results even more robust, although Boomtown could not determine if the company is in talks with the San Francisco start-up.

What this means is that the true value of the data will be realised. What it also means is that they won't let it go to two-bit startups building their own businesses by giving twitter data away for free to build their own businesses.

From the Back to the Future Dept - Nielsen finds that:

In a nutshell, there is a segment of the online population that uses social media as a core navigation and information discovery tool — roughly 18 percent of users see it as core to finding new information. (See Chart Above - Ed)

Good heavens, who'd'athunk it! (Cough - this was why the whole world went into a tizzy about SocMed, the dream of it being able to capture Googlegold was a big part of the hype story in 2006/7, so well caught up Nielsen )

Anyway, as is the wont of marketing wonks, they divide the world into 3 groups:

At the root of the changing nature of content discovery is the sheer amount of information that is available on the Web. If you want to learn more about the latest smartphone released into the market, your favorite search engine is sure to provide you with hundreds, if not thousands, of articles about the device. But with the increasing number of resources available, it’s difficult to know what you should believe or take at face value. Socializers – those who spend 10 percent or more of their online time on social media – feel this effect more than others do. When asked, 26 percent feel that there is too much information available on the Internet, compared to 18 percent of people who predominantly use portals [Portalists]and just 5 percent of people who primarily use search engines [Searchers].

Can one be both a Socialiser and a Searcher, one wonders....? Seems to me that social media users are much more scared of navigating the sheer volume of info on the 'Net than pure searchers, and want it mediated heavily by people they trust. Portalists are somewhere in between.

Anyway, useful data to back up stuff you've known for 3 years - but the thing that astounded me was the size of the Portalist community. By 2009 I thought it would be in the minority!