Friday, November 11. 2016
Lest we forget.....
' "Good morning; good morning!" ' the General said
When we met him last week on our way to the line.
Now the soldiers he smiled at are most of 'em dead,
And we're cursing his staff for incompetent swine.
"He's a cheery old card", grunted Harry to Jack
As they slogged up to Arras with rifle and pack
But he did for them both by his plan of attack.'
- Siegfried Sassoon
Thursday, November 10. 2016
(Tracking the US Election memes - Trump is the big blob way ahead on the right)
On Saturday November 5th we decided to "go public" with our prediction that Donald Trump was in pole position to win the US Presidential Election. We were in a small minority, most polls and researchers were calling it for Clinton with odds ranging from high 70's to near 100%, but we trusted what we were seeing on our Dataswarm analytic engine. The system had worked for Brexit, after all (though we were too uncertain to publicise it then, preferring no egg on face to instant fame). So, suitably caveated, we posted it up. Carpe Diem, and all that.
However, it seemed that the minute we posted it, all the good news for Trump started going bad, which from our self-interested point of view was also bad news. First the FBI investigation into Clinton's emails was cancelled, and we were told her support was rising. Then news came through that she was storming ahead in early voting in a number of key states. If she won we would have egg on our faces, if we retracted the prediction we would too.
(We make no comment about anyone's views about the political outcome, this is all about how the technology worked)
Waking up in time for the 6 am BBC morning news in the UK (5 hours ahead of the US) we heard that Trump had almost won, with a far wider margin than our system had shown was possible. By 8 o'clock most pundits had called it. Trump was the next President -elect.
Our system had got it right - it had worked.
So why had we got it right when nearly all the other polls and pundits had called it wrong? Now we have had a day or so to look at the outcomes, we think there are 4 main reasons.
Firstly, Internet vs human polling. Our system is looking at verbatim Social Media data, from Twitter. We had come to the conclusion while monitoring previous UK general elections that people were more willing to share their true thoughts on social media than with pollsters, especially if their views were "non-PC" (in this case, pro-Trump). After the election we read that the LA Times poll, which had consistently been more pro Trump (and been roundly criticised by nearly every pundit), had been an internet poll, not using people to ask questions, and they believed (and were proved right) that people had been more honest on that. In effect by monitoring social media, we were getting the same sort of uncensored opinions, and in that uncensored world Trump was doing a lot better than the standard polls were predicting. Also, we knew from UK elections about the "shy Tory" effect where people say one thing - typically to look good (virtue signalling as it is called) - in public, and do another at the ballot box (To misquote Phil Ochs, Liberals are 10% left of centre in public, 10% right of centre at the ballot box).
(Update - there has been a rush of Social Media monitoring companies saying they saw the same thing as us (here and here for example), though they seemed a little bit more reticent than us about calling it before the event )
Secondly, the way our system works helped quite a bit. It was initially designed to satisfy a BBC requirement to "Find the Zeitgeist" across its media output, as well as compare it to others' output. To solve this we used a fairly obscure technique that we had become interested in called memetic analysis, that groups memes into groups of fellow travellers (called "memeplexes" in the lingo) rather than look at things one by one, as Boolean analysis forces one to do. We started it going the day after Trump became REpublicam Party candidadte, and by Nov 8th it had crunched a frelevant sample of c 170m tweets and was tracking 4.5m memes. What our system was showing was that from the get go, Trump had dominated the memespace (as he had in the primaries too). In meme theory as originally proposed by Richard Dawkins (who coined the term meme - or cultural gene), the view is that memes colonise your mindspace - so in effect the Trump memeplex was hogging the electorates' mindspace, starving out competing memes. Clinton was not anywhere near. To be sure, not all Trump memes were positive, but in essence Trump was using a "Wildean strategy" (The only thing worse than being talked about is not being talked about).
You can see how this works in the Youtube video of the system tracking Trump, above
Thirdly, we knew from Brexit that the "non voters" were very motivated to come out to vote if someone could credibly promise an "out" from the current political system. Trump seemed to be doing that succesfully. (One can argue about the morality of his tactics, but the effectiveness of the strategy had been proven for Brexit), and we thought it was happening again.
Lastly, Confirmation bias. We knew that after Brexit and the Republican Primaries, UK and US pollsters had looked at why they had got it wrong. They had realsed that their unwillingness to countenance a Leave/Trump victory had made them look at the data from a point of view of what they wanted to see, not what the data said. Given that the US media and pollsters seemed to be even more pro Clinton that the UK equivalents were pro Remain, we suspected there was a Clinton bias in the polling uncounted supporters -
That is why, with Trump just ahead, we thought he would probably win.
For what its worth, we had thought if we were wrong it would be because the vote split would go Clinton's way in marginal districts, due to the reputed strength of the Democrat "ground game", beating the above factors. But as with Brexit, the Trump voters proved more motivated and got out and voted.*
(We did note, in our weasily caveats on the 5th, that Clinton could win the Electoral college, but we thought Trump would have the popular vote. Ironically, it turned out the other way).
*Update - it seems Clinton got more of the popular vote than initially reported on the day (though less than most of the polls thought), this is interesting as we thought she'd get about the same or less. We were still more pessimistic than the pollsters about her chances and that made the difference in the call I guess - though as we point out here even by their own now revealed calculations, the 70%+ chances the pollsters were giving here were not really justified.
Saturday, November 5. 2016
Broadsight's DataSwarm memetracker, 5 Nov 2016
Coming into the final days, our tracking of the memes around the election on Twitter show that in volume terms Clinton related memes have caught up with Trump related memes since we last wrote (see here)
Remember, memes are a proxy for mindspace (as reported on social media, anyway) that various topics are occupying in people's thoughts. We are not looking at what the mainstream media are pumping out, but what people are talking about. There are all sorts of allegations of Trumpbots, and the US mainstream media being "rigged" against Trump, but we are tracking social media output here at vast scale. We did turn the spamkiller algorithms off to see what the raw output looked like, it will be one of the things to look at afterwards to see what their effect was.
Now to be honest we don't know what all this means in terms of election outcome, we set up this project to see how to calibrate a meme tracking system to the on-the-ground results in a US political election. What we can say is that, since we last wrote, the volume and reach of Clinton related memes have caught up with Trump related memes overall, so he no longer has quite the dominance in social media memeshare. But this is not necessarily a good thing for Clinton, a not insignificant rise has been in the memes around topics like "wikileaks" and "Benghazi" (see smaller orange dot in centre) and this is not all complimentary to her. Rather interestingly the Bernie Sanders "feelthebern" movement is still trundling along at scale (orange dot on top right), again this is not necessarily good news for Clinton. Trump's women-groping problems have not really lasted as an item, in essence there was a lot of noise among a relatively small number of people for a limited time. The current fastest rising complimentary Trump mean has been on immigration.
We also saw no real impact from the debates in terms of sentiment shift, just more traffic. Again, how one interprets this is uncertain but our hypothesis is that the vast majority of people talking on social media knew who they supported before and just rallied to their candidate, and the undecideds who talked online are a small % so didn't really shift the dial.
What is often useful in mapping the "Zeitgeist" - the amalgamation of all the small trends and shifts - is looking at "the rate of change in the rates of change". Trump sentiment remains roughly the same, he has followed a Wildean strategy (the only thing worse than being talked about is not being talked about) to dominate the memespace, so maybe it will work for Clinton too as she has caught up, but quite a bit of what is driving her rise are negative memes.
The other thing we do know from tracking the last two UK elections is that "un-politically correct" opinions are under-represented, people are shy of stating these opinions publicly but do vote for them. (To misquote Phil Ochs, liberals are 10% to the left of centre in public and 10% to the right of centre in the voting booth). Trump may have let more people feel they can say what they really think, but we'd suspect there is still a residual "shy Trump" voting bloc. Also, demographically many of the people who are natural Trump supporters are less likely to be on social media. Now whether these will get out and vote is another matter, but it was these people coming out to vote (and failure to correct for "shy brexiteers") that caused the upset in the UK's Brexit vote.
(Update - signs are that in Nevada the Clinton machine is getting the early vote out and the Trumpites are staying at home)
As mentioned in our first article, Brexit differs from the US election in that it was an overall referendum, i.e. every vote counted equally wheraes in the US election it depends very much where a vote is cast. A vote for Trump or Clinton above those needed to secure victory in any one particular ward is essentially a wasted vote, it doesn't count towards the outcome.
But if we may speculate on one possible interpretation of the meme data - Trump has dominated the memespace from the beginning, and is probably more represented in the non-online population. So it may be that he will have more people actually voting for him, but they may be in the "wrong" places. So there may be a Clinton win, but with more actual votes for Trump.
Anyway, we shall know in 3 days and then we will be able to have a good look at our data nd how it mapped to real outcomes
Saturday, October 22. 2016
Yesterday some major sites were brought down by DDoS attacks, and the machines that were used were "dumb" IoT devices with poor or no security - El Reg:
Today, a huge army of hijacked internet-connected devices – from security cameras to home routers – turned on their owners and broke a big chunk of the internet.
In the old days the delay between prediction of bad things happening and some occurrence event in new technologies were several years, if not over a a decade - nowadays its months.
Still, interesting is that, while Twitter was brought down, its share price went up 7% - possibly people only realise how essential it is when its not available (or, as others suggest, some are prepared to pay to keep it switched off)
The more serious issues are around the clear problem the IoT poses (i) to the Internet, and the need (for now) to keep these services off the 'Net and (ii) for countries, its clear that this is a very vulnerable point for cyberwarfare
Sunday, October 9. 2016
We predicted 2 weeks ago that the media would throw a huge amount of memes at Trump, but never imagined that it would come to this!
At any rate, so far it hasn't really made a difference to the memetic makeup overall - yes its the story of the moment, but the anti and pro isn't hugely different on an initial analysis, so it seems like the story is falling along pro and anti Trump supporter lines. If it is changing the minds of independents, there is no sign as yet.
Anyway, with the debate tonight this should all be exacerbated massively, so it will be interesting to see what shifts occur
Just a few hours in, not seeing any seismic shift yet in the incoming data in terms of any major mind changes, but the volume of comments on Trump's pre debate issues seems to have dropped for now and been replaced by other topics that came up in the debate.
Friday, October 7. 2016
Interesting study on the (lack of) difference between cities' medieval and modern usage from SFI:
In both medieval and modern European cities, larger settlements have predictably higher population densities than smaller cities.
Although, one wonders if these cities lack of change is due to them being pretty much the same physically over that period. But anyway, the lesson is that 700 years apart, people still behave much the same way.
There is a lesson there for all the "New Ways of Working / Living / Communicating" evangelists....
Thursday, October 6. 2016
Motherboard has an article on a recent DDoS attack that, for the first time used IoT devices as the slave intelligences that launched it - it's over here written by one of the people we really rate on security, Bruce Schneier
But the really critical point in the article, in our view, is this:
In other words the added value per unit is so small it is simply not worth putting in the level of security for these systems to be safe on the Internet. Not only that, many of the devices don't have enough intelligence to be secure (even putting any form of non-trival security in a Raspberry Pi or Arduino is a challenge)
And even if you could, the installed base of dumb IoT is so large that its going to be a problem for a long time - as Motherboard notes, it's a market externality:
.tIn other words, there is ready made market failure emerging. The answer has to be intervention by authority, but if it is government intervention it is only countrywide (though a grouping like the EU could have more impact) but it's clear a lot of countries won't do this, or if they do will do it with minimal rigour, as it costs money and there is litlle value to them.
Monday, September 26. 2016
DataSwarm system chart of the US Election meme swarm
(Update - we wrote this before the TV debate, we make a few comments about what we see in the last 12 hours or so since the election at the end)
I guess like quite a few Data Analytics/Insights companies, we've been tracking the US election using our systems (more details on our DataSwarm system here) just to see what it is telling us vs what we can see in the press etc, and vs. what others are saying.
Anyway, one of the predictors we follow closely is Nate Silver, as his 5-38 operation has called quite a few elections accurately before. Like many others, he is tracking the rise of Trump from what was considered a low (c 13.6% chance of election) just after the Democrat National Convention, and a steady rise since then till now they are about neck and neck (see their time chart here, just below the breakline.)
This (and similar) is interesting to us, as our system has only ever seen Trump out in front from the get go, the only thing that has varied is how far. To be sure this is because we are looking at different factors, we are looking at the memetic impact of Trump vs Clinton, not poll data, and Trump has never flagged in the meme race.
A quick refresher on Memetics - the term meme was coined by Richard Dawkins as a "mental gene", and its role is to replicate itself by colonising other minds. Like genes, memes travel in groups called "memeplexes", and the ones that are prevalent in a culture or subsector are what we term the "zeitgeist". There is a branch of mathematics called Memetic Analysis which is quite useful for analysing this, its a sort of cross breed between the maths of Viruses spreading and feedback-loop System Dynamics. Anyway, the chart above shows the output of what we call "zeitgeist tracking", that presents as a "data swarm" (especially as it moves over time) of the relative position of relevant memes to a topic (in this case the US Election), here shown on on two quite useful axes:
Also shown are 2 other blobs of memes - on the Y axis are relevant ones that haven't yet gained much traction (the Darwinian Stew of memes in the culture waiting to find believers - Terry Pratchett's Small Gods describes this issue perfectly) and on the X axis, memes that are second order - they are part of the memeplex but not highly relevant to the topic under examination (nonetheless, they can be useful for splitting out sub-tribes)
We also tracked the primaries at the time, and Trump moved off ahead of all the Republican hopefuls wit gathering pace, and never looked back. That allowed us useful data to to calibrate the memetic algorithms and now we want to see what they predict for the US Election outcome. We have been tracking the US Elections since the Republican Caucus, and Trump has always been in front, even during the Democrat National Convention. If the lessons of the Primaries hold true, he will win - not by much, but he will win, right now our system says he holds the dominant Zeitgeist and the rate of change is not slowing vs a vis Clinton.
Of course things can still go totally wrong, and tonight's Candidate's Debate may be just that, as in memetic terms the debate will be throwing around a number of highly relevant memes via a medium that has a potentially huge Influence. We are quite looking forward to seeing how it pans out memetically!
(Update - 12 hours on, we see no sign at all that Clinton has made any discernable difference in relative positioning yet. It's probably too early to see enough data to be clear until US has had a full day to talk about it, so by tomorrow morning (28 September) UK time.. The mainstream media pundits are already saying she won, but as far as we can see the data is saying it was too close to make any difference to the big picture. The risk for Clinton is that (if Brexit is anything to go by) the MSM tend to suffer from confirmation bias and readily believe their own agendas. We expect to see a lot of attempts by the Clinton media to change the memes in circulation in the next few weeks.)
The essential Caveats
To be sure, this approach is far from foolproof but we are curious to see what it can do. The major pitfalls are:
Firstly, the General Election is somewhat more complex terrain than the standard fare of our system (customer service optimisation, consumer insight, brand messaging, influencer identification etc etc) as the nature of political constituencies give the ecosystem a series "break points" - ie a 55% / 45% advantage for the winning candidate can still result in then losing, due to uneven spraed of support - ie a minority of "over-won" states and a majority of "just-lost" states. And to make it more interesting, the states are not the same size. But this didn't impact the primaries that much (however, that may have been because by the time of voting nearly all the other Republican candidates' share of the meme-space had become relatively insignificant).
Secondly, the Zeitgeist is a measure of the recordable transmission of an idea and we know from following UK elections that people tend to publically talk politically acceptable and privately vote in their own interest, but Trump (in general) is the non-acceptable face so those showing positive interest in him are probably under, not over represented in the public discourse. We do track Zeitgeist sentiment, and pro Trump sentiment is on average lower than pro Clinton, but not hugely so (see above note re acceptable), as neither are particularly popular (or more accurately, both are highly polarised) but strictly speaking meme transmission is Wildean in our experience - what is more important is the scale of the conversation.
Friday, September 9. 2016
....is that they do things like this - Grauniad:
They also banned the poster of the photo when he complained. The photograph was the one above. You may remember it as iconic of the Vietnam era, and it turned opinion against the war globally. Problem is the Facbook algorithms don't like it, nor did the algorithm's handlers as:
"it is difficult to create a distinction between allowing a photograph of a nude child in one instance and not others".
Anyway, after fairly worldwide complaint, criticism and derision, they have reinstated that single photograph because even though its usually considered nakedness and child porn etc, in this case "the value of the photograph outweighs the value of protecting the community".
There is a certain irony in that, probably completeley missed by the Facebook PR team. But it points to the underlying problem, I suspect, that due to a combination of age, education (all tech) and arrogance (who needs to know anything except tech) a large number of Facebook employees* didn't (still don't?) have a clue about why it is important.
The limitation of curation-by-algorithms has been cruelly exposed in all its over-simplicity.
The scary thing is that Facebook is now the world's largest de facto publisher, and the most profitable. If this (and their propensity for political censorship) is the future, it will truly Change the World - but probably not in a good way.
I feel it is almost the responsibilty of the Rest of Humanity to start bombarding Facebook with images and stories that upset their algorithms but are important. Starting with Renoir.....
* Evens money on "the intern is to blame" tomorrow, anyone?
Friday, September 2. 2016
Gartner diagram of the future Digital Workplace
From the Deflating Bubbletime Dept:
A few days ago I read the takedown of Gartner's view on Robotics in Horses for Sources, essentially it was a fisk of Gartner's current trendency to make fairly lurid predictions about our Digital Future - they have predicted that by 2018, more than three million workers globally will be supervised by "robo-bosses" and that one in three jobs will be converted to software, robots and smart machines by 2025, for example.
It's not just Robotics - reading their thoughts on the Top 10 Technologies Driving the Digital Workplace my initial thought was it was a well needed piss-take on the current Digital Everything trope (see the diagram above) but reading the article I realised they were deadly serious. I'm not going to summarise it here, the link above goes to the article if you are interested. Our view is that the actual technologies of the Digital Workplace will look like the diagram above, rather than their dreams of Office Automation 1.0, for quite some time.
No, the real question I'm asking is similar to the one the article on the robotic analysis asked:
So why, pray tell, is Gartner, a respected voice in IT research, continually pounding us with continual scaremongering that we're all doomed to the will of the robot
Similarly, why are there such clearly overblown claims for "Office Automation". What's going on here? These guys are supposed to be analysts, not boosters - Gartner invented the darn Hype Curve, but now they are seemingly invested in filling it up!
To an extent there is a geneal trend in "rspected" media these days to become more clickbaity - I've noted even such august organs such as the Graniad and Economist are much more prone to this these days, and the "Top 10" Listicle style here is another signal. Does this mean the assumption is no one reads real research unless it's lurid - as one of the comments in the Horses article suggested - imply that:
Or is it something else...
"Perhaps they are spewing off someone else's agenda? #followthemoney"
This second point also links up with something else - from about thetime of the O'Reilly "NextEconomy" conference onwards, I do get the impression the "analyst" and other boosting media of the Tech World has been ordered to get out and push the product as the bubbletime starts to deflate.
The valuation of many Unicorns, current and desired, and all that ride on them may depend on it....
More Broad Stuff
Poll of the Week
Will Augmented reality just be a flash in the pan?
Creative Commons Licence
Original content in this work is licensed under a Creative Commons License