Friday, December 2. 2016
The increasingly standard Election Prediction Method of predicting one winner till the total volte-farce at the last minute
After their superb showing in Brexit and the US Election, we bring you the best yet - the Richmond by-election in London. One candidate is definitely going to win, until the last minute when it seems the other is. (Betfair win probability graph, hat tip to Mike Smithson)
Those who don't read this blog may not know that we correctly predicted Trump's election (and had the balls to go public about it 3 days before the election), but what we never understood was what the pollsters were up to - our monitoring systems told us that Trump was way ahead from the get go. Yes there was a spike for Hillary at the end, but it was from behind.
Yet the pollsters consistently showed her (and Remain, and now Zac Goldsmith) way in front till the Shock (hah!) Ending - how come? There are really only 3 possible reasons:
i) Their methods are crap, they are talking to their choir
We originally thought it was (ii), but now we begin to wonder....
Saturday, November 19. 2016
So, news today that Facebook has recanted from its original view that Fake News did not influence the US election - CNN:
"The bottom line is: we take misinformation seriously," wrote Zuckerberg. "We take this responsibility seriously. We've made significant progress, but there is more work to be done."
So why the sharp reversal of views from a typically very canny guy?. Well there are two options:
One is that Facebook, in a Road to Damascus moment, has decided that is has a public duty to start to manage the Fake News Problem. (OK, OK, just joking - this is Facebook after all). But this effort (even the announcement of it) in a stroke gets rid of all those annoying Do Gooders and their Bad Publicity, and reduces too sharp a focus on the ongoing problems of scalabe content management problems it is having.
The second is that it reinforces the impression that Facebook does influence people big time, a necessary story to keep on selling Advertising. The most difficult part of saying that Fake News didn't influence anyone is admitting that Ads on Facebook don't either.
Far better to claim major Fake News Influence, kick a few Alt-Right feeds off (for now), and while the sun of public approval shines make hay with the tacit impression that Ads work, big time.
At one stroke, do well by doing good. Genius....
One wonders what MZ was thinking initially. Oh, what's that? Telling the Truth? Now now, we'll have none of that Fake News here....
Monday, November 14. 2016
Spent some of the weekend reading various analyses of why polling for the US election predicted the wrong candidate. The overall point being made, time and again, was that, in very close races, there is a small difference between the candidates and in this case (and Brexit's case) the margin fell the "wrong" way and the unpredicted side won out.
To which the only real response is something on the lines of "well, they would say that" (aka "bollocks")
Because if, as they are now claiming, everything was so close and within their margins of error, and had been close for a while before the day, then you would have expected one or both of the following effects:
Instead there was a relentless "Clinton (Remain) is winning" story and we went into the final days with a Clinton win being predicted with between 70 - 99% probability from every major polling outfit. How the $^% does that square with the claim now being made that "everything was close and within the margin of error". At the very least - the absolute very least - they should have been giving higher odds of winning to Trump (Leave) then!.
Contrast to what we were seeing in our tracking of the memes on social media, which showed huge support for Trump, with Clinton only really closing the gap in the last month or so. (And that's just social media, many more conservative people tend to be in the demographics that doesn't use it that heavily)
We suspect something else was happening.
There was a rather interesting report after the Brexit poll fiasco, which said that in essence the polling companies saw Remain winning because they wanted to see Remain winning - that there was confirmation bias in a number of ways. Nate Silver said something similar after failing to call Trump in the Republican party candidate elections. At the end of the day all this stuff is open to interpretation, and it seems to us the most plausible explanation is that there was a strong tendency to bias to Clinton in nearly every case where an interpretation option came up. Add to that that everyone else is saying the same thing, and it becomes very hard to remain unbiassed. You have to set some ground rules.
Before we started tracking the US Election, we had been tracking Brexit using a system dynamic prediction model (see here) and as a result of the Brexit analysis we set up a number of rules for tracking the US election. Rule (ii) is very apt here:
(ii) Beware Hubris - assume the gap is less than you think, especially if you believe you [ie your preferred option] have the "moral" advantage
Given that most of the people who do this sort of work are highly likely to be in a pro Clinton (Remain) demographic it may in fact be more than conformation bias, it may even be a complete inability to realise that there Is another interpretation/option. I note with interest that Ogilvy's head of their PR business in the UK recently suggested that his staff get out of London or risk being out of touch.
(Of course, if you follow the Conspiracy theory route, there is a better one - that Clinton's direct influence pwned the media and polling companies)
Friday, November 11. 2016
Lest we forget.....
' "Good morning; good morning!" ' the General said
When we met him last week on our way to the line.
Now the soldiers he smiled at are most of 'em dead,
And we're cursing his staff for incompetent swine.
"He's a cheery old card", grunted Harry to Jack
As they slogged up to Arras with rifle and pack
But he did for them both by his plan of attack.'
- Siegfried Sassoon
Thursday, November 10. 2016
(Tracking the US Election memes - Trump is the big blob way ahead on the right)
On Saturday November 5th we decided to "go public" with our prediction that Donald Trump was in pole position to win the US Presidential Election. We were in a small minority, most polls and researchers were calling it for Clinton with odds ranging from high 70's to near 100%, but we trusted what we were seeing on our Dataswarm analytic engine. The system had worked for Brexit, after all (though we were too uncertain to publicise it then, preferring no egg on face to instant fame). So, suitably caveated, we posted it up. Carpe Diem, and all that.
However, it seemed that the minute we posted it, all the good news for Trump started going bad, which from our self-interested point of view was also bad news. First the FBI investigation into Clinton's emails was cancelled, and we were told her support was rising. Then news came through that she was storming ahead in early voting in a number of key states. If she won we would have egg on our faces, if we retracted the prediction we would too.
(We make no comment about anyone's views about the political outcome, this is all about how the technology worked)
Waking up in time for the 6 am BBC morning news in the UK (5 hours ahead of the US) we heard that Trump had almost won, with a far wider margin than our system had shown was possible. By 8 o'clock most pundits had called it. Trump was the next President -elect.
Our system had got it right - it had worked.
So why had we got it right when nearly all the other polls and pundits had called it wrong? Now we have had a day or so to look at the outcomes, we think there are 4 main reasons.
Firstly, Internet vs human polling. Our system is looking at verbatim Social Media data, from Twitter. We had come to the conclusion while monitoring previous UK general elections that people were more willing to share their true thoughts on social media than with pollsters, especially if their views were "non-PC" (in this case, pro-Trump). After the election we read that the LA Times poll, which had consistently been more pro Trump (and been roundly criticised by nearly every pundit), had been an internet poll, not using people to ask questions, and they believed (and were proved right) that people had been more honest on that. In effect by monitoring social media, we were getting the same sort of uncensored opinions, and in that uncensored world Trump was doing a lot better than the standard polls were predicting. Also, we knew from UK elections about the "shy Tory" effect where people say one thing - typically to look good (virtue signalling as it is called) - in public, and do another at the ballot box (To misquote Phil Ochs, Liberals are 10% left of centre in public, 10% right of centre at the ballot box).
(Update - there has been a rush of Social Media monitoring companies saying they saw the same thing as us (here and here for example), though they seemed a little bit more reticent than us about calling it before the event )
Secondly, the way our system works helped quite a bit. It was initially designed to satisfy a BBC requirement to "Find the Zeitgeist" across its media output, as well as compare it to others' output. To solve this we used a fairly obscure technique that we had become interested in called memetic analysis, that groups memes into groups of fellow travellers (called "memeplexes" in the lingo) rather than look at things one by one, as Boolean analysis forces one to do. We started it going the day after Trump became REpublicam Party candidadte, and by Nov 8th it had crunched a frelevant sample of c 170m tweets and was tracking 4.5m memes. What our system was showing was that from the get go, Trump had dominated the memespace (as he had in the primaries too). In meme theory as originally proposed by Richard Dawkins (who coined the term meme - or cultural gene), the view is that memes colonise your mindspace - so in effect the Trump memeplex was hogging the electorates' mindspace, starving out competing memes. Clinton was not anywhere near. To be sure, not all Trump memes were positive, but in essence Trump was using a "Wildean strategy" (The only thing worse than being talked about is not being talked about).
You can see how this works in the Youtube video of the system tracking Trump, above
Thirdly, we knew from Brexit that the "non voters" were very motivated to come out to vote if someone could credibly promise an "out" from the current political system. Trump seemed to be doing that succesfully. (One can argue about the morality of his tactics, but the effectiveness of the strategy had been proven for Brexit), and we thought it was happening again.
Lastly, Confirmation bias. We knew that after Brexit and the Republican Primaries, UK and US pollsters had looked at why they had got it wrong. They had realsed that their unwillingness to countenance a Leave/Trump victory had made them look at the data from a point of view of what they wanted to see, not what the data said. Given that the US media and pollsters seemed to be even more pro Clinton that the UK equivalents were pro Remain, we suspected there was a Clinton bias in the polling uncounted supporters -
That is why, with Trump just ahead, we thought he would probably win.
For what its worth, we had thought if we were wrong it would be because the vote split would go Clinton's way in marginal districts, due to the reputed strength of the Democrat "ground game", beating the above factors. But as with Brexit, the Trump voters proved more motivated and got out and voted.*
(We did note, in our weasily caveats on the 5th, that Clinton could win the Electoral college, but we thought Trump would have the popular vote. Ironically, it turned out the other way).
*Update - it seems Clinton got more of the popular vote than initially reported on the day (though less than most of the polls thought), this is interesting as we thought she'd get about the same or less. We were still more pessimistic than the pollsters about her chances and that made the difference in the call I guess - though as we point out here even by their own now revealed calculations, the 70%+ chances the pollsters were giving here were not really justified.
Saturday, November 5. 2016
Broadsight's DataSwarm memetracker, 5 Nov 2016
Coming into the final days, our tracking of the memes around the election on Twitter show that in volume terms Clinton related memes have caught up with Trump related memes since we last wrote (see here)
Remember, memes are a proxy for mindspace (as reported on social media, anyway) that various topics are occupying in people's thoughts. We are not looking at what the mainstream media are pumping out, but what people are talking about. There are all sorts of allegations of Trumpbots, and the US mainstream media being "rigged" against Trump, but we are tracking social media output here at vast scale. We did turn the spamkiller algorithms off to see what the raw output looked like, it will be one of the things to look at afterwards to see what their effect was.
Now to be honest we don't know what all this means in terms of election outcome, we set up this project to see how to calibrate a meme tracking system to the on-the-ground results in a US political election. What we can say is that, since we last wrote, the volume and reach of Clinton related memes have caught up with Trump related memes overall, so he no longer has quite the dominance in social media memeshare. But this is not necessarily a good thing for Clinton, a not insignificant rise has been in the memes around topics like "wikileaks" and "Benghazi" (see smaller orange dot in centre) and this is not all complimentary to her. Rather interestingly the Bernie Sanders "feelthebern" movement is still trundling along at scale (orange dot on top right), again this is not necessarily good news for Clinton. Trump's women-groping problems have not really lasted as an item, in essence there was a lot of noise among a relatively small number of people for a limited time. The current fastest rising complimentary Trump mean has been on immigration.
We also saw no real impact from the debates in terms of sentiment shift, just more traffic. Again, how one interprets this is uncertain but our hypothesis is that the vast majority of people talking on social media knew who they supported before and just rallied to their candidate, and the undecideds who talked online are a small % so didn't really shift the dial.
What is often useful in mapping the "Zeitgeist" - the amalgamation of all the small trends and shifts - is looking at "the rate of change in the rates of change". Trump sentiment remains roughly the same, he has followed a Wildean strategy (the only thing worse than being talked about is not being talked about) to dominate the memespace, so maybe it will work for Clinton too as she has caught up, but quite a bit of what is driving her rise are negative memes.
The other thing we do know from tracking the last two UK elections is that "un-politically correct" opinions are under-represented, people are shy of stating these opinions publicly but do vote for them. (To misquote Phil Ochs, liberals are 10% to the left of centre in public and 10% to the right of centre in the voting booth). Trump may have let more people feel they can say what they really think, but we'd suspect there is still a residual "shy Trump" voting bloc. Also, demographically many of the people who are natural Trump supporters are less likely to be on social media. Now whether these will get out and vote is another matter, but it was these people coming out to vote (and failure to correct for "shy brexiteers") that caused the upset in the UK's Brexit vote.
(Update - signs are that in Nevada the Clinton machine is getting the early vote out and the Trumpites are staying at home)
As mentioned in our first article, Brexit differs from the US election in that it was an overall referendum, i.e. every vote counted equally wheraes in the US election it depends very much where a vote is cast. A vote for Trump or Clinton above those needed to secure victory in any one particular ward is essentially a wasted vote, it doesn't count towards the outcome.
But if we may speculate on one possible interpretation of the meme data - Trump has dominated the memespace from the beginning, and is probably more represented in the non-online population. So it may be that he will have more people actually voting for him, but they may be in the "wrong" places. So there may be a Clinton win, but with more actual votes for Trump.
Anyway, we shall know in 3 days and then we will be able to have a good look at our data nd how it mapped to real outcomes
Saturday, October 22. 2016
Yesterday some major sites were brought down by DDoS attacks, and the machines that were used were "dumb" IoT devices with poor or no security - El Reg:
Today, a huge army of hijacked internet-connected devices – from security cameras to home routers – turned on their owners and broke a big chunk of the internet.
In the old days the delay between prediction of bad things happening and some occurrence event in new technologies were several years, if not over a a decade - nowadays its months.
Still, interesting is that, while Twitter was brought down, its share price went up 7% - possibly people only realise how essential it is when its not available (or, as others suggest, some are prepared to pay to keep it switched off)
The more serious issues are around the clear problem the IoT poses (i) to the Internet, and the need (for now) to keep these services off the 'Net and (ii) for countries, its clear that this is a very vulnerable point for cyberwarfare
Sunday, October 9. 2016
We predicted 2 weeks ago that the media would throw a huge amount of memes at Trump, but never imagined that it would come to this!
At any rate, so far it hasn't really made a difference to the memetic makeup overall - yes its the story of the moment, but the anti and pro isn't hugely different on an initial analysis, so it seems like the story is falling along pro and anti Trump supporter lines. If it is changing the minds of independents, there is no sign as yet.
Anyway, with the debate tonight this should all be exacerbated massively, so it will be interesting to see what shifts occur
Just a few hours in, not seeing any seismic shift yet in the incoming data in terms of any major mind changes, but the volume of comments on Trump's pre debate issues seems to have dropped for now and been replaced by other topics that came up in the debate.
Friday, October 7. 2016
Interesting study on the (lack of) difference between cities' medieval and modern usage from SFI:
In both medieval and modern European cities, larger settlements have predictably higher population densities than smaller cities.
Although, one wonders if these cities lack of change is due to them being pretty much the same physically over that period. But anyway, the lesson is that 700 years apart, people still behave much the same way.
There is a lesson there for all the "New Ways of Working / Living / Communicating" evangelists....
Thursday, October 6. 2016
Motherboard has an article on a recent DDoS attack that, for the first time used IoT devices as the slave intelligences that launched it - it's over here written by one of the people we really rate on security, Bruce Schneier
But the really critical point in the article, in our view, is this:
In other words the added value per unit is so small it is simply not worth putting in the level of security for these systems to be safe on the Internet. Not only that, many of the devices don't have enough intelligence to be secure (even putting any form of non-trival security in a Raspberry Pi or Arduino is a challenge)
And even if you could, the installed base of dumb IoT is so large that its going to be a problem for a long time - as Motherboard notes, it's a market externality:
.tIn other words, there is ready made market failure emerging. The answer has to be intervention by authority, but if it is government intervention it is only countrywide (though a grouping like the EU could have more impact) but it's clear a lot of countries won't do this, or if they do will do it with minimal rigour, as it costs money and there is litlle value to them.
(Page 1 of 284, totaling 2831 entries) » next page
More Broad Stuff
Poll of the Week
Will Augmented reality just be a flash in the pan?
Creative Commons Licence
Original content in this work is licensed under a Creative Commons License