Entries by Alan Patrick

Prediction is always difficult, said Yogi Berra, especially about the future. However, it's something we do quite a bit of (have you seen our work on election prediction - Brexit, Trump, UK 2017, Germany etc - we got them all right) so I went along to the NESTA Futurescoping 101 event on this last night see what they had that I could steal to offer.

There were 4 talks on various ways of predicting the future, or more accurately 4 ways of making scenarios (reading entrails of dead animals is still outre, but with the return to the Age of Unreason we are seeing I wouln't be surprised it it makes a comeback, albeit maybe with dead plants). Anyway, in summary they covered these 4 approaches:

Speculative Design - Cat Drew

Speculative Design is primarily the use of physical things to provoke debate, rather than using documents/words/numbers etc. For example Cat did one session on blockchain visualisation using a blood transfusion drone made of felt and other blue petery stuff to stimulate thoughts of what (electronic vampire?) blood donation services may be like.

Whether you agree with the approach or not (In my experience with Tech you do often do have to build a Future Thing for people to be able to think concretely about, and this looks useful for that - but it's risky to abstract the results too far without doing the hard sums....) she also noted some things that are important to scenario building no matter what your approach, viz:

- multidisciplinary teams
- can't be fantasy, don't make it too remote
- base it on evidence of research
- Suspend belief but no trickery

Games and Simulation - Florence Engasser

Simulation and game design are more my comfort zone area, so to an extent I knew more abot this approach than the others - anyway, it was all sensible stuff and Florence noted the main problems one hits with respect to modelling complex and unpredictable things (aka most future problems), and the benefits. As she notes, simulation helps in a number of ways:

- Model the trade-offs and outcomes (and some aspects often become clear in the dynamics of the model)
- Visualisation makes it easy to see things
- Collaboration - seeing things the same way (modelling often is a good way to bring people up to speed with a situation)
- Experimentation - safe place to experiment with changing factors and assumptions

Her summary of how to make it work was good, I liked the checklist of what a simulation or game needs, not so much the first 3 which are almost a given, but the last 3 which are often neglected by techie types, and are part of the "soft factors" that are so essential:

- objectives (getting these right is non trivial in my experience)
- mechanics (wish she'd said more on what they do here)
- balance between complex and simple (again critical for model acceptance)
- narrative for the scenarios
- choices people can make
- a journey, putting it all together

What I didn't know is there are a bunch of Nesta games in existence or production:
- Innovate
- Superbug
- Consortium


Inclusive Place based futures - Harry Armstrong

This was about how to have inclusive conversations with people about changes to their cities, buildings etc. Typically top doesn solution go awry, it's better to get the locals involved for a variety of reasons and avert failure:

- Appeal to democratic/ethical notions to get buy in
- To drive collective action needs a shared vision
- Driving nclusion of a large number can show wider benefits to make the project more attractive, and show flaws and avert failure that a small group won't see

Examples used were the Newcastle 2065 future city strategy and the Island of Aruba's 2000+ strategy for itself. Looking at the latter:

- Aruba in the early noughties took at collective approach for its future strategy and engaged 60% of the island's c 100 000 population,
- They used the Appreciative Inquiry approach (Discover, Dream,Design, Deliver )
- Then went on with scenario planning approach, across 10 workstreams

What interested me about this example is that Aruba had a change of Govt during the process and teh new people wanted to change or cancel some of the streams, but concerted citizen action forced their continuation - now that is buy in

Lessons about this approach are salutary:

- works if it's authentic,
- don't do it if you won't act, expectation raising can be risky
- needs buy in from major players
- need neutral spaces for trust
- need to manage "the usual suspects" being involved

DiY Scenarios - David Finnigan

The approach here is to acting out a scenario as a short play or event. His view is that a scenario is an Imagined vision - doesn't have to be correct, but you need a range including a best and worst. So far same old same old, but his approach is to use the play as a vehicle to slip in the assumptions and outcomes as just part of teh narrative.

This evening he acted out making a nature documentary in the future, but members of the audience had to answer assumptions on the role of nature reserves in the future, impact of tourism, and the various other assumptions about the future, using the play-acting as a vehicle to sideways slide in the assumptions of the real scenario, ie the setup conditions as part of the building the story, and the edenouemaent is anarrative end of teh reults of the various 2x2 essumption matrices.

Overall impression - the evening seemed a bit heavy on the qualitative and light on the quantitative, my experience is at some point someone with the power of decision making says "show me the numbers behind the grand vision" (which is where simulation type modelling has an advantage), so while I could see how some of the more narrative approaches could really help the envisioning.

I think - as Cat noted in the first talk - always base your scenarios on evidence of research.

So, Twitter has expanded the 140 character limit to 280 for all users. I've used it for a week or so like this, and overall what I've found is:

- It's useful as you don't have to artificially concatenate twts, and you can be clearer in what you mean
- If you stick to around 140 characters and stick to the point, it flows as well as the original.

But...
- Some users (too many) put no new, useful information in the extra 50%, the signal-to-noise ratio is effectively halved
- A pageful of 280 character twts is half as much information as one of 140 character twts, and given too many are full of woffle it can be even worse.
- A 280 character tweet needs some formatting functions to be easy to read.

Now no doubt Twitter could put in basic editing functions, but this seems to be adding complexity to a system where the major attraction is speed and simplicity.

IMO Twitter is for fast-to-read bullet point information, not micro-essays, and especially not micro-essays by the verbose.

Lest we never forget, not all unknown soldiers are dead

(Rogers cartoon, Pittsburgh Post-Gazette)

Covered in boingboing and LabSix- what happens when people start to spoof image processing algorithms..Can you make a cat look like guacamole?

Three researchers from Kyushu University have published a paper describing a means of reliably fooling AI-based image classifiers with a single well-placed pixel.

It's part of a wider field of "adversarial perturbation" to disrupt machine-learning models; it's a field that started with some modest achievements, but has been gaining ground ever since.

But the Kyushu paper goes further than any of the research I've seen so far. The researchers use 1, 3 or 5 well-placed pixels to fool a majority of machine-classification of images, without having any access to the training data used to produce the model (a "black box" attack).

Must admit I'm intrigued and a bit heartened, there may be a way out of the continual CCTV snooping world that is emerging after all. Research paper is over here .

Latest news on the Russians on Facebook brouhaha from Recode

At Facebook, roughly 126 million [or about half the voting population] users in the United States may have seen posts, stories or other content created by Russian government-backed trolls around Election Day, according to a source familiar with the company’s forthcoming testimony to Congress....

Google, which previously had not commented on its internal investigation, will break its silence: In a forthcoming blog post, the search giant confirmed that it discovered about $4,700 worth of search-and-display ads with dubious Russian ties. It also reported 18 YouTube channels associated with the Kremlin’s disinformation efforts, as well as a number of Gmail addresses that “were used to open accounts on other platforms.”

And Twitter will tell Congress that it found more than 27,000 accounts tied to a known Russian-sponsored organization called the Internet Research Agency:

There is a bit more from NBC:

Facebook says in the testimony that while some 29 million Americans directly received material from 80,000 posts by 120 fake Russian-backed pages in their own news feeds, those posts were “shared, liked and followed by people on Facebook, and, as a result, three times more people may have been exposed to a story that originated from the Russian operation

.
Beforehand, Facebook had said they had found about $100,000 of Russian spend on the platform over the election.

To put these numbers into context, the Clinton campaign spent about $1.4 billion on media during this period, and that is without the free publicity from nearly every US media organ being mainly on the Democrat side. The Russian media pieces would have been swamped by this spend and volume, never mind all the rest of the pieces of media on the average social media timeline. The Trump campaign spent c $1bn itself, not on Russians.

A BBC article says Facebook estimates that:

"just one in 23,000 or so messages shared on the network were from the Russians".

That's a ratio of 23,000/1 (about 0.004%), and if you compare the ratio between the Russian spend of c $100k vs the c $2.5 bn spent by Trump & Clinton (25,000/1) then they are no more effective in getting their message transmitted than any other piece of media on average. If the Russian spend is underestimated then they are in fact quite poor at writing attractive content, unless of course the volume is also underestimated in which case it approaches this average again. .

The election was won on c 55,000 voters voting for Trump in marginal wards, of a total of c 130m voters. about 0.04% of all voters. So for the Russians to have "stolen the election" with 0.004% of all Ads, their Ads must be 10x as persuasive as all others just to be have an equal chance. Or they have to be extremely targeted. But from the above, it seems the Russians reached c 50% of the potential voting population or about 130m people , so doing the paper-napkin maths at the most optimistic (The 130m messages seen went only to the "right 50% of the 130m who actually voted) so 23,000/1 is now 11,500/1, about 0.02%, so they are 2x more targeted and thus have to be only 5x as persuasive.

So in order to believe the Russians "stole the election" you either have to believe that either:

(i) They are extraordinarily persuasive copywriters and great targeters to boot, and can do wonders on a shoestring, beating the best US Ad agencies that Democrat money could buy (they bought the best) or

(ii) Facebook (and the others) are massively underestimating both their spend and % of all messaging, but it has to be at least 10x more just to get parity on reaching those 0.04% of marginal voters who turned it for Trump, never mind persuading them

Or alternatively, you have to believe they were a pretty marginal force in the election (albeit with intent).

Incidentally, we used our data analytics technology to predict Trump's win (and a few other recent elections too) correctly, and we did it by analysing social media support for Trump vs Clinton. It was highly predictive, forecasting a neck and neck race, and we edged it to Trump based on small signals. We're pretty confident our system is fairly good at negating the obvious bots (which are the ones that operate in volume) but there's no doubt we saw some Russian Fake News in the results. However our main impression is by and large people knew which candidate they wanted (or quite frequently, which one they most certainly did not want) and shared media that agreed with them in their own filter bubbles.

Although we didn't track the Czech Election with the full system (unlike these 5), we have mapped it to our OECD political predictive model which we built after the Norwegian and Dutch elections. It states that:

1, The main Centre-Right party moves considerably more Right, adopting quite a few of the Far-Right policies and narratives. This ploy may lose some of its more centrist supporters (to who though?) but it prevents the Far Right from taking far more right wing supporters. If it doesn't (Germany) then the Far Right takes a larger share

2. The Centre-Left is decimated, their various "Deplorables" classes - traditional working class voters - go both left and right. This leads to a rise in the Far-Left numbers. The hard to predict move is the Centre left voters who move rightwards, as there is no natural home for them at the moment, and they seem to scatter depending on local conditions.

So far it's The Czech Election mapped to this, with one "known" exception we've seen before (the "French Disconnection"):

- Parties closer to EU liberal establishment values were left massively depleted. The ruling centre-left Social Democrats (CSSD), saw its share of the vote tumble to become the sixth-largest party.

- Far-right and anti-establishment groups made gains in the election. Far Right SPD support was up to 10.6%, and The Pirates will make their debut in parliament with 10.8% seats. The centre-right Civic Democrats second with 11% each.

- But, as with Macron in France. a populist centreist party, ANO (Yes) collected a share of almost 30%. Our model predicts the Centre left moves right and left, but in most other countries there has not been an easy centre-right choice - except France with a populist centre-ist, and now the Czech Republic.

And as elsewhere, more parties with a wider range on the polltical spectrum are in the parliament, making it harder to put together larger majority coalition.

Resulting setup is:

ANO (Yes): 29.6% +11%
Civic Democratic Party: 11.3% +4%
The Czech Pirate Party: 10.8% +8%
Freedom and Free Democracy party (SPD) : 10.6% New Entrant +10.6%
Communist Party of Bohemia and Moravia (KSCM): 7.8% -7%
Social Democrats (CSSD): 7.3% -13%
KDU-CSR (Christian Democrats) 5.8% -1%
TOP-09 (Liberal) 5.3% -7%

More is certainly to come, as the EU still refuses to shift it's position. As mathematical historian Peter Turchin points out "the governing elites of the EU behave as though they all believe these disintegrative trends that I and others have written about are just a "blip". The more they believe that, we predict the less it will be.

We always believed Bitcoin was overpriced, now it appears that the cost of mining one Bitcoin is very, very high in energy terms. From ZeroHedge - "James Stafford, editor of Oilprice.com not only does the math, but explains the energy-driven geographic arbitrage currently driving bitcoin mining".

The bitcoin boom is well and truly underway, and investors are constantly looking for new ways to gain an advantage in this space The best way to do this, it seems, is by cutting the energy costs of mining this precious commodity. The bitcoin mining industry consumes 22.5 TWh of energy annually, which amounts to 13,239,916 barrels of oil equivalent.

With 12.5 bitcoins being mined every 10 minutes, that means the average energy cost of one bitcoin would equate to 20 barrels of oil equivalent.

At c $59 a barrel at the moment that's about $1,200 to produce each bitcoin

With Bitcoins currently valued at about 100 barrels of oil, there is still clearly a profit - but two things are not clear:

1. So, who is paying this bill? The yearly cost of the energy necessary to mine Bitcoin determines its economics. Cheap electricity is exactly what made China the Bitcoin mining king, but the economics are now moving to naturally cold areas like Iceland. But why pay for mining yourself? We read of increasing attempts to spoof your personal computers to mine bitcoins.

2. Where are the Green/Climate lobby in all this - they are usually after Energy intransigence like a very noisy rash, but have so far been very quiet on this, but if you search the Google there is nary a peep. Maybe they know that making a fuss will piss off a lot of their normal support and funding base?

We have not been monitoring the Austrian election, nor predicting it formally (and correctly) as we have done for the major elections from Brexit onwards. However we did predict that most OECD countries are moving to a specific model, viz our Euromodel (initially built after the Dutch Election in March. based on what we saw in Brexit and the US Election):

In essence, it is a move away from the status quo, via a movement to the poles - right and left. How it plays out depends on the country, so far in the northern European and US it has mainly been a major shift to the right and a smaller one to the left. It may well be the reverse for Mediterranean countries, but in northern Europe the model predicts:

1, The main Centre-Right party moves considerably more Right, adopting quite a few of the Far-Right policies and narratives. This ploy may lose some of its more centrist supporters (to who though?) but it prevents the Far Right from taking far more right wing supporters. If it doesn't (Germany) then the Far Right takes a larger share

2. The Centre-Left is decimated, their various "Deplorables" classes - traditional working class voters - go both left and right. This leads to a rise in the Far-Left numbers. The hard to predict move is the Centre left voters who move rightwards, as there is no natural home for them at the moment, and they seem to scatter depending on local conditions.

Thus this is what we expect to see in Austria, we shall be keeping track.

Update - as of c 5pm UK time Sunday, the following early results are being reported by by Austrian broadcasters:

People’s Party 30.2 % (Center Right)
Freedom Party 26.8 % (Right)
Social Democrats 26.3% (Center - Left)

Update - Monday c 5pm - Final results are

People’s Party 31.4 % (Center Right)
Freedom Party 27.4 % (Right)
Social Democrats 26.6% (Center - Left)

The Right Wing Freedom Party has made gains, and the Centre-Right People' Party has moved more right in order to counter them making even greater gains. (Stipulation 1 of the Euromodel). This is also lowest the Social Democrats have polled ever (though it is hardly changed from the last election), but they are now the 3rd party and it appears they will not be a part of the governing coalition for the first time ever and that represents a major drop in their influence (Stipulation 2 of the Euromodel) though they have been saved further ignominy by Austria's Greens (our model says a shift to the far left is the main flow, Austria has confounded this a bit by having a Far left implosion)

There are the usual protests of "dirty campaigning" and "fake news", and this is a recurring refrain across every election since Brexit. The reality is however, as even former US President Obama has alluded, that there is a major sociopolitical trend happening across the OECD. There are also headlines of a "shock result" whichh by now is very odd, as we have seen it in elections in country after country over the last year or so, and there will be more to follow unless some major structural changes are made in the EU and even more broadly in the OECD.

Big picture, Democratic OECD countries seem to reaching c 50% of the population who believe the current system is not serving them, and these are voting against the status quo and driving major changes country by country. How and why they do it depends on the country, though there are a lot of common traits and issues. To mitigate this trend however requires these countries and their greater institutions like the EU to make fairly major changes to their status quo structures, which so far we see little sign of - so these "shock" votes will probably continue.

Above - system map of the US Election dataswarm in process. Trump is in front - this was at a time the mainstream US media and polling was saying Clinton was far in front, and we then knew an upset was very likely coming.

Using our systems we have managed to predict 5 elections (strictly speaking 6 as the French one is 2 rounds) correctly over the last year, 3 very much against all the polling predictions (Brexit, US, UK 2017) and one that still produced "shock" results (Germany). We went public on 5 of them, before election day in each case. (The bullet points below link to blog posts on our DataSwarm site of each of the individual election prediction summaries)

• Predicted Trump win – we went public with our prediction 3 days before the election. The polls said Hillary was a shoo-in, but the system saw a close race and a dynamic support base that Trump had built up from the beginning and plumbed for him.


• Predicted UK election – went public 2 days before. The system predicted a hung parliament, we got one. Polls were predicting a major Tory win.


• Predicted French Election (both rounds) – went public 2 days before both. Got both right. System underestimated scale of Macron win in second round, but in our review we learned how to more accurately predict future voter flow from the system's memetic linkages - major learning point bonus!


• Predicted German Election – went public 2 days before. Predicted AfD surge and CDU crash better than any polls.


• Brexit was an initial side project, done with a bit of social media sampling and system dynamic modelling, but as that worked and we got it right - despite all the poll opinions being completely the opposite - we decided to carry on with other elections as as we found that a "known" outcome at a point in time allowed us to calibrate the algorithms quite well.

We find that with commercial work measuring aspects like "sentiment" or "influence" and similar can give rather nebulous results. Measuring the true picture is not always simple. How do these really work, and how do they relate to what we can measure? Using known outcomes like election would really help calibrate these metrics, and we could show our clients how our system definitely was working correctly. Besides, if we got a few elections right we might open up a new area of business!

Each election taught us something new about the way the algorithms should work, and what needed tweaking or expanding, and we learned some new insights about how human behaviour in reality maps to the metrics you can "see" with social media analytics. We also learned quite a bit about how media works in persuading people, and how "fake news" et al operates. We also proved to ourselves that the system was good in 2 English speaking cultures (US and UK), and could also operate in French & German language and culture - and could crunch tens of millions of units of social media data.

We also had quite a few insights about what is happening at a population level politically, and our views differ quite a lot from a lot of the "conventional wisdom" we saw during and after the elections.

These are some of the high level takes on each election:

Brexit, 2016


We did not put the main system on this, but used a series of small data samples over the election period, and then built a fairly simple system dynamic model to predict outcomes of a 2-horse election race. Turned out it worked, and against all the conventional wisdom and poll predictions to boot. The Remain camp (the favourites) seemed to go in with a static strategy and refused to shift it when it was clear it was losing. In essence, Remain's arguments, exaggerated by "Project Fear" style messaging, were increasingly being perceived by neutrals to exaggerate the risks and that led to an increasing resistance to their message and it gave a foothold to the pro-Leave media to start to land some telling blows (with similarly exaggerated, but more positive claims). Cycle this through a few times, increase the hectoring volume as Remain started to panic a bit, and more and more people just switched off to these messages. Also, the increasingly vicious insults on Leave voters (more on insults later in the paper) alienated many people. The sampling showed increasing support for Brexit and it was increasingly clear to us that Remain were increasingly mortally wounded and would lose (Probably - there is always a range of error in these things).

The US 2017 Election

We tracked the primaries to "train" the main system, and it correctly predicted Trump & Clinton wins, though Sanders was a major challenger to Clinton for a very long time. We don't count these elections as some of our predictions, as we never predicted them per se (Maybe we should in future).

In tracking the actual Presidential election race from August – Sept 2016, we saw Trump was winning from the beginning, we believed the polls were wrong and suspected another upset was coming. Clinton had caught up a lot by the end, but the system was saying Trump edged it. We thought it was going to be very close, very certainly not a Clinton barnstorm as the polls were all saying. We had seen this in Brexit as well so we decided to trust the system. We gulped and went to press, calling it for Trump, and he won. (Only just, as the system predicted). Why did we see what was right while the polls got it so wrong? We think there were a number of causes, we think we were getting a more realistic picture via social media, but frankly another reason seemed to be the pollsters themselves - they seemed unable to believe what the data said.

The UK 2017 Election

The Tories were supposed to have gone into the UK election with a major lead. We saw no huge Tory lead, ever, they were just a bit ahead. In week 1 the LibDems made most of the running but quickly fell away as the main parties got going, After another week or so Labour started to close the gap on the Tories. Labour's Manifesto was a step change - from about then on Labour started to gain on them much faster. As with Brexit, the Tories seemed to go in with a static strategy and didn't shift it when it was clear it was losing ground. The only topics to "break the surface" as very influential were Brexit and the NHS, and Scotland in far 3rd place. All others were in the noise, it wasn't the Economy, stupid. From about 2 weeks before election day our system was predicting a hung parliament. Now here come the embarrassing bit. Us data analytic humans messed it up, we "knew" from the UK 2010 and 2015 elections that social media was still a bit left leaning so we over-compensated for that and called a range, from hung parliament to a stay the same outcome, with a reduced majority as the most likely outcome. It was a hung parliament, the system was bang-on, and we decide from then on to "trust the system" (We were still far closer than any polls by the way, they were all predicting an enlarged Tory majority)

The French 2017 Election - round One

This was more complex than the previous 3 elections we had tracked in that there were multiple candidates, but with support levels far closer to each other than the US primaries where clear winners emerged. Our system indicated Macron and Le Pen were front runners, Macron had more support than Le Pen, and it was proved right. Another correct call to us. But we had an interesting lesson here. Quite close to the end the system was saying another candidate, Hamon, looked like a strong competitor, even potentially leading Le Pen. The French polls, which are usually very accurate, said he had started well but was fading fast. When we looked at our latest daily data he was indeed not near the front. It turned out his support had fallen over the first round but our algorithms were hanging on to too much history, over-weighting his early high ratings. That was a very useful lesson in algorithm trend tracking calibration!

The French 2017 Election - round Two

It second round was a two horse race, as with Brexit, the US and the UK elections. Easier going for us. Our system indicated Macron was in front, Le Pen was second, and we predicted he would win and we got it right. But, the system said Le Pen was closer behind than the eventual polls showed. This was the only time in all the elections that we tracked that the polls were more accurate than we were. So we started to backtrack through the system's calculations and found something very interesting. We found the relationships of other candidates to Macron or Le Pen predicted the relative shift of that candidate’s supporters to the two finalists. Summing the relative shifts of those voters gave a far closer split to the actual election results. We had just stumbled over how to analyse future voter intent. We did not have this problem with Clinton/Sanders, probably because there was more time between primary and final election so voter shifts had largely happened by the time voting started.

The German 2017 Election

Tuning the system using the lessons learned in the French rounds helped hugely, as again it was a multi-horse race which is more complex. We called the German election pretty well, given this challenge. We predicted both the “shock” AfD jump and CDU/CSU drops correctly, against the polls’ predictions. Our overall error % was the same as the polls on the day. Why? Most of our error was due to another "fast faller" - the SPD - and the system, despite our tweaks for Monsieur Hamon in France, still over estimated the early history. and we over-estimated SPD support from this history. More tweaking will be required on this. Another point - we found underlying AfD support is larger than the election would suggest, but that influence on voting outcome was lower than the other parties on the day. We think true AfD support is several % points higher. This will probably come to light once they get into parliament, and get more exposure. Assuming that is, they don’t fall apart instead – their largest personality (by far), Frauke Petry, left the day they were sworn into the Parliament!

The System

The system's logic has in our view proved it is dependable over a range of conditions, languages etc, and that the algorithms are seeing and indicating a true picture and are fairly accurate in their predictions. We have learned a lot about calibrating some of the more vague terms such as sentiment and influence. We had also built our own hardware stack (we found commercial cloud systems too slow / too expensive for this sort of data crunching) so we were quite pleased it held up to the task. We also noted that we got these results not by spending long hours tramping the ground, or the focus group grind, but by our computers crunching tens of millions of social media data items. The only walking we had to do was to the coffee machine.

Political Trends

There is a huge amount of interesting data we now have to look at political trends and insights, and to derive lessons for future elections, and we are going through it all with great interest - but some general points that were immediately apparent are:

• In all 4 countries, over 6 elections, there are a large number of voters - traditional working class and many traditional "non voters" - who turned out for the candidates who promised a shift away from the status quo. Politicians ignore this trend at their peril.


• There seems to be a decline in the centre of politics, and a shift to the poles of each political spectrum. Various countries' electoral systems and parties handles this in various ways but the trend is marked.


• That "insult" thing we mentioned above - casting ad hominem insults on the other sides' voters seems to be a bad idea. We noted in all elections, in various ways, that insulting the other sides' voters just created more of them. So a lesson for future candidates - don't call the other side's voters "Deplorables" (and worse) - that is guaranteed to generate more of them.


• Humans are humans, culture, language, electoral system are lower order variables - 6 elections, 3 languages, 4 cultures, the same approach worked.


• In similar vein, the system also worked across various alleged levels of "Fake News", Russian involvement, Electorate manipulation, voting machine rigging, Bot spoiling etc etc, and pretty much delivered a good result each time. The system handles most bots quite well (in our humble opinion), but we are starting to think that - given our algorithms were basically the same and produced the same approximate quality of predictive results across apparently highly varying levels of all these spoilers over a number of elections - that "Fake News", "The Russians" et al were not the decisive factors in any of these elections that some believe they were.


• In addition, there is some form of major systemic problem with US and UK polling, the French and German polls mapped to the final results far more closely.

Lots more to come on this aspect over time though.....what we can say is that we disagree with quite a bit of the "popular" analysis of the political press and pundits about why various things happened, and what it all means going forward.

They say predictions are dangerous, especially about the future. For better or for worse we tracked Brexit, the US Election, and the UK 2017 election and got them right despite the polls. We also tracked the French one and got the right guy, but underestimated the support level - see over here)

Well, we have been tracking the German election using social media since February, and 6.8 million tweets later we have some predictions on Friday we made an attempt to predict the German election using our own data analytics systems that analyse social media, and the results for the 7 main parties were were (assuming the 7 main parties are c 95% of all votes:

AfD Election outcome = 15% (- but a wide range variance of c 12 - 18%)

CDU 25%

CSU 8% (CDU + CSU = 33%)

FDP 7%

Grüne 7%

Linke 9%

SPD 27%

The range is typically +/- 2% (except the AfD, as discussed below)

The usual caveats

Note that German Twitter usage is relatively low as a % of population, so (as was true in UK and US in the low penetration years) it may still be left leaning if it is still mainly early adopters, so our predicted SPD and Linke figures may be a little high by a few %. Ditto, the FDP figure could be under-represented.

But the real question mark is over the AfD. Why is our system showing a most probable result so much higher than the polls, and saying there is a wider range. (Incidentally, it shows there is even a remote chance it could be greater than 20% so could there be another upset?) The issue is the sheer volume of support (see blob picture above) yet lower possibility of votes (it's quite far to the left of the CDU and SPD on a log scale), i.e. about 1/2 the impact.

Or are we just wrong re the AfD - the polls are saying 11%?

(Update - many adjusted to 12% on Saturday after we submitted this)

In brief, when it comes to these "alternative" party outcomes the system has seen the "right" picture in all previous elections, and we are betting it will do again. Our ingoing assumption is the the "Shy voter of unpopular party" effect is in play, because it has been before in similar situations.

(And it ain't about bots...bot activity has been very low (and besides our system is quite good at ignoring them)

The rest of the discussion about what we are seeing and thoughts about why are over here on our Dataswarm site

(Update) As of Sunday evening exit polls it's been none too shabby though it seems the SPD has undershot at the benefit of the smaller parties and the AfD is c 13-14%, not 15% as we predicted, but higher than poll predictions. We shall see by tomorrow