Calling the General Election

Posted on : 31-03-2015 | By : richard.gale | In : General News

Tags: , , , , , ,


With the General Election now only 37 days away, can we harness new technology and the “wisdom” of social media crowd to help call it?


Big Data and Sentiment Analysis


In the run up to the election you will hear a lot about the power of big data and sentiment analysis, for example in this announcement from TCS.

The technology behind this and similar projects is undoubtedly clever, although I do think the state of the art in sentiment analysis is being overplayed – I recently attended a talk by a professor of computational linguistics who revealed that it helps with their currently not very effective sentiment analysis of twitter if people include emoticons in their messages!

However this attempt to benefit from the “Wisdom of the Crowd” (often cited as a rationale) is doomed for a reason that is as old as computing itself:


“Garbage in, Garbage out”


The sample set of political tweets will largely encompass two analytically unhelpful groups; political activists, who are the least likely to change their vote and the young who are least likely to vote in the first place!

Indeed, at the close of the Scottish referendum campaign, the SNP were convinced that they had won by a good margin off the back of analysis based partly on trends in social media.

(This is different to the US Republicans in 2012 who, at the close of the presidential campaign, also thought they had won but based on no data at all from the non-working system that they had built. This is not relevant to the current discussion except that both instances led to a narrative of the vote being “stolen” developing amongst true believers but I include the aside as I think this article is required reading on how not to roll-out a project to non-technical users)


Opinion Polls


What about the more familiar opinion polls?

As the polling companies are always keen to tell us, polls provide a snapshot and not a predication.  This is true of course but the snapshot itself is not really a snapshot of how people will vote at that moment in time. It is a snapshot of how they answer the question “which party would you vote for if a General Election were held tomorrow”.

They subsequently give different answers when asked specifically about their constituency and different answers again when the local candidates are named. Liberal Democrats have found in their private polling of their stronghold seats (a fast dwindling set!) that naming the local candidate can make a 10-point difference in their favour (the incumbency effect) and indeed this is the glimmer of hope that they cling to this time round.

What opinion polls also cannot factor in is anything more than a rudimentary use of past knowledge of voter behaviour – polling firms do differ on how they allocate “Don’t know”s, either ignoring them altogether or reallocating a proportion of them based on declared past voting.

Of course this gives its own problems as people are inherently unreliable and misremember. For instance, when asked, far more people claim to have voted Liberal Democrat in 2010 than the total number of votes the party actually received.


Betting markets


Of course the one source that can take all the above and more into account including past knowledge and contemporary analysis is the betting markets.

In pre-internet days, the above was still true but easy access to information and on-line betting has supercharged this in terms of both numbers and overall quality.

In this case, the “Wisdom of the crowd”, often touted for things like sentiment analysis, does actually hold true because the crowd in this case are actually wise (unlike the twitterati), both individually and collectively.

Political betting is a niche pursuit and as such attracts both amateur and professional psephologists along with those with “inside” knowledge. This means that the weight of the money in the market is quite well informed.


Past-it First Past the Post


The importance of inside knowledge is magnified by a creaking voting system that means that national polls and sentiments are all well and good, but the real result lies in the hands of a hundred thousand odd voters in a handful of marginal constituencies.

This means that those will real insight are those on the ground and this time around, the proliferation of smaller parties eating into each of the main parties’ votes makes the situation even more volatile and local knowledge even more important. The constituency betting markets will be made by political activists on the ground with access to detailed internal canvass data.

So my advice would be to ignore the siren call of the new (social media) and the reassurance of the old (opinion polls) and just follow the money, the informed betting money that is.


This article was authored by Andrew Porrer of Heathwest Systems and represents his personal opinions. Andrew can be contacted at