Predicting referendum results in the Big Data Era

Big Data are large data volumes characterized by the 8 Vs: ‘Volume’ [1], Variety, Velocity [2], Variability, Value, Volatility, Validity, and Veracity [3]. A popular way to access and use this vast amount of information is the analysis of online search queries [4, 5], with the most notable research on the subject taking into account data from Google Trends [6]; a tool that measures variations in the online interest. The validity of Google Trends’ data has been undoubtedly shown [7], while many studies have suggested that they are valuable, accurate, and of high benefit for forecastings [8‐13], predictions [14] and nowcastings [15], in analyzing online interest [8], and in decision making [3].

Google Trends as a tool is highly integrated in research, with notable contributions in the fields of health and medicine [16‐23], economics and finance [4, 24‐27], and the environment [28]. The use of Google Trends in political science in general and in predicting election and referendum outcomes in specific is showing promising results so far. According to recent research [14], what can be observed is an increased use of online data in polls and measuring voting intention. As Internet penetration is significantly growing in western countries and the significance of the Internet cannot be doubted [29], the monitoring and analysis of online search traffic data can be proven valuable in nowcasting election and referendum races. Burnap et al. [30] suggest that changes in online behavior can be accurately depicted on Internet data, while political campaigners increasingly use social media and online platforms [31]. Up to this point, the field of predicting referendum results using online search traffic data has not been much explored, though Google Trends’ data have been employed to predict the results of the 2015 Greek Referendum [14].

Finding new ways of predicting the outcome of elections and referendum is crucial, as the poll agencies have not always been successful in the recent past to well approximate voting intentions and results. During the short pre voting period of the critical 2015 Greek Referendum, we monitored the Google queries on the YES and NO keywords, so as to predict the voting intentions and the referendum outcome. Our results better approximated the official referendum outcome than poll agencies, the latter suggesting that the YES and NO difference was very small, or that YES was to win the race [14]. This method has also been applied to five other referendum races concerning crucial EU or constitutional matters and received international attention over the past years, namely the 2014 Scottish Referendum, the 2016 UK Referendum, the 2016 Hungarian Referendum, the 2016 Italian Referendum, and the 2017 Turkish Referendum.

This paper aims at presenting the methodology of how to predict and nowcast referendums with the use of online search traffic data; a methodology tested and validated in six occasions. This methodology aims at becoming a point of reference for the next generation of polling, while showing that Google Trends is a valid tool in nowcastings and predictions, with great potential. The rest of the paper is structured as follows: The “Research methodology” section consists of the method of predicting referendum results, while the “Results” section consists of the presentation of the results of six referendums over the past years (2014–2017) in the European Continent. The “Discussion” section consists of the discussion of the results, while the “Conclusions” section consists of our conclusions and future research suggestions.

Google Trends allows the monitoring of the change in the online interest in a term in a country or region over a selected time frame, e.g. a range of years, 1 year, 90 days, 30 days, 7 days, 4 h, 1 h, or a specified time period. It also allows the multiple relative comparisons of a term in different regions, or the comparison of various terms in one region, while a feature offers the opportunity to compare different terms in different regions. Data, depending on the time frame selected, are either monthly, weekly, hourly, or even of 1 min intervals.

Data from Google Trends are downloaded online in ‘.csv’ format and are normalized over the selected period, so as to allow the easy comparison between the respective examined terms. The adjustment of the data, as reported by Google, is as follows: “Search results are proportionate to the time and location of a query: Each data point is divided by the total searches of the geography and time range it represents, to compare relative popularity. Otherwise places with the most search volume would always be ranked highest. The resulting numbers are then scaled on a range of 0 to 100 based on a topic’s proportion to all searches on all topics. Different regions that show the same number of searches for a term will not always have the same total search volumes.” [32].

In general, data from Google can be accurate for measuring the public’s interest if the terms are carefully selected [33]. Preis et al. [5] for example, in developing the Future Orientation Index, only used Arabic numerals as the selected keywords, as they are universal, so spelling or translation errors would not produce bias. In a referendum, it is imperative that the terms selected are the ones to give an accurate result. Thus the wording of the referendum plays a significant role in predicting the outcome. In most cases, the question of a referendum is answered with a simple YES or NO, though, for example, in the 2016 UK Referendum, the wording did not allow for this, deeming this race more complicated to predict. In this case, the available answers were ‘Remain a member of the European Union’ and ‘Leave the European Union’, so we selected the terms “Remain” and “Leave” in order to predict the result. For the rest of the examined cases, we used the translation of the YES and NO keywords in each respective language, that is ‘YES’ and ‘NO’ in Scotland, ‘NAI’ and ‘OXI’ in Greece, ‘IGEN’ and ‘NEM’ in Hungary, ‘SI’ and ‘NO’ in Italy, and ‘EVET’ and ‘HAYIR’ in Turkey. Note that the keyword choice is not case sensitive in Google Trends.

For predicting the referendums, we mainly use weekly and monthly data (daily intervals), apart from the case of Greece, where daily and hourly data were used due to the very short pre-voting period of 9 days. Apart from the 2014 Scottish Referendum and the 2015 Greek Referendum, data were downloaded from the field ‘Campaigns and elections’ in Google Trends that eliminates noisy data, i.e. hits that are not attributed to the examined event. If data in this category are not sufficient due to low interest on the subject (as, for example, in the Hungarian Referendum), the search can be widened to include hits in the field ‘Politics’.

The time-frames for the data used in this study are as follows:

For the i-th set of data, i.e. monthly, weekly, 4-h, hourly, the hits’ averages for the YES (and Remain) \(\left( {Y_{{t_{i} }} } \right)\) and NO (and Leave) \((N_{{t_{i} }} )\) keywords are calculated, and are then percentized as \(Y_{{t_{pi} }} = \frac{{Y_{{t_{i} }} }}{{Y_{{t_{i} }} + N_{{t_{i} }} }}\) and \(N_{{t_{pi} }} = \frac{{N_{{t_{i} }} }}{{Y_{{t_{i} }} + N_{{t_{i} }} }}\) for YES and NO, respectively, with \(Y_{{t_{pi} }} + N_{{t_{pi} }} = 1\).

This section consists of the results of six referendums in the European Continent over the last years (2014–2017), namely the 2014 Scottish Independence Referendum, the 2015 Greek Bailout Referendum, the 2016 UK European Union Membership Referendum, the 2016 Hungarian Migrant Quota Referendum, the 2016 Italian Constitutional Referendum, and the 2017 Turkish Constitutional Referendum.

This paper consisted of the presentation and validation of a novel method of predicting referendum results by monitoring online search queries from Google, tested and verified in six different occasions from 2014 to 2017, i.e. the 2014 Scottish Independence Referendum, the 2015 Greek Bailout Referendum, the 2016 UK European Union Membership Referendum, the 2016 Hungarian Migrant Quota Referendum, the 2016 Italian Constitutional Referendum, and the 2017 Turkish Constitutional Referendum. The results in this study closely approximated the respective official referendum results, and in some cases better than traditional polls. Based on the results of this paper and the analysis of online search queries in general, what is observed is that we have entered the era where Internet has brought significant changes in terms of monitoring, analyzing, and predicting human behavior.

In the EU, many referendums about EU matters have been conducted in the past, though during the last two decades the NO option seems to be becoming more popular [85]. This is not always attributed as an actual answer to the referendum question, but as a means of expressing dissatisfaction to the respective government [86]. Lately, poll agencies more than often fail to well approximate referendum and election outcomes. Therefore, new methods of predicting voting intentions and results have been examined [87‐89], with online surveys taking the upturn.

The collection of referendums in this paper is notable for two reasons. At first, said referendums were significant in terms of policy and EU matters. Secondly, they present cases that will assist in future research using online queries as a polling tool, dealing with various special circumstances that arose. To elaborate, the very short pre-voting period in Greece was a good example of how to nowcast a referendum. In this case, the data had to be downloaded in very short time frames, i.e. every hour, every 4-h, and every day so as to extract robust results. In Hungary, the low turnout and interest in the referendum was depicted in the searches in Google during the pre-voting period. Furthermore, what was notable in the 2016 UK Referendum, was the effect of the murder of Labour MP Jo Cox, where the opinion shifted towards the Remain camp in the next couple of days after the murder. The UK Referendum was also significant in terms of the wording that did not allow for a simple YES/NO answer, thus the selection of keywords for monitoring the interest was not trivial. The most interesting case, though, was that of the Turkish Referendum, in a country where Internet restrictions are a serious issue. In Turkey, the use of Google Trends’ data were not useful in predicting the result. Despite that, they provided good approximations for the three EU countries where significantly large Turkish population voted, i.e. in France, Germany, and the Netherlands. We thus conclude that this method cannot be applied in regions with low levels of freedom of media, Internet, and speech, or with low Internet penetration.

Table 1 consists of the averages of the weekly and monthly percentized hits for YES and NO (Remain-Leave for the UK) for the examined referendum races, and their respective statistical significance for mean comparisons. Such comparisons can provide beforehand the result regarding the side of the outcome, i.e. whether the public shifts towards YES or NO, while the proximity with the final percentage is increasing as the time span approximates the ballot closing time, as discussed above.

As is evident, the results for the Scottish and the UK referendums exhibit high statistical significance on the final outcome. Regarding the Turkish Referendum, all examined comparisons are on the same side as the official results and are statistically significant, with the exception of the weekly results for France. Regarding the Italian referendum results, two of the compared datasets are statistically significant, while the rest do not statistically prove the ‘NO’ response.

As is suggested in Fig. 10—depicting the comparisons of the official results in each examined country with the predictions using Google Trends’ data for the last week of the pre-voting period—the analysis of online search queries is a valid method of approximating the result of a referendum and the voting intentions. Note that in the Greek and the Hungarian Referendum, our results using Google Trends’ data were the closest approximation to the final result, better than the poll agencies’ predictions using traditional methods. Also note that for the UK Referendum, the result refers to the percentized hits of the last day of the weekly time series.

As we validated that the results of using Google Trends for voting intentions and result predictions are accurate and, in many cases, better than official polls, we argue that this method will be adopted in the near future by poll agencies and political researchers in this field, as it tackles the economic risk, the cost, and the uncertainty barriers of poll taking.

However, some limitations do exist. At first, this method could not be applied in regions where the use of the Internet is restricted, in the ones having low scorings in freedom of press, or in those with low internet penetration, where data manipulation is possible, i.e. intentional googling of a certain outcome. Furthermore, the sample can not be proven representative. Not all internet users vote and not all voters use the Internet to search for the respective referendum keywords, thus not each hit can be linked to referendum voting, and in no way is it to be implied that such a 1–1 correspondence exists. In addition, a limitation of the tool is that it has been observed that data retrieved on different time points for the same time-frame may slightly vary. Despite the above, many studies in various subjects have shown that indeed Google data can be used to analyze or predict behavioral variations [4, 5, 8, 13, 16, 23, 26, 90‐92] and that empirical relationships exist between online search traffic data and human behavior [93, 94].

In this paper, we presented a novel methodology for predicting voting intentions and referendum results using online search traffic data from Google, which was tested and validated in six referendum races. Said referendums, i.e. the 2014 Scottish Referendum, the 2015 Greek Referendum, the 2016 UK Referendum, the 2016 Hungarian Referendum, the 2016 Italian Referendum, and the 2017 Turkish Referendum, were of significance for the European Union and received wide national and international attention. Employing data from Google Trends, we estimated the respective referendum results using data on the (translated) YES and NO keywords. Our results exhibited good performance, while, in some cases, were more accurate than official polls.

Since online behavioral changes can be measured by online data [30], the potential benefit of using Google Trends’ data as a poll taking method is high, especially as traditional polls did not always manage to accurately predict the outcome, and, in many cases, were not even on the winning side of the results. Google Trends has been shown to be a credible means of examining behavioral changes, as it uses the revealed and not the stated preferences of the users. Thus in regions were Internet is widely accessible and not restricted, online data are useful in analyzing and predicting human behavior in many research topics.

This method of poll taking, based on empirical relationships, could be of interest to political researchers, as it is a valid analyzer of human behavior, indicating that Internet data can give insight to behavioral variations towards political matters and election races. Future research on the subject could focus on developing more sophisticated models using online data, as well as the combination of various online sources, or the combination of online with traditional survey data. Overall, the above suggest that monitoring Internet data in general and Google Trends data in specific will be a polling tool to address the challenge of Big Data in the future.