Inferring the votes in a new political landscape: the case of the 2019 Spanish Presidential elections

With the Twitter Streaming API,² all the posts containing one or more hashtags related to the presidential election are collected every day as follows:

The tweets are collected in real time, 24 h after their publication. The aim is to get statistics related to the popularity of the tweet such as (a) the number of shares or retweets, which shows how many users shared the tweet with their followers; (b) the number of likes, which shows how many users linked the tweet (c) the number of retweets which shows how many tweets were re-broadcasted on the media. Between April 12th and April 28th 1.170.000 tweets were posted and Table 1 shows the amount of data collected in terms of users and tweets per candidate.

When working with the Twitter API it was necessary to take into account certain limitations of it. The frequency limit of the interface only allows 450 requests every quarter of hour.

The dataset resulting from data collection is a collection of tweets. In order to apply SA techniques, data must go through a four-step process, known as pipeline, in which the input of each step is the output of the previous one.

The first step is called the text cleaning. A tweet is a complex object with many properties. But what it is important is the message that the user writes in the form of a text. So, the URLs, emails are deleted and necessary functional words of the Twitter lexicon such as \RT, \HO, \HT, unknown characters are replaced to their closest ASCII variant, using the R Unicode Library. Each document is then transformed into a list of words between blanks or punctuation which is called tokens. During this step, the information about the author of the tweet and the date of publication are lost. But the reference to the object to be able to link the result with the original tweet is kept.

The second step is called the lexical normalization. It removes the functional words which do not have a clear referential semantic, such as articles, pronouns, preposition such as (“the”, “an”, “and”, “of”, etc.), numbers or dates. They are usually called as stop words. Uppercase letters are transformed to lowercase. This allows the algorithm to consider identical words such as “Big” and “big” and consequently as two counts of the same word. It deletes also any repeated sequences of letters in the words (such as “greaaaat”, “mooorning”, etc.) reverting them back to their original English word.

The next step is the feature extraction applying the term frequency (TF) and Inverse Document Frequency (IDF) R package. The Vector Space Model (VSM) is one of the most commonly used methods for text retrieval and to represent a document as a vector of terms [31, 36]. It consists of transforming each token (i.e. tweet text) into a feature vector in an Euclidean space.: t = {t1, t2,… tn} ∈ R^s where s is the size of the vocabulary which is built by considering unigrams in the collection (i.e. the training test) [33]. The sum of tokens builds a sparse matrix or also called Corpus (C).

A SA system is highly sensitive to the domain in which the data used to train are extracted. It can obtain poor results if the training dataset is not political [4, 56]. Due to the lack of a sentiment lexicon for non-English languages, the creation of a new polarity lexicon is decided for the Spanish political event issuing from two different sources. The starting point is the dataset created by [40] for Spanish tweets. Then, a random sample of 1.000 tweets is chosen as the training set. In order to label a tweet as either positive, negative or neutral, eight volunteers in group of 2 (i.e. 4 groups) are asked to assign a label for each tweet according to the sentiment they feel it conveys. The evaluators have access to the full tweet, which is tagged according to its global polarity, indicating whether the text expresses a positive, negative or neutral sentiment. To prevent ambiguity, testers have to assess the tone of some tweets after screening the other tweets of the account. Moreover, if there is no agreement inside the group, a third independent volunteer from another group is asked to help to decide. Volunteers agreed on 80% of tweets which support the statement [52] and determined which additional word had to be included in the lexicon. In total 187 words related to this political campaign are added based on this manual testing and inserted in the final vocabulary. These additional words are for instance: “movilización, impulso, progreso, traición, sinverguenza, matando, etc.”—English translation: “Mobilization, impulse, progress, treason, shamelessness, killing, etc.).

The method of estimation of a linear regression consists of determining the coefficients which minimize the sum-of-squared deviations of the observed values of Y from the predicted values based on the model. Two principles distinguish a Logistic Regression (LR) from a linear regression. The first is that the outcome variable in LR is binary or dichotomous. In any regression problem, the key quantity is called the conditional mean and is expressed as ‘E (Y |x)’ where Y denotes the outcome variable and x denotes a specific value of the independent variable. In the linear regression model, an observation of the outcome variable may be expressed as y = E(Y|x) + ε. The quantity ε is called the error and expresses a deviation from the conditional mean. The most common assumption for a linear regression is that ε follows a normal distribution with mean zero and some variance is constant across levels of the independent variable. In turn, in a logistic regression, the binomial distribution is the statistical distribution of the errors on which the analysis is based [25]. This is the second difference.

The SA system classifies a given vector as either positive, negative or neutral by adding all the number of positive words and subtracting the number of negative words. It assigns only one label class to the tweet. This type of analysis makes sense if each tweet expresses a single opinion. It may seem like a limitation, but in practice it works well since users usually focus on a single topic in each tweet. Surely in other contexts, or if this limitation of message length were not available, it would be necessary to consider more complex systems that allow more granular analysis [12].

Linear discriminant analysis (LDA) also known as Fisher Discriminant Analysis is very similar to Logistic regression [55]. It is a method used to find a linear combination of features that separates different classes of objects. LDA searches for those vectors in the underlying space that best discriminate among classes (rather than those that best describe the data). The LDA is optimized when the independent variables are normally distributed. It is indeed an important constraint of the model. Mathematically wise, for all the samples of all classes, two measures are defined (1) and (2):

Then, LDA can be formulated as an optimization problem to find a set of linear combinations (with coefficients \(w\)) that maximizes the ratio of the between-class scattering to the within-class scattering. One way to do this is given by the following generalized eigenvalue problem (3)

Generally, at most c − 1 generalized eigenvectors are useful to discriminate between c classes.

Classification and Regression Trees (CART) is an umbrella term used to refer to decision or regression trees [13]. While the target variable can take a discrete set of values, decision trees are called classification trees. It is a flow-chart structure where each fork represents conjunctions of features that lead to class labels (also called the leaves). Since the branch can be seen as a split in a predictor variable, each end node contains a prediction for the outcome variable. When the target variables can take continuous values (i.e. real numbers such as voting prediction), decision trees are named regression trees. The decision tree continues to expand with additional nodes being repeatedly inserted until the stopping criteria is met. In other words, it stops when the predicted number of iterations is reached, or a reasonable prediction is achieved.

k-Nearest Neighbour (kNN) classification determines the local decision boundaries. The technique assigns any object to the majority class of its k closest neighbours. The main constraint of this method is that the results may change if we change k. KNN is also a ‘lazy learner’ since all computations are deferred until function evaluation. In this study, the kNN regression is used to estimate continuous variables. This algorithm uses an average of the k nearest neighbours, weighted by the inverse of their distance.

Vapnik [50] firstly introduced Support Vector Machines (SVM) to solve two-class recognition problems. The general idea is to find the decision surface that maximizes the margin between data points, i.e. between classes. A good separation is achieved by the hyperplane that is furthest from the nearest training-data point of any class (so-called functional margin). In general, the larger the margin is, the lower the generalization error of the classifier is also. In the case of originally linearly non-separable data points, the original data vectors are mapped to higher dimensional space to achieve linear separability again.

The Random Forest (RF) is an algorithm that randomly creates and merges multiple decision trees into one “forest.” The goal is not to rely on a single learning model, but rather a collection of decision models to improve accuracy and reduce the variance. The primary difference between this approach and the standard decision tree algorithms is that the root nodes splitting is generated randomly. RF corrects the decision trees recurrent error of overfitting of the training set i.e. it has low bias, but a very high variance. RF generates many predictors, each using regression or classification trees. It reduces instability by averaging multiple decision trees, trained on different parts of the same set of data. Then, it develops a final prediction which usually improves the prediction performance of a standard decision tree.

According to Jungherr [28], model performance tends to fluctuate depending on the methods employed even if the impact of the parameters is hard to know. Heredia et al. [24] add that models based on specific counts of tweets that only mention a single party outperform others that combine SAs with volumetric information. Therefore, a correct setting of the hyper-parameters helps to limit negative effects like over- or under-fitting. Respectively in an SVM, kNN or RF method, a higher C, k, or max-depth may cause the model to misclassify less but is much more likely to generate overfit. The Appendix 1 presents the selected settings of hyperparameters. The experiments for this study were carried out using the fit.models library in R software. The evaluation criteria for the ML techniques are presented in the next sub-section.

This paper incorporates certain evaluation metrics to validate the efficiency of the proposed model. The metrics MAE and RMSE are used to analyse and compare the performance of ML methods. The Mean Absolute Error (MAE) measures the average magnitude of the errors in a sampling of predictions with absolute differences between prediction and actual observation where all individual differences have equal weight. It is formulated in (4). In turn, the Root Mean Squared Error (RMSE) measures the average magnitude of the error, which is the square root of the average of squared differences between prediction and actual observation. The equation is (5).

Since the errors are squared before they are averaged, the main differences between RMSE and MAE is that RMSE gives a relatively high weight to large errors. This means it would be more useful when large errors are particularly undesirable [54].

Jaidka et al. [27] state such consideration is important when minor political parties end up becoming more active on social media platforms than leading parties. In this study, large and small parties are equally active (Table 1), so the five models are benchmarked using MAE results.

Precision, Recall, and F-score are used to measure classification performance. Formulas applied are, respectively (6) and (7) [32]:where TP = True positive; TN = True negative; FP = False positive; FN = False negative.

Precision is the division of retrieved instances that are significant. Recall is the fraction of applicable instances that are recovered. In dual classification, recall is known also as sensitivity i.e. the possibility that a relevant document is recovered by the query.

The F-score balances the use of precision and recall measuring the accuracy of experiments. It is the weighted harmonic mean of the precision and recall (8).

For the lexicon-based sentiment classifier, the data are split and the process of cross validation is repeated during k iterations. K is optimized when increased slowly and classifier accuracy is sought. The optimization process (k = 3) is presented in Fig. 2.

The system achieves a macro-averaged F1-score of 92.65% and an accuracy of 92.70% on the test set. These similar scores also confirm there is neither over-, nor under fitting of the model.

The five Machine learning inferring model

The dataset used to predict the voting share of each candidate is composed of the following independent variables:

The features are computed in a daily basis. Then they are normalized by applying the moving average smoothing technique over a window of the past 7 days. The polling data is considered as the dependent variable of our model.

The voting intention inference is approached as a multiple regression analysis. In this way, several regression models are built to infer the vote of each candidate in the electoral campaign, using the aggregated polling as the output variable of the models. In total, five models are built i.e. one model per candidate. The five different algorithms: LDA, CART, kNN, SVM and RF are then evaluated.

The performance results of the five algorithms are analysed for one candidate: Pedro Sanchez (Fig. 3). The complete results by model and candidate are shown in Table 2. The empirical results demonstrate that kNN with the lowest MAE are the best predictions and outperforms the rest of algorithms on the five models. LDA with the highest MAE has the worst performance.

Using the function predict in the R package, the 2019 Presidential voting shares are shown in Table 3 using the best model (kNN).

The first column of Table 3 shows the 2019 Spanish Presidential official results and in the third one the voting intention inference based on kNN best method. As far as the second column is concerned, it presents the last polling results taken 5 days before the election in compliance with Spanish law regulating the presidential election campaigns. The results are discussed in the next section.

Presidential campaign prediction using Twitter data is a much-discussed research topic and the number of studies are still growing [9, 10, 24, 27, 37, 51]. There is, however, no real benchmarking on the performance [22] to identify the best method to determine the winner of the elections or the voting share with the minimum error. Nevertheless, works from other research domains such as Medicine, Engineering or Finance, highlight calibration as a remarkably effective method to obtain better performance [16]. They conclude also LR performs better for smaller data sets and has worst results for larger ones [44]. The inferring results of this study corroborate them. Indeed, LDA with the poorest MAE has the lowest training time required compared to the other algorithms and the data set comprised of 17 days of campaign and 1.170.000 tweets can be considered as large.

When predicting the vote rates, Gayo-Avello [22] confirm MAEs are normally used to measuring accuracy. However, they inform that it is difficult to compare MAEs between campaigns. Having said that, a MAE baseline between 1 and 2% can be considered as a good prediction model. Consequently, the MAE of the kNN algorithm positions this paper in the upper quadrant of Twitter prediction performances [22].

The results of this paper are less accurate than the research institute to estimate the voting share at candidate level: Casado, Rivera and Iglesias. These latter positioned in the centre of the Political Exchequer (Table 3) make up the zone with a large number of undecided voters. Nevertheless, the sum of their votes is 49.6% according to the polls and 47.6% according to the proposed model. The official election results are 46.9%. So, the difference (as a whole) is better for the model compared to the research institute. It demonstrates that our model has been able to better estimate the high volatility during the campaign and the transition of votes between the three blocks: Sanchez, the three candidates (Casado, Rivera and Iglesias) and Abascal.

‘Social Big Data’ is an extremely important asset in the predictions of undecided voters. Silva et al. [48] observe a similar migration of votes among blocks of candidates during the Brazilian electoral campaign. To deal with this issue, they introduce a variable called ‘voting decision’ in their predicting model which indicates the degree of certainty of the voter in relation to his/her declared vote for president. Further lines of investigation can test their method to the present dataset to check if it improves the overall results.

In the following lines, we suggest additional ways to improve this present model.

The results reveal a gap between what the user writes in a tweet and what he or she really thinks and finally votes, in other words, the online user credibility. Abu-Salih et al. [4] define the construct “Trust” for social media such as the credibility of the content posted by users in a particular domain. Abu-Salih et al. [1] propose an ontology-based domain framework to analyse the semantic of the tweets considering not only the users’ messages but also their metadata e.g. the nature and the volume of Retweets and Likes. Their method aims at better understanding users’ interests and at highlighting inconsistent behaviour. In other words, it evaluates tweets credibility in a specific domain. Their approach is unique since most of the studies in the past had neglected the domain level of the message along with the factor of time [2‐4]. Further study should analyse if including this framework could improve the inferring results by discarding inconsistent tweets injected in the prediction algorithms. Nevertheless, the outcome could be limited since the tweets contain hashtags which were carefully selected to ensure they were strictly related to the Spanish Presidential election domain analysed (e.g. #28A, #28Abril, #Elecciones Generales).

Many presidential candidates hire companies to create robot accounts to follow them. These anomalous users or spammers post tweets supporting them. Those messages and Twitter accounts don’t represent the general public opinions and have to be discarded. Their objective is to create positive or negative rumours. In this manuscript, detection of spams or irrelevant posts is realized by implementing a ML supervised method, a RF algorithm using the following features: spammer users previously identified, age of the user account, retweet rate and number of favourites. But, more recently, Abu-Salih et al. [3] include a time-based semantic analysis at domain and user levels. Their technique provides a solution to rank tweeters’ credibility. Consequently, further work could analyse if this novel approach improves the results of the cleansing phase to detect untrustworthy users i.e. spammers.

In ML techniques, the splitting of the data is performed in a stratified way and the process of cross validation is repeated during k iterations. This method is more precise but could have the disadvantage of slowing down the ML computational times. Ahmadvand et al. [6] suggest the sampling of the computing techniques as an alternative. They add that this usually generates the desired quality of result when resources such as time, cost or energy are limited. Another option resides on progressive computation i.e. when large data space is pruned progressively to look for the results [5]. The processing times were affordable, but this method could be used if scholars decide to extend the time period of data collection or the number of hashtags and get a larger dataset.