National happiness index monitoring using Twitter for bilanguages

Social media are now recognised as an important source of near-instantaneous information on events, news, ideas and more importantly opinions and emotions. It represents an effective and immediate method of communication among individuals and communities. The informal nature of social media has enabled it to become a rich medium for direct exchange of opinions and expressing sentiments, hence an effective medium for carrying out sentiment analysis to monitor happiness indices close to real time.

To understand the general sentiment of a certain populace is an important goal of organisations and governments globally. Social media offer first-hand insight into the thoughts, feelings and concerns of the population. Therefore, monitoring social media enables organisations and government to measure the degree of population happiness without involving significant costs.

Some research has been carried out on sentiment analysis from social media, but most of which focuses on sentiment analysis for a particular service or product/brand and mainly for languages with simple grammar, such as English. However, general sentiment analysis from short messages (tweets) for multiple languages has been challenging due to the following reasons:

However, due to the wide usage of social media in Arabic countries, there is great demand for an accurate system to monitor the happiness index of general sentiment from social media. Such a system needs to be at least bilingual. Given the challenges and high demand above, this paper proposes a general system for population happiness index monitoring using sentiment analysis from a social media stream (Twitter). The contributions of this work can be highlighted as:

This paper is organised as follows: we discuss relevant work to sentiment analysis from social media in Sect. 2 and present our proposed system in Sect. 3. To prove the validity, we applied our proposed method to benchmark datasets and the results and comparisons are discussed in Sect. 4. Section 5 presents the results of our proposed method when used for general populace sentiment analysis in Abu Dhabi, UAE. At last, Sect. 6 discusses our conclusion and future work.

Diverse research has been carried out to investigate how to improve the accuracy of sentiment analysis from social media. Some research focussed on the importance of pre-processing for sentiment analysis from social media (Krouska et al. 2016; Singh and Kumari 2016; Sahu et al. 2015; Jianqiang and Xiaolin 2017). For example, Singh and Kumari (2016) investigated the importance of pre-processing and claim that their proposed method is robust to size of datasets but also improved the accuracy. Some research was done using ensemble methods (Troussas et al. 2016; Hassan et al. 2013; Kanakaraj and Guddeti 2015; Abdelwahab et al. 2015; Zhao 2016), removing the imbalance of dataset and solving the problem of not enough training data, with the aim of improving the overall accuracies. Hassan and Abbasi (2013) proposed a bootstrap ensemble framework for Twitter sentiment analysis to solve the data imbalance problem and achieved better accuracy (between 27 and 80% across different datasets). Kanakaraj and Guddeti (2015) analysed how different ensemble methods help with the accuracy improvement for sentiment analysis from twitter. Zhao (2016) used ensemble learning (representing words as vectors) and combined sematic and prior polarity for boosting Twitter sentiment analysis. Krouska et al. (2017) presented a system for sentiment analysis for figurative language (emotion) based on both content (text) approach and an emotion pattern approach, which showed that combining context analysis and emotion pattern analysis improves the overall accuracy for sentiment analysis from social media. Krouska et al. (2016), Troussas et al. (2016), and Nguyen and Jung (2017) evaluated how the pre-processing (Krouska et al. 2016) and data ensemble (Troussas et al. 2016) help improve the sentiment analysis accuracy for different machine learning methods, and showed ensemble improves the accuracy significantly. They also compared the performance of using different algorithms for sentiment analysis on benchmark datasets (Nguyen and Jung 2017). Abdelwahab and Bahgat (Abdelwahab et al. 2015) investigated the relationship between the size of training set and accuracy. Sahu and Rout (Sahu et al. 2015) used a lexicon-based method for sentiment analysis and claimed that sentiment analysis in different domains need different lexicons, which indicates the inflexibility of lexicon-based methods. Kontopoulos et al. (2013) proposed an ontology-based sentiment analysis for tweets. Furini and Montangero (2016) proposed and investigated a gamification approach for sentiment analysis from tweets. Bravo-Marquez et al. (2016) investigated the transferring between words in sentiment lexicon and tweets’ sentiments. From the state of the art, machine learning techniques are also applied to sentiment analysis including naïve Bayes, maximum entropy (MAXENT) and support vector machine (SVM), as well as artificial neural network (ANN). Nakov et al. (2016a, b) discussed different algorithms and performances for sentiment analysis from short messages based on benchmark datasets, SemEval. Their investigation shows that most systems were supervised and used a variety of handcrafted features and Twitter-specific encodings. Chiassi et al. (2013) and Zimbra et al. (2016) proposed hybrid system using n-gram and dynamic artificial neural network for brand sentiment analysis based on manually defined lexicon set and claimed their accuracy to be over 95%. Ajay and Sudhir (AIIT 2016) compared the performance of six popular machine learning methods for sentiment analysis from tweets. Shyamasundar and Jhansi (2016) focused on feature extraction using TF-IDF to improve the sentiment analysis accuracy from machine learning techniques (naïve Bayes and SVM) for benchmark datasets. Siddiqua et al. (2016) proposed a rule-based classifier based on the emoji and sentiment lexicons for Twitter sentiment analysis and compared with two known systems (SentiStrength and Semantria) and claimed better accuracy in three out of four benchmark datasets.

Some of the proposed methods are applied to various benchmarks or their own collected datasets and achieved accuracy between 60 and 85% depending on the algorithms and datasets. Ren et al. (2016) improved the word embedding method for Twitter sentiment analysis and achieved an accuracy of 78.57% without expert knowledge and 81.02% with expert knowledge for benchmark datasets. Wu et al (2016) proposed a method to extract sentiment knowledge from massive unlabelled data (tweets) and achieved an accuracy of 85% to 88% when applying it to different datasets. Saifa et al. (2016) introduced a lexicon-based approach which achieved better results for two datasets but worse result for the third. Pandey et al. (2017) proposed a novel k-mean-based cuckoo search for sentiment analysis from tweets which showed an accuracy of 67% to 84% for different benchmark datasets. Zhao and Cao (2015) proposed a method to make use of the co-occurrence statistics and contextual semantic relations as features for sentiment analysis and produced an average accuracy of 82% for different benchmark datasets. Christos and Maria tested different algorithms for sentiment analysis services. In this context, five well-known learning-based classifiers (naïve Bayes, support vector machine, k-nearest neighbour, logistic regression and C4.5) and a lexicon-based approach (SentiStrength) have been evaluated based on confusion matrices, using three different datasets (OMD, HCR and STS-Gold) and two test models (percentage split and cross-validation). The results demonstrated the superiority of naïve Bayes and support vector machine regardless of datasets and test methods.

Some methods have been applied to particular real-life problems, e.g. sentiment for a particular brand for customer services, events or political elections. Much research has been done for sentiment analysis from tweets for brands for customer services (Chiassi et al. 2013; Zimbra et al. 2016; Qaisi and Aljarah 2016). Philander and Zhong (2016) used a dictionary-based method for social media microblogging data sentiment analysis for hospitality operators and used Las Vegas integrated resort casino as use case, which they claimed to yield reasonable reliability for reviews ranking. Qaisi and Aljarah (2016) used tweet sentiment analysis for two different cloud service providers. Gupta and Kohli (2016) used Twitter sentiment analysis in healthcare. Yang and Wang (2014) proposed a method to analyse real-time tweets for US soccer fans during 2014 FIFA World Cup games and analysed the dynamic emotion changes of fans through the courses of the games. Schumaker et al. (2016) used sentiment analysis from social media to predict football matches outcomes and yielded an accuracy of 67%. Peng et al. (2016) used twitter sentiment analysis for drugs incidents using Weka packages and the sentiment analysis accuracy was just over 62–65%. Ramteke et al. (2016) used Twitter sentiment analysis for election result prediction. Other applications included how to use social media to assist in solving other problems, e.g. economic analysis and financial prediction by using social media as supplementary information. Smailovic et al. (2014) proposed a system to use social media to help financial markets prediction for more profitable online trading. Porshnev et al. (2013) used different dictionaries for sentiment analysis from tweets to help financial and stock prediction. Daniel et al. (2017) focused on event-based sentiment analysis from tweets and discovered that the tweet events’ sentiment influences financial events for the community/company.

Although much work has been carried out, social media sentiment analysis is still challenging due to the limited information in one single message of social media, in addition to indirect expressions, as well as the noisy information within social media such as advertisements and news. In all the of above prior art, the importance of noise filtering and tweet pre-processing is mentioned. Pre-processing includes a wide range of approaches to reduce noise.

All the research in the state of the art is focused on a subset of tweets out of the overall tweet stream, e.g. sentiment analysis via keywords search from tweets for relevant brands/products. When we consider the whole tweet stream to investigate the general (overall) population sentiment, we need to consider all tweet-carrying sentiment information. We need to filter out tweets with no sentiment bearing content, e.g. advertisement, news and spam. As mentioned in the state of the art, filtering is very important, yet challenging. Filtering noise from a sub-stream of tweets related to a particular subject matter, event or brand has been difficult; filtering noise from the whole general tweet stream is even harder. Hence, the first challenge is how to filter out the information with no sentiment element from the general tweet stream. After filtering all non-sentiment tweets, sentiment analysis for general tweets remains more challenging than sentiment analysis for a specific application, this is due to the vast diversity of vocabularies and expressions used across different areas. It gets more complex when the tweet does not belong to any single defined subject matter, e.g. “Good day today,” and has sentiment information but not for any particular area. However, this information is indicative of the individual being happy and contributes to the overall happiness index for the whole population. On the other hand, given sentiment analysis from overall general tweets, obtaining sentiments for specific topics, brands, events, etc. is merely a tweet sub-selection task using either keywords or machine learning techniques to extract tweets from the overall tweet stream.

In summary, a sizeable amount of work has been done on sentiment analysis from social media, which we cannot list fully. However, as stated in the recent survey (Yue et al. 2018), most work focuses on consumer and population opinion mining for companies conducting surveys about corresponding products and services, national security or public views on events such as elections. Very little work is done on general populace sentiment analysis due to the challenges mentioned in Sect. 1. This paper proposes a general system for population happiness index monitoring using sentiment analysis from social media data (Twitter) which is introduced in Sect. 3.

There are five main components: tweet smart harvester, tweet filter, sentiment classifier, tweet sub-stream selector and visualiser (as shown in Fig. 1).

Tweet sentiment analysis is a difficult problem of text classification. Text classification accuracy is the key for a happiness index monitoring system and is used for both the filtering process (classify all tweets into sentiment-relevant and sentiment-irrelevant tweets) and sentiment analysis (classify sentiment-relevant tweets into positive, negative, neutral and junk).

From the state of the art, various machine learning techniques for text classification are applied to sentiment analysis including naïve Bayes, maximum entropy (MAXENT), support vector machine (SVM), artificial neural network (ANN) and latent Dirichlet allocation (LDA). LDA (Blei et al. 2003) was introduced by Blei and is a statistic model for classification based on word–topic frequency distribution. When LDA is applied directly to tweets, we lose the accuracy due to the shortness of tweets. We proposed a method, the improved LDA, to tackle this difficulty which is the subject of a pending patent application.⁵ Compared with other machine learning methods, no keywords are needed to be pre-defined for LDA itself which makes it more attractive for text mining. We proved the higher accuracy of our proposed improved LDA over SVM and other algorithms in our previous work (Wang et al. 2017; Wang and Al-Rubaie 2015).

We are using our improved LDA for tweet sentiment analysis, i.e. classification of positive, negative, neutral and junk, and as filter, i.e. classification of sentiment-relevant and sentiment-irrelevant tweets. Our proposed improved LDA is also compared to state-of-the-art algorithms for benchmark tweet datasets.

To prove the advantages of using our improved LDA for sentiment analysis in comparison with the state-of-the-art algorithms, we applied our improved LDA to a set of benchmark datasets used in the literature as shown in Table 1.

We first apply our proposed method to benchmark database OMD (Wu et al. 2016), STS-Gold (Wu et al. 2016) and HRC database (Ren et al. 2016; Wu et al. 2016). The comparison of our proposed method with existing methods and baseline methods are shown in Table 2 and Figs. 8, 9 and 10. When comparing with the results in (Wu et al. 2016), our proposed improved LDA outperforms all their algorithms for all datasets, mainly because their methods are lexicon-based, while our proposed improved LDA is machine learning based. When compared with the best result of other machine learning techniques with ensemble learning (Troussas et al. 2016), our proposed improved LDA is comparative to their accuracy (0.8579 using ours and 0.8774 using ensemble-based methods for OMD dataset, 0.8629 using ours and 0.8510 using ensemble-based methods for HCR dataset and 0.9130 using ours and 0.8902 using ensemble-based methods for STS-Gold dataset). Their ensemble learning is based on the combined/best results of popular machine learning methods (NB, SVM, KNN and C4.5) which proves that ensemble learning increases the accuracy of machine learning methods in general. Therefore, we expect the accuracy for our proposed improved LDA to increase when embedded in ensemble learning techniques, which is part of our future work. Our proposed improved LDA is also better than Zhao’s method (Pandey et al. 2017) for STS-Gold dataset. (Result for other datasets is not available in their paper for comparison here.)

Then we apply our proposed improved LDA to STS and Sanders datasets (Ren et al. 2016) for accuracy comparisons with different systems. We apply the same strategy of fivefold cross-validation to compare our accuracy with the result in Ren et al. (2016) and Pandey et al. (2017), as shown in Table 3 and Fig. 11. We can see our proposed improved LDA obtains the best accuracy, which is overall 2% better in accuracy for both datasets of STS and Sanders. When compared with Zhao’s method (Pandey et al. 2017), our proposed method obtains better accuracy for STS-manual dataset, and the accuracy for Sanders dataset is not available from Zhao’s paper (Pandey et al. 2017).

Other tweet datasets we use for accuracy comparisons are: test data manual 2009.06.14 (Saifa et al. 2016) (also referred as STS-manual test (Ren et al. 2016)), Twitter-sander-apple2 (Saifa et al. 2016), Twitter-sander-apple3 (Saifa et al. 2016) and Twitter dataset online (Saifa et al. 2016). The accuracy results are shown in Table 4 and Fig. 10. From Table 4, we can see that our proposed improved LDA achieves the best accuracy for the datasets of test data manual 2009.06.14 (Ren et al. 2016; Saifa et al. 2016), Twitter-sander-apple2 (Saifa et al. 2016) and Twitter dataset. When the proposed improved LDA is applied to Twitter-sander-apple3 (Saifa et al. 2016), we achieve an accuracy of 0.7836 which is the second best among all compared methods. Twitter-sander-apple3 has 3 categories instead of 2 as in the other datasets. This is expected, as when using training-based classifiers the accuracy decreases when the number of categories increases.

Zimbra et al. (2016) tested the performance of various algorithms for sentiment analysis by using the Sanders Twitter dataset. The accuracy comparison in precision, recall and F-score is shown in Table 5. Our proposed improved LDA and random forest tend to achieve better results than other methods. Random forest obtains the best precision but very low recall for positive and negative categories, which makes random forest less useful here because it fails to identify most actual positive and negative tweets. The F-score for positive and negative categories from random forest is very low due to their low recall rate. Our proposed improved LDA achieves the best F-score for positive and other categories, and the second best accuracy for the negative category.

In this paper, we proposed a system for general populace sentiment monitoring from social media. We highlighted and discussed the challenges of sentiment analysis from general tweets rather than specific streams such as brands, events, related tweets, as well as the challenges for sentiment analysis in Arab countries.

For general sentiment analysis, we need to accurately identify tweets with sentiment among all tweets. To do this we, proposed and used multi-level comprehensive filters that are able to filter out non-sentiment tweets accurately. An accurate filter is a prerequisite for accurate sentiment analysis.

A short message classifier is key for accurate filtering and accurate sentiment analysis. We used our proposed improved LDA for tweets (Wang et al. 2017; Wang and Al-Rubaie 2015; Blei et al. 2003)⁷ sentiment analysis which is able to achieve the best accuracy for benchmark tweet datasets when compared with existing methods as shown in Sect. 4. For general sentiment analysis from all tweets, the proposed improved LDA is able to achieve similar or better accuracy for sentiment analysis than human tagging as shown in Sect. 5. A comprehensive tweet visualiser was also developed to show the users’ sentiment changes, either in general or for particular interest(s), in geo-location distributions and through time.

Although our improved LDA shows better accuracy in benchmark datasets, the accuracy for general sentiment analysis using real-world content is 68% and still needs to be improved. Deep learning and BERT (Devlin et al. 2019) are becoming popular for NLP tasks including text classification. However, BERT started for English and has been extended to other languages such as Arabic, i.e. AraBERT through Hugging Face (Wolf et al. 2020) very recently. Our future work will be investigating AraBERT for general sentiment analysis and how to combine it with our proposed method for better performance in accuracy.