Deep learning based topic and sentiment analysis: COVID19 information seeking on social media

We collected Twitter conversation in the Australian Sphere on COVID19 starting from 27 November 2019 when the first break out occurred in China. The data collection is done via the Queensland University of Technology (QUT) facility of Digital Observatory² using the Twitter Stream Application Programming Interface (API). The dataset consists of 2.9 million tweets from 27 November 2019 to 7 April 2020. Every tweet in the dataset contains or uses as a hashtag at least one of the following keywords: coronavirus, covid19, covid-19, covid_19, coronovirusoutbreak, covid2019, covid, and coronaoutbreak.

The body of each tweet, i.e. tweet message, is used for analysing sentiment, topics and impact. Location and time information of each tweet gives it spatio-temporal dimensions. The location information for each tweet comes from either of three sources based on their availability: (a) tweet location, i.e. the location user was in when the tweet was posted; (b) user location, i.e. residence of the user; or (c) location mentioned in the tweet message. The locations are mapped to capital cities, states, or the country Australia depending on how granular level location information are extracted. The time information of each tweet comes from the time and day the tweet was posted. Table 1 shows a few examples.

For preprocessing, we removed stopwords, punctuations, and invalid characters. We dropped any non-English tweets. We fixed repeating characters, converted text to its lowercase, replaced an occurrence of link or URL with a token namely xurl, and stemmed the text. We used Named-entity Recognition (NER) from spacy³ to extract locations.

For evaluating the proposed sentiment analysis model, we collected the sentiment140 (Senti140) (Go et al. 2009), COVID19Senti (Sentiment Analysis 2021) and GeneralSenti (Twitter sentiment xxxx) datasets. Sentiment140 contains 1.6 million tweets annotated for two classes of sentiments: positive and negative. COVID19Senti contains around 41,000 tweets annotated with five types of sentiments that we grouped into two: positive and negative. GeneralSenti has around 32,000 annotated tweets that include two groups of sentiments: positive and negative.

For evaluating the proposed topic analysis model, we used three datasets: COVID19Senti (as discussed above), EastAsianHate (Vidgen et al. 2020); Bashar et al. 2021) and RandomHate (Twitter hate xxxx; Bashar et al. 2021). EastAsianHate contains around 8,000 tweets annotated as whether a tweet is East Asian relevant or not and, if so, what is the stance. RandomHate has around 4,000 random tweets annotated as whether a tweet is hate speech or not. We use the same preprocessing as the Australian Sphere data to prepare these datasets.

If there is not sufficient training data, especially when application domain is impacted by data shift, machine learning models can overfit and provide poor accuracy Bashar et al. (2021). During a suddenly changed situation such as COVID19 pandemic, domain is impacted by data shift as the underlying data distribution on twitter before pandemic and during pandemic changed significantly Bashar et al. (2021). Word clouds generated from two such datasets shown in Figs. 3 and 4 highlight the variations in two distributions, e.g. vocabulary, term frequency and the use of language. This problem can be potentially addressed by integrating prior knowledge into the training process von Rueden et al. (2019); Bashar et al. (2021). This process of integration is known as Informed Machine Learning von Rueden et al. (2019). Improving machine learning models by integrating additional prior knowledge into the training process has gained a huge research interest recently. Combination of data- and knowledge-driven approaches is becoming popular in many research areas von Rueden et al. (2019).

In essence, an informed machine learning model uses a hybrid information source that consists of data and prior knowledge. The prior knowledge is pre-existent and independent of learning algorithms. In this research, we propose an INN model for sentiment analysis that uses the sentiment as prior knowledge in the form of lexicon and rules obtained from VADER C. J. Hutto (2014). VADER lexicon consists of 7,500 features and has sophisticated linguistic rules to produce sentiment scores. The prior knowledge is utilised to estimate three sentiment scores (positive, negative, and neutral) for a given tweet (i.e. a candidate for analysis). These three scores reveal probabilistic relations between a text and sentiment and are used in regularising the INN model in the training process.

Figure 2 shows the architecture of the proposed INN Model for sentiment analysis. This architecture has two fully connected layers as two main components. One layer FC\(_d\) learns the latent representation of the input data and another layer FC\(_k\) learns the relevant prior knowledge representation of the input data. Firstly, each feature \(x_i\), where \(1 \le i\le n\), of a tweet of n words is sequentially fed to a LSTM unit for sequentially representing the features in the hidden states (or latent space) \(h_i\). We take the final hidden state \(h_n\) that accumulates all prior hidden states in the sequence as a summary and feed it to a fully connected layer FC\(_d\). The output of this FC\(_d\) is the latent representation of data in our model. The goal of this component is to learn sentiment prediction features from training data. However, if there are not enough training data, this component may overfit the training data. Therefore, simultaneously, we send the input features (\(x_1, \dots x_n\)) to Expert Knowledge component to obtain three scores (positive, negative and neutral) for a tweet based on prior knowledge. These three scores are fed to a fully connected layer FC\(_k\). The output of this FC\(_k\) is the prior knowledge representation of sentiment analysis. The goal of this component is to regularise sentiment prediction in final layer \(FC_s\). This regularisation helps in reducing model overfitting when there are not enough labelled data. The outputs of FC\(_d\) and FC\(_k\) are concatenated [FC\(_d\), FC\(_k\)] and fed to final fully connected layer FC\(_s\). Output of FC\(_s\) is distributed over sentiment classes and normalised through a softmax function. Our proposed model INN is evaluated in Sect. 3.2.

The proposed SNTM, shown in Fig. 5, is inspired by LDA Blei et al. (2003) and implemented utilising a neural network architecture named Variational Auto-Encoder (VAE) Kingma and Welling (2013).

The idea behind LDA Blei et al. (2003) is that observed terms in each document (or text such as tweet) are generated by a document-specific mixture of corpus-wide hidden topics Blei et al. (2003); Bashar and Li (2017); Alharbi et al. (2018). It assumes the number of hidden topics are fixed to T. It represents a topic \(z_j\) as a multinomial probability distribution over the vocabulary (i.e. V terms) as \(p(t_i|z_j)\), where \(1 \le j \le T\) and \(\sum _i^V p(t_i|z_j) = 1\). LDA represents each document d as probabilistic mixture of topics as \(p(z_j|d)\). Therefore, the probability distribution of ith term in a document d can be modelled as a mixture over topics: \(p(t_i|d) = \sum _{j=1}^T p(t_i|z_j)p(z_j|d)\). Here the only observable variable is \(p(t_i|d)\). The other two variables \(p(t_i|z_j)\) and \(p(z_j|d)\) are hidden. In this research, we use VAE Kingma and Welling (2013) for learning the two hidden variables \(p(t_i|z_j)\) and \(p(z_j|d)\). Using VAE, we directly map a document to an approximate posterior distribution \(p(t_i|d)\). More specifically, VAE maps d to \(z_j\) for \(1 \le j \le T\) using the encoder part, i.e. encoder part approximates \(p(z_j|d)\). Then, decoder part maps \(z_j\) to \(t_i\) for reconstructing the document d, i.e. decoder part approximates \(p(t_i|z_j)\). Overall the network approximates \(p(t_i|d) = \sum _{j=1}^T p(t_i|z_j)p(z_j|d)\). Suppose the reconstructed document is \({\hat{d}} = p(t_i|d)\) and \(z_j\) is sampled from \({\mathcal {N}}(\mu _j, \sigma _j)\), i.e. \(z_j \sim {\mathcal {N}}(\mu _j, \sigma _j)\). The loss \(L_1\) for the reconstruction is calculated as follows.VAE regularises distribution \({\mathcal {N}}(\mu _j, \sigma _j)\) by enforcing the distributions to be close to a standard normal distribution \({\mathcal {N}}(0,1)\) through minimising KL divergence Kullback and Leibler (1951) of them. This regularisation prevents the model to encode data far apart in the latent space and encourage returned distributions to overlap. Thereby, this model satisfies expected continuity and completeness, where continuity means two close points in the latent space should not yield completely different contents once decoded and completeness means for a chosen distribution, a point sampled from the latent space should yield meaningful content once decoded. Therefore, the loss \(L_2\) for the reconstruction is calculated as follows.However, random sampling (to obtain \(z_j\)) that occurs in the encoder part of VAE will prevent backpropagation through the network. Therefore, the sampling process is expressed using reparametrisation trick that allows the gradient descent possible Kingma and Welling (2013). Reparametrisation is done as \(z_j = \mu _j + \sigma _j \zeta\), where \(\zeta = {\mathcal {N}}(0,1)\).

In some cases, we may have a portion of documents that are already labelled to certain topics or classes. Such labels can be leveraged to learn discriminative features useful for gathering similar features under similar topics, and separate topics from each other. Therefore, we use a classification layer in decoder part of the model. The input to this layer comes from the latent representation of document in encoder part of VAE. Suppose the label of a document is y and the predicted label is \({\hat{y}}\). The overall loss of our deep topic model can be written as.To improve topic coherence, similar to ProdLDA Srivastava and Sutton (2017), we assume distribution over individual words is a product of experts Srivastava and Sutton (2017) rather than the mixture model used in LDA. Similar to Combined Topic Model (CTM) Bianchi et al. (2020), we use contextualised document embeddings from Sentence-BERT (a extension of BERT that allows quick generation of sentence embeddings) Reimers and Gurevych (2019). Firstly, the document embeddings are projected to the same dimensionality as the vocabulary size through a latent layer, and then concatenated with the Bag-of-words representation. Our proposed model SNTM is evaluated in Sect. 3.3.

Choosing a reasonable number of topics is important. Too few topics could lead to merging distinct topics whereas too many topics could result in fragmented topics that otherwise could make a cohesive topic together. We manually evaluate topic models with topics ranging from 5 to 50, to determine the optimal number of topics.

In Bag-of-words representation, we remove keywords and hashtags (e.g. covid19, coronavirus, etc.) that we used for collecting our tweets. This ensures that the topics discovered are meaningful and not dominated by the same top words. We also remove rare words (i.e. words with a very low frequency) to reduce noise in the topics.

Finally, we use dynamic topic modelling in Blei and Lafferty (2006) to observe how topics evolve over time. Dynamic topic modelling can capture the evolution of topics in a sequentially organised collection of tweets or documents. For example, the tweets published in different time periods can be related to a specific topic namely coronavirus cure, however the topic of ‘coronavirus cure’ can appear differently in later time stage than the early stages. The themes in a tweet collection evolve. It is of interest to explicitly model the dynamics of the underlying topics. In our setting, tweets are grouped by weeks and ordered by successive weeks, to understand their weekly evolution during the period of data collection.

Table 2 compares the experimental performance of the proposed INN model with baseline models Vanilla LSTM (VLSTM) Hochreiter and Schmidhuber (1997) and Universal Language Model Fine-tuning (ULMFiT) Howard and Ruder (2018). We use the same LSTM architecture in INN and VLSTM to represent the data. INN has an additional neural network part to represent the prior knowledge that VLSTM does not have. ULMFiT is a state-of-the-art classification model that has a medium size parameter set. Compared with INN and VLSTM, ULMFiT is a large neural network based on LSTM and uses a pretrined language model. For evaluating models, we used six metrics such as Accuracy, Precision, Recall, F\(_1\) measure, Cohen Kappa Score, Area Under Curve. A description of these metrics is available in Bashar et al. (2020).

Table 2 shows that, in all six measure, INN performs significantly better than VLSTM and ULMFiT on the COVID19Senti dataset. Compared with VLSTM, the performance improvement of INN is 27.79% for Cohen Kappa Score, 11.61% for Area Under Curve, 9.63% for Accuracy, 9.43% for Precision, 6.98% for F\(_1\) Measure and 4.52% for Recall. Compared with ULMFiT, the performance improvement of INN in different measures range from 8.44% to 2.3%. On the GeneralSenti dataset, the performance of INN is better than VLSTM in all six measures. For example, the performance improvement is 5.51% for Recall, 4.40% for F\(_1\) Measure and 4.76% for Cohen Kappa Score. On this dataset, INN outperforms ULMFiT in all the measures except precision. Even though ULMFiT has marginally better precision, its Recall and F\(_1\) measure is lower. In other words, ULMFiT achieves better precision by sacrificing Recall, i.e. ULMFiT misses more number of positive sentiments than INN. On the Senti140 dataset, INN performs marginally better or similar to VLSTM and ULMFiT.

Senti140 is a large dataset that include 1.6 M annotated tweets. Due to its large size, this dataset covers sufficient information. Consequently, the addition of prior knowledge in INN does not contribute much for performance enhancement. On the other hand, COVID19Senti and GeneralSenti are small datasets where integration of prior knowledge contributed the most. Even if the prior knowledge has limited contribution in challenging dataset (such as GeneralSenti) due to context dependency, prior knowledge can still help in improving the overall model such as INN. For example, when we used prior knowledge-based Lexical sentiment analysis on GeneralSenti dataset, we obtained Accuracy 0.219607161, Precision 0.04735092, Recall 0.526367669, F1 Measure 0.086885779, Cohen Kappa Score -0.048870461 and Area Under Curve 0.361347359. In spite of this poor performance from Lexical sentiment analysis, prior knowledge improved the performance of INN in this dataset.

The significance of this finding is that prior knowledge integration with machine learning model can significantly benefit small datasets and such integration is always beneficial, even for large datasets. In this research, we only utilised lexicon and rules C. J. Hutto (2014) as prior knowledge. Exploring other source of prior knowledge will prove the robustness of the proposed model INN in other tasks. We will investigate this in our future work.

The following results of sentiment analysis are based on the proposed INN-based model applied on Twitter conversation in the Australian Sphere dataset. Figure 10 shows geospatial and temporal distribution of the ratio of positive sentiment tweet counts vs total tweet counts. As soon as COVID19 hit the world, the positive sentiments dropped sharply (roughly from 85% to 48% on average). The percentage stayed there for up to around 12 weeks. Then it gradually changed for three weeks with a very marginal positive increment. For the final two weeks, the increment was a bit more than the previous three weeks.

A possible explanation of the trend can be given as follows. As soon as COVID19 hit the world, the online community got shocked by the news. It took some time for world leaders to come up with plans on how to combat COVID19. During this period (12 weeks) people remained stressed. When the world leaders explained their combat plans and ideas, twitter users talked about those positive initiatives during this period (three weeks). In the final two weeks, the Australian government announced social safety plans, e.g. economic aids to organisations, businesses, and individuals; it announced more strict rules for social distancing and the COVID19 infection curve started flattening. People started to become slightly comfortable and discussed these positive aspects in their tweets. Consequently, the number of positive tweets increased. All these patterns show that monitoring conversational dynamics on social media can reveal how people feel during the COVID19 pandemic, and what initiatives work and makes people comfortable.

To have a closer look into the trend, the temporal and geospatial dimensions are decoupled in Figs. 11 and 12, respectively. Figure 11 shows the volume of COVID19 related tweets (total volume), the volume of positive tweets related to COVID19 (positive volume), and their ratio (positive vs total ratio). This figure shows that, roughly at any time, among all the COVID19 related posts, only 50% of them were positive. We see two significant drops in the ratio of positive sentiments, one is at the beginning when the world was hit by COVID19 and the next one is by week seven or third quarter of January 2020. During this period there were not many discussions of COVID19 in Australia. However, the second drop triggered an increase in the number of COVID19 related posts. In other words, this second drop alerted the community about the upcoming danger of COVID19. We can assume that the small number of tweets related to COVID19 might come from the people who are Journalists, social workers, health care workers, or people who are conscious of health issues.

During the period when a noticeable number of posts were related to COVID19 (week 8 to 18), there are two small drops in the ratio of positive sentiments. One in week 10 and another in week 14. Both drops are followed by a significant increase in the number of COVID19 related posts. Even though these two drops are small in sentiment ratio, the drops in the number of positive tweets were large enough to initiate triggers. It ascertains that monitoring the positive sentiment tweets can signal us the trigger in the increase in COVID19 related posts.

Figure 12a shows how the ratio of the number of positive tweets vs total tweets varies in Australian states and territories. It shows that all states and territories have the positive sentiment tweet ratio of around 0.5 except Northern Territory that has a slightly better ratio. This implies there is emotional stress in people over all the states and territories. However, this figure does not clearly capture the positive sentiment drop as cities are averaged over in the states and territories. In reality, some cities are affected more than others by COVID19. Therefore, we add capital cities in Fig. 9 along with states and territories.

Figure 12b shows the counts of COVID19 related tweets and positives tweets in states, territories, and capital cities. Capital cities and states that have a significant drop in positive tweet count are Sydney (syd), Melbourne (mel), Victoria (vic), and Queensland (qld). A comparison between Figs. 12b and 9b shows that these locations had most of the COVID19 cases. A drop in positive sentiment is correlated with the number of COVID19 cases. A drop in positive sentiment is also correlated with early mental health issues, informing that the community might need an allocation of mental health care resources in the near future.

Two interesting facts in Fig. 12b can be observed in varied behaviour between two pairs of state and its capital city, (qld, bne) and (nsw, syd) pairs. There is a significant drop in positive tweets in qld but not in bne. The majority of COVID19 cases in Queensland was in Gold Coast and other surrounding areas rather than Brisbane. Again, there is a significant drop in positive tweet count in syd but not in nsw. The majority of COVID19 cases was in Sydney rather than the other parts of nsw. This again emphasises that a drop in positive sentiment is directly correlated with the number of COVID19 cases.

Table 3 compares the performance of the proposed SNTM model with three baseline models namely CTM Bianchi et al. (2020), LDA Blei et al. (2003) and Non-negative Matrix Factorisation (NMF) Andrzej CICHOCKI (2009). We evaluate the models using three metrics namely Normalised Pointwise Mutual Information (NPMI) Lau et al. (2014), External Word Embeddings Topic Coherence (EWETC) Ding et al. (2018) and Inversed Rank-Biased Overlap (IRBO) William Webber (2010). NPMI measures how related the top-n words of a topic are and take average over the T topics. It considers the words’ frequency in the original dataset. EWETC measures how similar the top-n words in a topic when their external word embeddings Mikolov et al. (2013) are considered. More specifically, it computes the average pairwise cosine similarity of the word embeddings of the top-n words in a topic, then it takes average over the T topics. In this setting, word embeddings from Mikolov et al. (2013) are used. IRBO evaluates how diverse the topics generated by a model are. It uses the reciprocal of the standard RBO William Webber (2010).

Table 3 shows that our proposed SNTM model performs the best overall. It always gives the best IRBO performance with a high score, which means the topics generated by SNTM are diverse. This might be because SNTM utilises contextual information and some label information. Since both SNTM and CTM use contextual information, their IRBO score is significantly better than LDA and NMF. As SNTM additionally utilises labelled information its IRBO score is better than the state-of-the-art-model CTM. EWETC score gives an estimation of coherence in a topic. For EastAsianHate and RandomHate datasets, SNTM gives the best EWETC score and this score is similar to NMF for COVID19Senti. NMF provides the best NPMI score for EastAsianHate and RandomHate datasets, while a similar NPMI score is achieved by SNTM and CTM for COVID19Senti. The reason for NMF providing the best NPMI score might be related to two facts. 1) both NMF and NPMI disregard the contextual information. 2) NPMI is computed on the original data as computing NPMI on an external corpus is expensive Bianchi et al. (2020). Ding et al. Ding et al. (2018) pointed out that topic coherence computed on the original data is inherently limited. Overall, SNTM gives superior results for topic modelling even though there are room for optimising the architecture of SNTM in terms of the number of hidden layers used and the number of neurons used in each layer. In our future work, we will investigate in that direction.

We now show some of the experimental results on how COVID19 related topics changed over time semantically, morphologically, and sentimentally in the Australian Sphere dataset using the results of SNTM. Figure 13 shows the evolution of five topics; Topic 0: controlling the spread, Topic 1: staying in isolation and working from home, Topic 2: COVID19 cases, Topic 3: racism against the Chinese community, and Topic 4: impact of COVID19 outbreak worldwide.

Topics 0, 2, and 4 show a similar trend even though their magnitude and change rate are different. A close investigation shows that these three topics share a high similarity in subject matter. On the other hand, Topics 1 and 3 do not resemble any trend. However, they somewhat inversely follow each other. It is apparent that all the topics evolved over time in terms of semantics, morphology and sentiment. For example, in Topic 0 that talks about controlling the spread of coronavirus, the words ‘need’ and ‘worker’ newly emerged during weeks 11 and 13, whereas the words ‘island’, ‘china’, ‘travel’, and ‘ban’ lost their significance during the weeks 12, 15, 16, and 17, respectively.