Analyzing the changing landscape of the Covid-19 vaccine debate on Twitter

Data was collected from Twitter in the period from January 1st, 2020 to April 30th, 2021. The data collection process was carried out using the Twitter API ,³ which is publicly available, through the full-archive historic search endpoint and academic research product track. According to the API specification, only publicly available data in compliance with users’ privacy settings can be accessed. We collected tweets containing an URL and at least one keyword related to Covid-19 vaccines, such as ‘immune’ and ‘pharma’ (see Table 1). The final dataset consists in 3, 797, 305 tweets published by 985, 523 accounts.

For each tweet, we derived three different attributes: the language, the trust score, and the country of origin. The language was automatically detected by Twitter. The trust score was retrieved from NewsGuard, a tool that provides trust ratings for news and information websites on the basis of nine journalistic criteria. These criteria are individually assessed and then combined to produce a single ‘trust score’ from 0 to 100 for a given news media outlet. The scores are assigned by a team of journalists. Scores are not given to platforms (for example, Twitter and Facebook), individuals, or satire content. The criteria evaluate basic practices of credibility and transparency of news sources .⁴ Here, we use NewsGuard’s trust scores to distinguish between questionable and reliable tweets. According to NewsGuard documentation, a source is classified as questionable if its trust score is below 60, while it is considered reliable if the score is equal to or above 60. Therefore, every tweet was categorized according to the trust score of the source (URL) it referenced.

NewsGuard provides trust scores for information sources in five countries, including Germany, France, the UK, Italy, and the USA. Despite the ease of determining the country of origin for a source, assigning a country to a tweet is not always a straightforward process due to the lack of precise geolocalized data. This challenge is particularly pronounced when analyzing English language content, given its global usage. Moreover, we faced the challenge of determining how to assign tweets that mention sources originating from multiple countries.

To address these issues, we implemented a strategy of assigning tweets to a country only if the language and source information were consistent. By doing so, we sought to minimize any ambiguity that may arise from tweets containing mixed-language content or sources from multiple countries. For example, consider a tweet in English language referring to multiple sources in France, UK, and the USA. According to our strategy, the tweet is considered part of the debate in both UK and USA, but not in France. A robustness test was performed using alternative strategies, and no significant difference emerged .⁵ Additionally, to mitigate potential biases in the keyword selection toward either reliable or questionable news outlets, we computed the percentage of each keyword within both categories (see SI for details). The analysis revealed some minor variations between the two categories, particularly in terms of the frequency of vaccine brand and name mentions. However, these differences do not compromise the validity of our study.

To investigate the presence of segregated communities, we built and analyzed the retweet networks for each country. In particular, we built a weighted undirected network \(N= (V, E, w)\) with vertices V that represent the accounts retweeting or being retweeted at least once, where two vertices \(u, v \in V\) are connected with an edge \((u,v) = e \in E\) if and only if u retweeted v or v retweeted u. We assigned a weight w(e) to each edge \(e = (u,v)\) corresponding to the number of retweets between u and v and a trust score to each vertex \(v\in V\) corresponding to the average trust score of its tweets. Then, We employed the Louvain clustering algorithm (Blondel et al. 2008) to identify groups of accounts that exhibit greater connectivity with each other compared to the rest of the network.

To assess the evolution of communities and the possible changes in the consumption of questionable content, we employed dynamic clustering techniques. For each country, we created a Dynamic Network DN \(= (N_1, N_2, N_3)\), each DN consisting of three ‘snapshots’ \(N_t, t = 1,2,3\). The three snapshots were obtained by considering the retweets produced in the following time intervals: from January 1st 2020 until the Pfizer announcement (\(t=1\)), from Pfizer announcement until EMA announcement (\(t=2\)), and from the EMA announcement until April 30th 2021(\(t=3\)). On each DN we employed smoothed Louvain dynamic community detection (Aynaud and Guillaume 2010), which performs Louvain clustering on each snapshot, using the clustering on the previous slice as initialization. We used the default Jaccard similarity as the matching function and the default matching threshold of 0.3 (Greene et al. 2010).

To better understand the narratives of these communities, we considered the original tweets and extracted topics using BERTopic (Grootendorst 2022), a state-of-the-art topic modeling tool that extracts latent topics from a collection of documents. We removed URLs from the text and embedded the documents using 768-dimensional pretrained sentence classification embeddings. Afterward, we reduced their dimensionality to 5 through UMAP, using 15 neighbors and cosine similarity. Finally, we clustered them with HDBSCAN, employing Euclidean distance, and minimum cluster size of max\((10, \frac{N}{10^3})\), where N is the number of tweets to be clustered. Topic names were assigned by manually inspecting the most representative tweets provided by BerTopic for each cluster of tweets.

To study the evolution of content production from reliable and questionable sources over time, we quantified the changes in the production of questionable and reliable tweets in conjunction with the first Covid-19 vaccine announcement by Pfizer (09/11/2020) and the AstraZeneca suspension by EMA (09/03/2021). To do so, we fitted a model to questionable and reliable time series for each country, resulting in a total of ten time series. The model was designed to capture the underlying structural patterns of tweet counts in response to the events. We assumed the time series to be independent of each other and applied a Box–Cox transformation with \(\lambda = 0\) to stabilize the variance of the dependent variable \(Y_t\). Autocorrelation in the series was addressed by employing dynamic modeling and fitting a linear model with an ARIMA (p, q, r) error term. This resulted in the modelwhere \(Y^{(\lambda )}_t\) indicates the Box–Cox transformation of the original time series Y at time t. Changes in average tweet counts are captured by PFZ\(_t\) and EMA\(_t\), indicator variables which are set to 0 before the events and 1 otherwise. These changes are deprived of fluctuations in the time series, which in turn are captured by \(\eta _t\), an error term behaving as an ARIMA(p, q, r) process that optimizes BIC(Hyndman and Khandakar 2008). The relative change \(\Delta X = \frac{X_{t_2}-X_{t_1}}{X_{t_1}}\) (Table 2) measures how much the expected tweet production of \(X \in \{Q,R\}\) changed from \(t_1\) to \(t_2\), where Q and R stand for questionable or reliable content, and \(X_{t_1}\) and \(X_{t_2}\) refer to the volume of X tweets produced until time \(t_1\) and \(t_2\), respectively.

While our analysis offers valuable insights and is based on robust and widely utilized techniques, it is important to acknowledge certain limitations. Firstly, our estimation of user reliability is based on the sources they share. Although this approach is scalable and commonly employed in social media analysis, it may misclassify users who share articles with satirical or critical intents, as it does not consider the broader contextual information. Nonetheless, this approach has demonstrated its reliability, enabling consistent analysis, and aligns with similar techniques employed in related studies (Cinelli et al. 2021; Flamino et al. 2023). Secondly, in classifying news domains, we relied on data from NewsGuard, which may have varying coverage in each country. However, NewsGuard rates all sources that account for (at least) 95% of online engagement in each country. Thirdly, in the topic modeling analysis, we assigned names to each cluster of documents manually. To minimize arbitrariness, we began with the most representative documents identified by BerTopic, following the methodology of previous studies (Falkenberg et al. 2022). Lastly, while our set of keywords encompasses a wide range of topics related to Covid-19 vaccines, it is possible that we may have missed some niche keywords. Nevertheless, as mentioned in Sect. 2, our dataset appears to be balanced with respect to different types of sources.

We first look at the differences and similarities in the structure of the vaccine debate across countries relying on interaction networks (see Sect. 2.3 for details).

In Panel A of Fig. 1, the results of Louvain clustering on retweet networks for each country are presented. Each bar represents a cluster found in the network and its average trust score is color-coded. The clusters are arranged in ascending order of average trust score. Across all countries, the majority of communities are made on average by highly reliable users, with at least 74% of them having a high average score. However, a small number of clusters with low average scores are present in all countries.

Panel B of Fig. 1 reports the joint distribution of the accounts’ individual trust score against the average trust score of their neighbors. We observe the presence of separated communities laying on the diagonal, indicating that users tend to interact with others sharing sources with similar trust scores. The observed pattern supports the presence of an echo chamber effect, for which users tend to interact with others who hold similar views and rarely engage with those who have opposing perspectives.

To better understand the topics of discussion in each country for questionable and reliable sources, we conducted topic modeling analysis (see Sect. 2.3 for details).

Here, we concentrate on the distinctions in the overall narratives conveyed by the two groups, and therefore, we confine our analysis to the static version, without considering the temporal evolution of the debate. In the subsequent subsection, we delve into the temporal dynamics of the discourse. The results presented in Fig. 2 reveal some differences in the types of topics discussed: while both categories of tweets addressed vaccine side effects, this topic was more frequently discussed among questionable sources compared to reliable sources. Moreover, both types of tweets in each country also touched on political topics related to the government’s agreements to obtain, approve, and suspend vaccines. However, the narrative of the discussions differed between the two groups. Questionable sources often expressed concerns about excessive government control, while reliable sources focused on inefficiencies in the vaccination campaign and the lack of clarity in AstraZeneca vaccine recommendations. The topic of vaccine efficacy was also discussed in both groups, but the nature of the discussions differed significantly. While reliable sources mainly addressed the efficacy statistics of the vaccines, questionable sources talked about alternative treatments, signaled by the presence of terms like ‘hydroxychloroquine’ or ‘vitamins.’ Furthermore, they questioned the usefulness of vaccines and challenged medical studies on their efficacy. One of the prominent topics discussed by reliable sources was Covid-19 official reports. On the other hand, in questionable sources topics such as fear of side effects (including concerns over ‘facial paralysis,’ ‘DNA damages,’ ‘abortion,’ and ‘fertility risks’) and conspiracy theories (such as ‘Bill Gates’ involvement in the pandemic, ‘illegal experiments on humans,’ ‘sterilization programs,’ and ‘chip implantation’) were more present.

Figure 3 shows the amount of content (number of tweets) produced by questionable and reliable sources over time. We notice that content production increased in correspondence with both the announcement of the Pfizer vaccine (November 9th 2020) and the EMA suspension (March 9th 2021). To better understand the changes in tweet production following the considered events, we fitted a dynamic model to data, represented by the gray line in Fig. 3 (see Sect. 2.3 for details). Our models indicate that there was a statistically significant increase in tweet production of both questionable and reliable sources following the Pfizer announcement, however, some differences among countries arise when comparing the growth of questionable and reliable content. Indeed, while Germany experienced a higher growth of questionable sources in comparison with reliable ones, in all the other countries, reliable sources grew more than questionable ones. To measure these differences, we calculated the relative changes in content production and compared them by computing the ratio between the changes in questionable content production and those in reliable content production. If the growth rate of questionable sources surpasses that of reliable ones, the resulting ratio will be greater than 1, whereas it will be lower if the opposite is true.

The results presented in Table 2 show that all countries but Germany experienced a greater growth in the number of reliable tweets following the Pfizer vaccine announcement. This can be explained by the different volumes of questionable sources before the event: while Germany had a very small volume of tweets referring to questionable sources (10% of reliable sources volume), in France this presence is higher than in any other country (48% of reliable sources volume).

This suggests that the questionable debate in France may have experienced a saturation effect due to the already high preexisting volume of questionable content, resulting in a smaller growth in questionable tweets after the event. Conversely, statistics on Germany reveal a lower level of tweet activity that saw the largest growth in the number of questionable tweets. Nonetheless, the volume of questionable sources remained significantly lower than that of reliable sources in all countries.

Next, we examine the effect of the suspension of the AstraZeneca vaccine by EMA, which represents the second event under consideration. The models reveal an increase in both questionable and reliable content, but the magnitude of this change is significantly lower than that observed during the vaccine announcements. Indeed, the second half of Table 2 shows that the increase is one order of magnitude smaller. Moreover, Italy is the only country that shows statistically significant increases in both questionable and reliable sources. In contrast, neither the UK nor the USA show statistically significant changes in either type of source.

Given that the temporal analysis has revealed significant changes in the prevalence of questionable and reliable sources, a natural question is whether these events are also associated with changes in the community structure and opinion of users. To study the evolution of the communities, we performed Dynamic Community Detection on the networks. This approach enabled us to track the changes in trust score over time, at both the community and account levels. Figure 4 shows the evolution of the communities during three distinct time spans: before the Pfizer announcement, between the Pfizer announcement and the AstraZeneca suspension, and after the AstraZeneca suspension. The distribution of clusters varies significantly across countries, with France having a sizable community of questionable accounts, while other countries like the UK and Germany have much fewer such accounts, consistent with the findings in the previous sections. Moreover, we can notice that, in the final snapshot, the merging of the most questionable community with a reliable one resulted in a community with an average score in the reliable range, whereas in other countries such as Italy and the UK, the communities appear to be more stable over time. Also, we observe that considering the trust score threshold of 60, the communities tended to remain relatively stable over time, with only a few accounts shifting between them. Overall, we may notice that the communities with a trust score threshold lower than the country’s average tended to exhibit greater cohesiveness and stability, while the more reliable communities appeared to be more dynamic and scattered. One possible explanation for this phenomenon could be that accounts within low-trust-score communities tend to be more active and engaged in the debate, while accounts within higher-trust-score communities are characterized by higher turnover rates. This is consistent with findings from other studies, which suggest that polarized users tend to be more active and engaged in debates (Schmidt et al. 2018). Differences across countries in terms of both production and consumption of questionable content, and response to external events might be influenced by a variety of factors, including level of trust in institutions and social environments (Zimmermann et al. 2023; Sturgis et al. 2021).