Reactions to science communication: discovering social network topics using word embeddings and semantic knowledge

The COVID-19 pandemic has altered the information-seeking behavior of people, much like other crises. This is evident from the shift in web popularity rankings, which indicates that individuals have specific information needs during times of crisis communication (Dreisiebner et al. 2022). As a result, they face challenges in determining the reliability and trustworthiness of the information they receive (Barnwal et al. 2019). Social media channels have become important sources of diverse scientific communication content aimed at meeting these needs. Given the significance of scientific understanding during a crisis, it is crucial to comprehend the patterns of information-seeking behavior (Montesi 2021). In particular, media creators must understand the quality criteria that users utilize in selecting their resources and the factors that influence their preferences. While information resources in general and multimodal science communication, in particular, differ in their portrayal of scientific information, research has not delved into the detailed examination of the most effective ways of disseminating scientific information.

With the aim of examining the online conversation surrounding the Corona crisis, specifically in the context of science communication, the goal of this study is to develop a solution that can help creators filter out feedback in the middle of millions of comments made in relation to their work. Our focus is on identifying and analyzing a specific subset of comments that users post in science communication channels as responses to the content presented. For the goals of this study, we collected 1.12 million tweets formed by the comments in a network of Brazilian scientists, governmental bodies, doctors and scientific communicators. It is worth noting that the majority of these comments are posted at scientific channels are not necessarily related to the content or format but rather pertain to the broader discourse around the Corona crisis. They often include political viewpoints and general comments on the crisis.

First attempts into filtering out this data were performed with topic modeling through traditional algorithms such as Latent Dirichlet (Blei et al. 2003), which proved to not be good enough in encountering niche topics when dealing with short documents in large data collections (Lima 2023; Mandl et al. 2023). After manual validation of our topic model, an ensemble method of document filtering was created through the creation of a word dictionary made out of the most relevant words in topics relevant to scientific communication and their top-n closest neighbors according to the cosine similarity of their word embeddings. A filtering heuristic that uses this dictionary was put in place, and managed to lead to severe improvements in our ability to filter documents related to scientific communication. With such techniques, this paper introduces an ensembling framework that combines topic modeling and word embeddings as steps that retroactively feeds data processing, generating a very effective dataset for machine learning tasks that need to identify niche topics in large databases, as seen in Fig. 1. The data and solution obtained from this study allows for an examination of tweets that were positively received in relation to science communication strategies during the COVID-19 pandemic. This information can be utilized to enhance the dissemination of scientific information during future crises. The solution devised is also theme-agnostic, and could be applied to other instances of filtering out niche topics in social media data.

The research of popular science communication channels indicates comparable communicative techniques taking into account aspects such as scientific insecure communication, factuality, complexity, emotionalization, and expert presentation. Because social media communication is multimodal, visual information, location, and body language are all important. Understanding how epidemiological information is transmitted and understood by non-expert audiences requires extracting meaningful comments from internet audiences (Jaki 2021). These could be helpful for improving communication strategies and for adapting scientific information to online audiences.

Previous research on public scientific material relating to the COVID-19 pandemic, on the other hand, has primarily concentrated on qualitative analysis, ignoring internet audience reactions (Bucher et al. 2021). While datasets available for social media communication about Corona are available, it is a considerable challenge to filter this volume of information, being shown that social media information propagation during the pandemic has largely shone a focus around general trends, political attitudes, and the spread of misinformation, which surpasses true information in spread (Vosoughi et al. 2018).

In such, there is a need for comprehensive studies on the quality of science communication, including topic modeling and classification, specifically exploring reactions to science communication with the public. Several studies have been already conducted in exploring topic modeling in regards to social media data during the COVID-19 pandemic (Boon-Itt and Skunkan 2020; Melo and Figueiredo 2021; Yin et al. 2022), but few tackled the topic of scientific communication. Much rather, such studies take a special perspective. Often machine learning is used to improve a sentiment analysis, e.g., for the attitude toward vaccines in Turkey (Küçük and Arıcı 2022). Similarly, LDA has been used to detect topics in the discourse on vaccines and the documents were further processed with deep learning methods to extract to determine the sentiment in a large collection (Zulfiker et al. 2022).

In a previous attempt to focus solely on tweets reacting to science communication, LDA modeling found only few comments (Lima 2023).

When analyzing short textual data, topic modeling techniques are helpful for clustering similar documents within a corpus and for finding common themes they may share. While Latent Dirichlet Allocation (LDA) (Kalepalli et al. 2020) exhibits a strong performance in the diversity of their generated topics and in its accurate categorization, more modern solutions based on transformer architectures such as BERTopic and Top2Vec, which are based on ensembled techniques and word vectorization, achieve better results in topic coherence (Egger and Yu 2022a) and have an array of different modeling strategies that tend to generalize toward more interpretable topics. Although there are comparative evaluations for LDA, BERTopic and other models, they do not give advice for filtering small amounts of tweets with high accuracy (Egger and Yu 2022b).

When looking for such niche topics, Guided BERTopic’s strategy of nudging topics together by their word similarity produces more interesting results when looking for a specific theme inside a corpus (Grootendorst 2022).

Futhermore, most of this work is done for English (Ng et al. 2022), so research for other languages on the COVID-19 discourse is necessary.

Although machine learning techniques and topic modeling has come a long way, the means of measuring topics by their human interpretability are still unclear on which are the best practices, with the often used coherence metric not being able to accurately depict actual human interpretability (Ramírez et al. 2012) and a comparison to human judgment is difficult to quantify (Chang et al. 2009).

Interestingly, metrics based on the utilization of word embeddings, such as measuring for cosine similarity, reflects into more accurate measurements of interpretability (Doogan and Buntine 2021). The construction of word dictionaries used to aid machine learning techniques in better handling of a specific problem set has been proven to be effective (Reveilhac and Morselli 2022), and is also used in several strategies of BERTopic. This approach is specifically helpful when dealing with word embeddings, with the utilization of Google’s Word2Vec model in conjunction of keyword lexicon lists proving that the mixture of techniques tends to prove successful, enhancing the capability of word embeddings inside a model, specially when dealing with filtering specific information which form a minority class inside a larger dataset (Hu et al. 2017; Koufakou and Scott 2020; Jin et al. 2018).

This study went through an iterative process between techniques: from beginning with an LDA topic model, we repurposed its results into a validated dataset that was used to train a Word2Vec model, as well as nudging BERTopic into better generalizing our niche topic.

In the first stage, relevant science communication channels on Twitter were identified. Then, data from these sources containing content created during the COVID-19 pandemic were collected and underwent several textual data processing techniques, resulting in a final corpus that was prepared for natural language processing. Lastly, a topic model was created to organize the massive text data into certain themes. The goal was to identify topics that grouped terms related to reactions to scientific communication. The effectiveness of the topic model was evaluated using the NPMI metric.

The results of these topic models were then manually analyzed by our team and classified into certain categories that pertained to science communication and if they were relevant or not. Relevant documents were those which commented on the design or content of the scientific postings.

The most relevant words for the topics with the largest concentration of relevant documents were sorted into lists by their categories. From these words, a Word2Vec model was trained on a lemma database formed from the corpus. Then, embeddings were created for the sorted word lists, and their top-5 nearest neighbors were added to the lists. With these list of words, we compared them to the corpus, using the trained Word2Vec model, and measured their proximity by cosine similarity and also by overall word count. Using this mixture of techniques, we managed to filter a considerably larger amount of relevant documents than by relying solely on topic modeling. The results were analyzed and validated, as well as compared against a BERTopic model trained on the database.

While this study focused on finding reactions to scientific communication in social media, these same techniques can also be applied when searching for any niche topic in collections of documents similar in size to tweets. We found that even state-of-the-art models such a BERTopic or Word2Vec needed a certain degree of heuristical validation to be able to generalize into niche topics. The approach that required the most amount of manual validation and combination with heuristics, which was the semantic filtering technique with Word2Vec embeddings, was also the most successful in filtering through our dataset and finding our desired topic.

The framework defined in this paper can be applied to help scientific content creators to better locate useful feedback in very large datasets, as well as point out, after a brief analysis, which subjects are the viewers most interested at, and what kind of content delivery are the most effective for good retention and positive feedback. With this, content creators are able to better understand how to broadcast their knowledge to different audiences, and through different avenues.

One big hurdle that this study had to deal with it was the nature of Twitter’s data: scientists and content creators do not tend to post educational content on Twitter, and prefer to use it to share important news and foster discussion—this characteristic of the social network led to difficulties in finding feedback toward scientific communication, and, when coupled with Twitter’s closing of their academic access API in March of this year, leads our future studies in the direction of gathering documents from other sources such as Youtube.

With a large dataset of thousands of tweets classified by their relevance toward our theme, this study also contributed with the data labellng necessary for an interesting possible next avenue: the training of classifier to identify if a document is relevant to scientific communication or not. This classifier can be seen as the end goal of our study: a tool for scientific content creators to better understand what their audience seek for in their content. With such a tool, scientific communication can be optimized to reach larger audiences.