Tracking the structure and sentiment of vaccination discussions on Mumsnet

Graphs are mathematical objects used to represent a complex network. A graph \(G = (V(G),E(G))\) is composed of a set of nodes V(G) and a set of edges \(E(G) \subseteq V(G) \times V(G)\), where each edge is represented by a pair \((u,v) \in E(G)\) for \(u, v \in V(G)\). The graphs we consider in this work are undirected, meaning that the order of the vertices in the pairs does not express direction, so \((u,v)\in E(G)\) implies that \((v,u)\in E(G)\). Graphs are assumed to be labeled, all nodes are assigned consecutive integer numbers starting from 0 and running to \(|V(G)|-1\). The size of a graph is the number of vertices in the graph, written as \(|V(G)|\). A size-k graph is a graph with k vertices. The density of a graph is the portion of edges present in the graph over the total potential ones, calculated as \(|E(G)|/\left( {\begin{array}{c}|V(G)|\\ 2\end{array}}\right)\) in undirected networks.

Graphs can be classified into different types according to constraints or additions to the set of nodes and edges. A graph is called simple if it does not contain multiple edges (two or more edges connecting the same pair of vertices) or self-loops (an edge of the form (u, u) that connects a vertex to itself). In this work, we consider only simple graphs. A graph is called connected if all nodes are reachable from each other. A node \(v \in V(G)\) is reachable from a node \(u \in V(G)\) if \((u,v) \in E(G)\) or if there is a path between u and v, i.e., a sequence of edges of the form: \((u,t_1), (t_1, t_2), \ldots (t_n, v)\). A bipartite graph is a graph whose vertices represent two disjoint groups of entities and there are no edges between members of the same group. Formally, \(G = (V(G_1) \cup V(G_2), E(G))\), such that \(V(G_1) \cap V(G_2) = \varnothing\) and \(E(G) \subseteq V(G_1) \times V(G_2)\), with each edge represented by a pair \((u,v) \in E(G)\) for \(u \in V(G_1) \Rightarrow v \in V(G_2)\) or \(u \in V(G_2) \Rightarrow v \in V(G_1)\). A weighted graph is a graph whose edges represent additional information, a weight, about the relationship between vertices, beyond the existence of that relationship. The most common example of an edge weight is a numeral attribute representing, for instance, the frequency of the relationship. In the case of a numerical attribute, the set of edges is defined as \(E(G) \subseteq V(G)\times V(G) \times {\mathbb {R}}\) and each edge is represented as a triple \((u,v,w) \in E(G)\) for \(u, v \in V(G)\) and \(w \in {\mathbb {R}}\).

The neighborhood of a vertex \(u \in V(G)\) is the set of nodes that u is connected to and defined as \(N(u) = \{ v: (v,u) \in E(G) \vee (u,v) \in E(G) \}\). The degree of a vertex is the number of edges it participates in, which is equivalent to the size of the node’s neighborhood, \(|N(u)|\). A node is called isolated if it has no connections, i.e., its degree is 0. The weighted degree of a node is the sum of weights of all edges the node participates in.

A subgraph \(G_k\) of a graph G is a size-k graph such that \(V(G_k) \subseteq V(G)\) and \(E(G) \supseteq E(G_k) \subseteq V(G_k) \times V(G_k)\) and is called induced if \(\forall u,v \in V(G_k), \; (u,v) \in E(G_k) \leftrightarrow (u,v) \in E(G)\). Two graphs G and H are isomorphic, written as \(G\sim H\), if there is a bijection between V(G) and V(H) such that two vertices are adjacent in G if and only if their correspondent vertices in H are adjacent. A match of a graph H in a larger graph G is a set of nodes that induce the respective subgraph H. In other words, it is a subgraph \(G_k\) of G that is isomorphic to H. The frequency of a subgraph \(G_k\) is then the number of different matches of \(G_k\) in G.

Orbits are unique positions of a graph, calculated by partitioning the set of vertices into equivalence classes where two vertices belong to the same class if there is an automorphism that maps one into the other (Ribeiro et al 2021).

Graphlets are small, connected, non-isomorphic and induced subgraphs (Pržulj 2007). The smallest graphlet considered is a single edge, which can be seen as a size-2 subgraph. An undirected edge has a single orbit (three in the directed case) and the frequency of a node in this orbit is equivalent to the degree of the node. Figure 1 shows the graphlets of size 2, 3 and 4 in undirected networks, alongside the respective orbits.

A graph where all nodes are connected to each other is called a clique. In other words, it is a graph with density of 1. In Fig. 1, corresponds to undirected graphs \(G_2\) and \(G_8\).

NetEmd (Wegner et al 2018) is a network comparison measure that relies on structural features of the network, mainly the distribution of orbit frequencies. The core idea is formalizing the intuition that the shape of the degree distribution is indicative of the network’s generation mechanisms, for instance, a network with a power law degree distribution is generated by a process distinct from a network with a uniform degree distribution. As the graphlet degree vector is a generalization of the degree distribution for graphlets of size \(k \ge 3\), the shapes of the distributions of each orbit also carry information about the topology of the network. Note that because the shape of a distribution is invariant under linear transformations such as translations, using the shape as the focus of the comparison is well suited to comparing networks of different sizes and densities. Wegner et al (2018) define a measure of similarity between distributions p and q, with nonzero and finite variances, using the earth mover’s distance (EMD) (Rubner et al 1998):where \({\tilde{p}}\) and \({\tilde{q}}\) are the distributions resulting of scaling p and q to variance 1. Any distance metric d can be used to generate \(d^*\); EMD was chosen here since it has been shown to be an appropriate metric to compare shapes of distribution in domains such as information retrieval and it produced better results than other distance metrics, like Kolmogorov (\(\ell _\infty\)) or Manhattan (\(\ell _1\)) distances.

Given two networks G and H and a set of m orbits \({\mathcal {O}} = \{o_1, o_2, \ldots , o_m\}\), the NetEmd measure is defined as:where \(p_{o_i}(G)\) and \(p_{o_i}(H)\) are the distributions of orbit i in graphs G and H, respectively. Note that \({\mathcal {O}}\) may be replaced by any set of network features; in this work, we focus on orbits only.

Silva et al (2023) propose an extension of NetEmd by noting that different orbits do not represent independent degrees of freedom and are expected to be constrained by the requirement of combinatorial consistency of the underlying full graph. These combinatorial constraints lead to added noise during the EMD calculation; thus, the authors propose using linear techniques of principal component analysis (PCA) and independent component analysis (ICA) as a noise reduction mechanism. The idea is to project the orbit frequencies to a lower dimension, training the dimension reduction model to lose minimal information while removing noise, project the reduced features to the original dimension and apply the NetEmd equation to the reconstructed frequencies.

The Mumsnet forum has been the subject of a number of vaccination studies (Skeppstedt et al 2017, 2018; Ford and Alwan 2018) with the MMR (measles, mumps and rubella) vaccine being a particular focus (Skea et al 2008; Weller et al 2016).

Skeppstedt et al (2017, 2018) explored 5 threads about general vaccination of children, manually labeling a dataset of 1190 posts as for, against or undecided about vaccination. The authors then built a machine learning model to measure the performance of an automatic classifier on this labeled dataset. The machine learning model chosen was a linear support vector machine (Cortes and Vapnik 1995) and the training data was obtained by preprocessing each post into a list of tokens and n-grams, i.e., sequences of n consecutive words from each post (the chosen values for n were 2, 3 and 4). The authors found among the manually annotated posts an even distribution of sentiment, with 38% for and 41% against. The machine learning experiment yielded a model capable of identifying the sentiment of this corpus with an \(F_1\)-score of 0.44, which lead the authors to conclude that their small training corpus makes classification a very hard task.

Ford and Alwan (2018) used Mumsnet as a platform for a cross-sectional study of influenza and pertussis vaccination uptake during pregnancy. The study’s objective was to learn whether information obtained from online social networks (Facebook, Twitter, etc.) influenced the decision to have these two particular vaccinations. They found that women who used social networks to obtain information were less likely to receive the pertussis vaccination, but this was not the case for the influenza vaccination.

Skea et al (2008) examined two threads about the MMR vaccine from the perspective of parental perception of herd immunity and the duality between parents not wanting to harm their child versus wishing to avoid harm to others. The authors suggested that vaccine promotional material should “include explanations of herd immunity” as it may influence parents’ decisions. Weller et al (2016) analyzed posts and threads related to MMR, finding that by 2010 the majority of the forum was in favor of MMR rather than against it.

The Mumsnet forum is organized in categories, which are further divided into more specialized topics, called subforums, within those categories. Users create threads on these subforums by typing a thread title and some text, called the original post. Other users can reply to the original post by typing in a text box; these replies, called posts, are displayed in chronological order of submission. One of the categories of the forum is health, and within it, there was a subforum about vaccination, which was closed down in July 2019, and another subforum about the COVID-19 pandemic, created in February 2020. We collect two datasets from Mumsnet, one from each of these subforums.

Prior to the closure of the vaccination subforum, we deployed a web scraper to acquire all threads and posts publicly available in the subforum, spanning back 10 years. Following the closure of the forum, we set the start date for our data collection to 1 July 2009, with the last post on this forum dating 28 June 2019. In this time period, there were 1128 threads created, with a total of 31,757 posts, written by 5715 users. As a shorthand, this dataset will be referred to as the vaccine or vaccination dataset.

The COVID-19 subforum contains threads about any topic related to the pandemic, so we filtered these according to their title. We included all titles that contained keywords related to vaccination and immunization (essentially a regular expression matching vaccin*, immun* or *vax*), as well as COVID-19 vaccine-specific terms, for instance, booster or the names of the vaccine manufacturers. There were only 20 threads before 1 November 2020, compared to 81 in November 2020 alone, so we set this as the start date for data analysis. We finalized the data collection on 2 January 2022 and the final dataset only includes posts dated before 2022. In this time period between 1 November 2020 and 31 December 2021, there were 4322 threads created, with a total of 223,625 posts, written by 25,989 users. Again as a shorthand, we refer to this dataset as the COVID-19 dataset, and every time we refer to the COVID-19 forum, we mean vaccination discussions in particular within this forum.

In order to study how discussions vary over time in Mumsnet, we segment the datasets into new and smaller datasets that span shorter periods. One way to accomplish this goal is to create disjoint time intervals so that there is no overlap between the smaller datasets. To do this, we would need a rule for allocation of posts to time periods. Consider the follow situation: A thread is created on 31 January; half the posts within that thread are submitted on that day and the other half are posted throughout February; and time periods are delineated by the first day of each month. One way to assign data to each time frame is by the date of the thread, but this means that the February dataset does not encompass the discussion that took place in that thread throughout February. Alternatively, half the posts within that thread are assigned to the January dataset and the other half to February. In this case, we are separating discussion that took place within a similar context by imposing an arbitrary boundary; it could be argued that a post from 31 January is more similar to a post from 1 February than to a post from 1 January.

To solve this, we create a rolling window of data, akin to a moving average. We assign posts to data slices according to their post date and ensure that related posts on the edge of the time split are assigned to the same data slice at least once. This method of temporally separating the data is controlled by two parameters, the time span, i.e., how much time each window covers, and the time jump, i.e., the amount of time between consecutive windows. The time overlap between consecutive windows, i.e., the amount of time in common between consecutive windows, is calculated by subtracting the time jump from the time span.

The way discussion is carried out in a forum is different from other social media platforms like Facebook, Twitter or Reddit, in the sense that users’ interactions with threads and messages appear sequentially ordered by time, regardless of relevance to the discussion or tagging features used. This structure lends itself to the creation of a bipartite network, as two disjoint sets of entities naturally emerge: threads and users, with a user u connected to a thread t if u wrote a post in t. A common technique for analyzing bipartite networks is to create a projection of one of the partitions, for example when one of the modes in the data holds particular interest over the other (Borgatti and Everett 1997). The most common way to achieve this projection is assuming that if two nodes in one partition are connected to the same node in the other partition, then they should be connected in the projection. In our case, since we are interested in studying patterns of connections between users, we create a user projection network, where users are connected if they posted in the same thread. Formally, if we let \({\textbf{X}} = [x_{it}]\) be the bipartite adjacency matrix, with row labels corresponding to users and column labels corresponding to threads, then the user projection network adjacency matrix elements can be obtained as outlined by Borgatti and Everett (1997) aswhere \({\mathcal {T}}\) is the set of threads in the bipartite network and \(x_{it}\) represents the number of posts of user i in thread t. In matrix notation, this can be simplified as \({\textbf{A}} = [a_{ij}] = {\textbf{X}}{\textbf{X}}^T\). This transformation produces a weighted undirected network, but it is not representative of the reality in forums, as it works under the assumption that all users in a thread read each other’s posts. Inspired by Newman (2001), we weight Eq. 2 to take into account the number of users and number of posts in a thread, similar to collaboration networks where a connection between two researchers is weaker when they publish a paper with a lot of co-authors and stronger when they publish a paper together or with a small number of other co-authors. The strength of the connection is also proportional to the quantity of papers where the researchers are co-authors, with more papers indicating a stronger connection. In our case, two users have a weak connection if they both post in a thread with many posts and a strong one when they participate in multiple threads together. The formula to calculate the projection network adjacency matrix then becomes the following:where \(\upsilon _t\) is the number of users in thread t and \(\phi _t\) is the number of posts in thread t. We define a minimum weight for connections between users, below which we assume that the users had insignificant influence over each other. This threshold was chosen as the maximum value that keeps the network connected, i.e., with no isolated nodes.

The networks we create from Mumsnet data are a means of representing user posting behavior and how users interact with each other within Mumsnet. A change in network structure therefore indicates a change in user behavior, which in turn may reflect changes in how people are discussing or organizing around a topic.

To identify differences in the structure of Mumsnet networks, we use NetEmd (Wegner et al 2018) to compare each pair of networks. As these networks share the same generation mechanism, we expect NetEmd to output a high degree of similarity between them; deviations from this similarity indicate that there is a difference in structure. Upon calculating the comparison, we can detect differences in structure by looking at the distance between networks from consecutive data slices. If this distance is larger than the median distance between any pair of networks, then we consider that there is a significant difference in the structure of the network. Due to overlapping time frames, differences in the network structure may also take two or three time steps to become apparent.

As graphlets can be seen as small units within a network whose patterns of connections encode real-world functionality, we can interpret the network comparison results by determining which orbits are responsible for the largest difference and inspecting these.

Returning to the work of Skeppstedt et al (2017) and their approach to automatically labeling Mumsnet posts according to their stance on vaccination, we note that the authors highlight the difficulty of this task, particularly in comparison with finding stance in tweets, as posts are usually longer than tweets, given that they are not restricted to 280 characters, and therefore contain longer and more elaborate discussions. The approach taken by the authors yielded a machine learning model that struggles to learn how to classify such an elaborate corpus, achieving an \(F_1\) score of 0.44.

Starting from Skeppstedt et al (2017)’s approach, we attempted to improve the classification performance of a machine learning algorithm in the same labeled dataset, with the intent of ultimately applying it to the larger corpus we consider in our work. We found that using a random forest (Breiman 2001) improved the classification performance slightly (up to a \(F_1\) score of 0.52). We now briefly describe alternative approaches that yielded comparable or worse performance than adapting the code of Skeppstedt et al (2017) to work with random forests.

We used the sentiment analyzer of VADER (Valance Aware Dictionary and Sentiment Reasoner)  (Hutto and Gilbert 2014), a gold-standard sentiment lexicon combined with rules to determine sentiment intensity. Using VADER, we attempted to classify each post according to its sentiment in three variations: First we analyzed the sentiment of the whole post, then we split each post in sentences and calculated the average sentiment in each sentence, and finally, we split each post in sentences and calculated the average sentiment in sentences containing words related to vaccination (a regular expression matching jab, vacc*, immunis*, immuniz or vax). We found that splitting the text in sentences did not help in sentiment classification (0.27 and 0.25 \(F_1\) score, respectively, for not filtering and filtering sentences with vaccination keywords) and classifying the whole post yielded an \(F_1\) score of 0.40.

Following these experiments, we attempted to reproduce the naïve Bayes classifier of Salathé and Khandelwal (2011), using the implementation available in the python package NLTK (Bird et al 2009). We used the python package gensim  (Řehůřek and Sojka 2010) for preprocessing steps (removing stopwords and punctuation, lower casing, tokenizing). We performed tenfold cross-validation on an ensemble classifier with the same architecture as the one proposed by (Salathé and Khandelwal 2011): naïve Bayes classifier to separate positive and negative classes, maximum entropy classifier to separate neutral posts, with the maximum entropy classifier taking precedence over the naïve Bayes classifier. We report an average \(F_1\) score of 0.53 over the 10 folds.

Finally, we used an out-of-the-shelf FastText model (Joulin et al 2017) fine-tuned on the Crowdbreaks platform (Müller and Salathé 2019) for vaccine sentiment analysis. We adapted the preprocessing steps on (Müller and Salathé 2019) to our dataset and used the pre-trained model to classify our corpus, achieving an \(F_1\) score of 0.48.

The poor performance of automatic labeling of sentiment in the training dataset of Mumsnet posts disqualifies this methodology from being applied to the larger corpus of posts, as the labeling would be unreliable. Another reason to avoid using a machine learning model trained on Skeppstedt et al.’s data is a concept drift (Widmer and Kubat 1996). This captures the idea that models trained on one snapshot of time may struggle to generalize for data outside of that time period, particularly in language-based models trained on Internet data, where context changes over time. Müller and Salathé (2019) showed the existence of concept drift in vaccination discussions by analyzing a Twitter dataset in the second half of 2018 and at the beginning of the COVID-19 pandemic (Müller and Salathé 2020). The issue of concept drift is particularly relevant when comparing vaccination discussions before and after the COVID-19 pandemic. Changes in the vocabulary, arguments, context and the general discourse in these discussions lead to an increasing likelihood that a model trained on pre-pandemic data will not be able to accurately label a post-pandemic corpus.

The issues we identified with automatic labeling of Mumsnet posts led to the decision to manually label the two datasets we acquired from Mumsnet, to create a ground truth against which we can compare the methodology we develop. We label threads according to the title and the original post. We classified threads using three labels: positive, neutral or negative. We do not consider negative sentiment to be equivalent to an anti-vaccination position, as it is possible to express negativity toward a vaccine or a vaccination schedule while simultaneously adhering to the policy that recommends taking it. An example of this is starting a discussion about suffering from side effects of a vaccine, without asking for help on how to overcome them, expressing regret over taking a vaccine or doubting the efficacy of a particular vaccine. The following is a non-exhaustive list of common arguments we marked as negative toward vaccination:The neutral label captures two types of posts: those asking for information without expressing a judgment of value toward the vaccine itself; or those with conflicting positive and negative views about vaccination (for example, planning to vaccinate on schedule but mentioning being afraid or anxious). Finally, we marked threads as positive if they contained arguments that were supportive of vaccination, for example detailing its benefits, encouraging calls for more vaccines, explaining consequences of the disease that can be prevented through vaccination or mentioning someone going out of their way to get a vaccine, for example privately.

The manual labeling process was completed by two annotators (MEPS and RS), who split the annotation load equally, with each annotating half the threads of each dataset. The threads were assigned uniformly at random to each annotator. Prior to annotating, the two annotators labeled 1% of the COVID-19 dataset together (40 threads, chosen uniformly at random), showing high reliability with a Cohen’s kappa score of 0.84 (90% agreement). The annotators also discussed the results to settle discrepancies and agree on the mutual strategy for the remainder of the data.

In order to study the distribution of sentiment within threads of the COVID-19 dataset, we sampled 129 threads (about 3.0% of the number of threads) which contained 6898 posts (about 3.1% of the number of posts), chosen uniformly at random such that they represent at least 1% of the threads and posts in each data slice. The sentiment in all these 6898 posts was labeled by a single annotator (MEPS) using the same criteria used to label the threads.

An example that motivates this metric is the following: consider a thread with N posts, where the first N/2 posts are of negative sentiment and the last N/2 are positive. By considering only the proportion of positive to negative posts, this thread would appear to contain a large amount of disagreement between users as half the posts express a diametrically opposed view to the other. However, by inspecting how the sentiment is distributed over the thread, we actually find that most users are agreeing with each other, with the exception of the shifting point from negative to positive. On the other hand, if a thread with N posts has N/2 negative posts interleaved with N/2 positive posts, then it is much more likely that users are constantly disagreeing with each other throughout the thread. In both cases, the proportion of negative to positive posts is the same, but the way discussion is being carried out is very different.

Our proposed metric discordance index attempts to capture this distinction of local variety of sentiment within a thread. The way it works is by considering a moving window of D posts, such that in a thread with N posts there are \(N-D+1\) windows of D posts. For each window, we compare every unique pair of posts within that window (\(D(D-1)/2\) comparisons). If one post is positive and the other is negative, we assign a distance of 2; if a post is neutral and the other is positive or negative, we assign a distance of 1; and finally, if the posts are of the same sentiment, we assign a distance of 0. The discordance within a window is the sum of distances between each pair of posts; the discordance of a thread is the sum of discordances of all windows, divided by the maximum discordance for that window size. The discordance index of a thread is the average discordance across multiple window lengths, which we vary between 2 and 5.

We start by identifying the appropriate window overlaps to split each dataset over time. Although answering the research questions we posed in the introduction should not depend on these parameters, they are responsible for controlling how much change and variability are in each data slice. For instance, picking too long a window span may lead to a uniform sentiment throughout the whole dataset, whereas too small a window span could lead to so much variability between consecutive time windows that any difference becomes meaningless.

The first result we use to inform the decision is looking at how long each user spends on the forum and how long each thread remains active (Fig. 2). These figures show that users behave differently in the two subforums. In the vaccine dataset, we find that 52% of users post only once and close to 10% of threads get no replies. This contrasts with the COVID-19 dataset, where 37% of users have only one post and 4% of threads did not get any replies.

The results show that user active times—the period between a user’s first and last post—are shorter in the vaccine forum than the COVID-19 forum: 80% of users in the vaccine forum are active for less than a week; in the COVID-19 forum, if we exclude users with a single post, only 20% of users are active for less than a week, and 42% of users are active for longer than this. This may be because the pandemic was an extremely disruptive event simultaneously affecting people worldwide, whereas the issues brought up in the vaccine subforum are more likely to be short-lived or affect individuals. Despite this, results highlight that there is a core group of users in the vaccine forum with long-term engagement: 8% of users were active on this forum for longer than a year and close to 3% were active for more than 3 years.

Inspecting the thread active times in the COVID-19 forum highlights how fast information changes during a global pandemic, particularly in the modern age where information is easily disseminated and accessible. Results show that 42% of threads last less than 12 h (46% if taking into account posts without replies); close to 28% of threads last less than a week but longer than a day and only 11% of threads last longer than a week. This is likely because as new information about the vaccines was being released, older discussions quickly lost their relevance. This contrasts with how discussions were being held in the vaccine forum, with only 62% of threads lasting less than a week and 1.1% of discussions lasting longer than 3 years, compared with only 1.3% of COVID-19 discussions lasting longer than 3 months.

These two sets of figures lead us to conclude that we should favor a longer time span for each data slice from the vaccine dataset, because many users post for a short amount of time, but their influence can be long lasting due to a substantial number of discussions lasting multiple months. On the other hand, the time span of data slices from the COVID-19 dataset should be smaller, as threads tend to last a short time and users show signs of repeated engagement with the forum.

Based on these observations, we narrow down the candidates for window span and window jump and decide to use a combination thereof by inspecting how the number of posts, users and threads varies over time (several combinations are visualized in Figs. 11 and 12, for the vaccine and COVID-19 dataset, respectively). For the vaccine dataset, we set the time windows to have a 4 month span with 2 month time jump, leading to a 2 month overlap between consecutive data slices. For the COVID-19 dataset, we pick a 1 month span with a 2 week time jump, leading to a 2 week overlap between consecutive data slices.

Upon splitting the datasets and creating a network using the process described in Sect. 3.3 for each time window, we obtain two datasets of networks. The networks in each dataset correspond to users connected through their interactions in a medium that does not change and has been created using the same the methodology, making it reasonable to assume that the networks have a similar structure. If the networks are dissimilar according to the NetEmd network comparison method, this indicates that the way that users are interacting within the forum is also different.

In both datasets, we observe that the number of nodes and the average degree vary significantly over time. In the vaccine dataset, the number of nodes varies between 101 and 469 and the average degree between 4.6 and 19.7. In the COVID-19 dataset, the number of nodes fluctuates between 1873 and 5743 and the average degree between 25.7 and 84.8. Experiments by Silva et al (2023) indicate that the best performance on real-world datasets of undirected networks with this disparity in the number of nodes and average degree is achieved by using size 4 graphlets. As indicated by their experimental setup, we use NetEmd with PCA and 90% explained variance, although we find that, in practice, \(>90\%\) of the structure differences signaled by this parametrization overlap with the ones found by the original NetEmd.

The symmetric distance matrix returned by NetEmd containing the distances between each pair of networks can be visualized with a heat map. This visualization technique helps to highlight clusters of networks that are similar and quickly identify networks that differ substantially from their predecessors and successors. The heat map for the vaccine dataset is shown in Fig. 3 and for the COVID-19 dataset in Fig. 4. On the axis of each heat map, we include the start date of the time window each network is built from. For example, in the vaccine dataset, the first network, labeled as “07-2009,” has data spanning from 1 July 2009 to 31 October 2009; in the COVID-19 dataset, the first network, labeled as “11-2020,” has data from 1 November 2020 to 30 November 2020, the second network (not labeled) from 15 November 2020 to 15 December 2020, the third (labeled “12-2020”) from 1 December 2020 to 31 December 2020 and so on.

We combine three metrics to summarize sentiment in each time window:For each of these metrics, we measure how the proportion of positive to negative, negative to positive and neutral to non-neutral sentiments changes over time. For example, in a given time window, we calculate the number of positive threads divided by the number of negative threads, the number of negative threads divided by the number of positive threads and the number of neutral threads divided by the number of positive or negative threads. We calculate these proportions for number of threads, number of posts and number of users, measuring how they vary over time.

We have shown how user interactions with the Mumsnet forum can be represented through complex networks. By splitting these interactions according to their date, we are able to obtain snapshots of these interactions over a fixed period of time. We employed the technique of network comparison to observe differences between each snapshot of interactions, by measuring how the structure of the network obtained from these interactions changes over time.

We found that differences in network structure often occur simultaneously to real-world events that affect discussions in Mumsnet. For instance, the three major differences in the network structure of the COVID-19 forum coincide with the approval of the Pfizer and Astra Zeneca vaccines, the emergence of the Delta variant and the start of the vaccine booster program. Some of the differences in network structure we identify can be traced to changes in the distribution of specific orbits, which form a basis for the interpretation of the reasons for the differences in structure. An example of a change in the distributions of a subset of orbits that leads to significant differences in the structure of the whole network are the distributions of cliques in the vaccination subforum; these orbits are the biggest contributors for the separation of the networks in this dataset into two broad clusters, before and after May 2015.

By performing sentiment labeling and analysis on these datasets, we were also able to uncover a link between differences in the network structure and changes in the sentiment toward vaccination in the forum. Out of 10 groups of network changes, only in one do we not find any connection to change in sentiment. Conversely, there are only two changes in sentiment that are not reflected in differences of network structure, both of which occur in the vaccination dataset.

Through a sample of the COVID-19 dataset, we studied whether differences in network structure were a better indicator of the distribution of sentiment within the posts of a thread. The distribution of sentiment in a thread can be seen as a proxy for the amount of disagreement within that discussion; although posts of the same sentiment do not necessarily indicate agreement, posts of different sentiment reflect opposing views. By using this metric of disagreement, results show that the time with most disagreement in our sample coincides with the second largest difference in network structure.

Considering Mumsnet as a whole, our analysis shows clear distinctions between how users discuss vaccination in general and how they discuss it within the context of COVID-19. This distinction starts in the way that discussions progress in each thread, with the COVID-19 forum characterized by short-lived threads with a high volume of replies. This contrasts with more drawn out threads, some that even last multiple years, in the vaccination forum. There are also big differences between the two forums in terms of the sentiment toward vaccination. Because the data for the vaccination forum spans a much longer period of time, it is natural that it contains greater shifts in sentiment over time. However, there is a much higher prevalence of neutral sentiment in the COVID-19 forum, as neutrality is the dominant sentiment in all time windows; the same does not happen in the vaccination forum, where there are time periods dominated by negative sentiment. Another key difference between the two forums is the number of users posting in threads of different sentiments, which can be seen as a proxy for the amount of information flowing between threads of different types. The proportion of users posting in threads of all sentiments is significantly higher in the COVID-19 forum than in the vaccination forum, which can be interpreted as an indication that there is a wider variety of opinions in each thread of the COVID-19 forum.

Underlying all these distinctions between the two forums, we find significant differences between the network structure of their user interactions. Although the contrast in network structure is to be expected for the reasons above, we observe unexpected similarities in the distributions of certain orbits. These similarities highlight common patterns of user interactions in Mumsnet as a whole and further accentuate the difference between the networks before and after May 2015, in the vaccination forum.

Sentiment labeling is a very laborious process that consumes a great deal of human resources and does not scale to large datasets, as an increase in the quantity of data requires a corresponding increase in the effort of labeling. Automatic methods have been developed, but the vast majority focus on Twitter data and are unsuitable for more nuanced arguments that do not fit the character limit of a tweet. Furthermore, automatic methods require representative samples for training and pre-labeled data for training is often inappropriate due to changes in the context in which the discussions took place.

On the other hand, the network comparison methodology we use to analyze user behavior in vaccination discussions is an easy to use out-of-the-box tool that scales to large social networks. Through network comparison, we were able to identify periods of time with differences in user behavior that were associated with changes in sentiment toward vaccination. In other words, our results indicate that shifts in opinion are also reflected in changes to the network structure. Our methodology naturally does not replace sentiment analysis, but instead it can be useful for researchers as a tool to identify shifts in sentiment that can help to target further analysis.

Another strength of our approach of network comparison is that we are able to inspect which graphlets and orbits are responsible for the differences in network structure. This knowledge can be used to compliment sentiment analysis to generate hypothesis about differences in user behavior.

We now provide an example of this ability to generate hypotheses from differences in network structure identified by the network comparison methods. In Sect. 4.2.2, we highlighted that the difference in the network starting on 1 October 2021 was being primarily driven by changes in the distributions of cliques, either of size 3 or 4. This change does not stem from an over or under representation of these subgraphs, as motif analysis indicates that the subgraph ratio profile (Milo et al 2004) is very similar for these two networks (0.61 in 15 September 2021 and 0.59 in 1 October 2021). Instead, in October 2021 we observe that more users participate in a greater number of cliques, compared to the previous time slice where we see a distribution of participation in cliques more similar to a heavy-tailed distribution, as expected (see Fig. 17 for a visual representation of this difference). Using this information, we can hypothesize that the network from 1 October 2021 is composed of a large set of users densely connected and this large set of users is sparsely connected to the remaining users, who form their own small clusters.

Building on this hypothesis, we can hypothesize about user behavior that led to this network topology. Recall from Sect. 3.3 that this network of interest is a sparsified one-mode projection on the user partition from a user-thread bipartite network. Therefore, to reach this network topology, the bipartite network needs to have a small number of nodes from the thread partition connected to a large set of nodes from the user partition. From a social network perspective, this is equivalent to saying that there is a small number of threads where the majority of the vaccination discourse was concentrated. This can indicate a sort of preferential attachment (Barabási and Albert 1999) toward popular threads, with mechanics resembling a Chinese restaurant process (Aldous et al 1985).

Although we are able to judge which graphlets are responsible for the changes in network structure, we were unable to determine whether there is a correspondence between changes in certain graphlets and the prevalence of a specific sentiment. We tried alternative techniques, such as motif analysis through the comparison against a null model or even simply inspecting the frequency of graphlets, but changes in network structure are often quite subtle; it is not the prevalence of each subgraph that leads to the differences in network structure, but rather how they are distributed over the network. This is what makes complex networks complex; characterizing their structure is often hard so it is important to develop heuristics that can help identify when the structure is different compared to other networks that are hypothesized to be similar. Nevertheless, the relationship between network structure and sentiment is still an open problem and we leave the application of other techniques to uncover this relationship for future work.

One of the biggest limitations of our analysis of the COVID-19 forum is the wide criteria we used to include threads in our dataset, as all threads that included any keyword related to vaccination were considered in the analysis. This lead to the inclusion of threads where the discussion was not focused on the vaccines themselves, but instead related to the pandemic in general. This wide criteria was particularly evident when labeling the sampled posts, where we identified discussions about lockdowns and opening up the country after the vaccination campaign, about Brexit in the context of Astra Zeneca vaccine distribution and other pandemic-specific topics like modeling, masking and testing. These discussions were labeled as neutral toward vaccination, but there was a large body of arguments in regard to each of these issues, with polarizing opinions between users.

It was unclear how to deal with this issue. Even if on a thread level it was possible to detect threads whose discussion was not aimed at the vaccines in particular when we labeled each original post, we found that even in threads that started off as discussions about the vaccine, the natural flow of discussion trended toward other issues. This was especially common in threads with a large number of posts, where off-topic concerns kept being brought up, usually as a reflection of the pandemic situation at the time.

These side discussions present a confounding factor in connecting the structure of the networks with the sentiment of the forum. The structure of the network carries information about how the discussions are being carried out, regardless of their content, whereas all the side discussions were grouped under the umbrella sentiment of neutrality. This affects all facets of our analysis but in particular the discordance index, as we observed long sequences of neutral posts that hid a high level of disagreement.

Another issue that affects both the COVID-19 and vaccination forums is moderation by Mumsnet staff in an effort to remove blatant misinformation. This is reflected in threads being deleted or posts whose content was deleted, as well as replies that quoted those deleted posts. Whereas the former only affects the overall perception of how much negative sentiment is in the forum, as naturally only threads negative toward vaccination were deleted, the deleted posts are still encoded in the network structure and were marked as neutral sentiment. The reason for marking deleted posts as neutral instead of negative is that they do not necessarily express anti-vaccination views borne out of misinformation, but instead often they are unfruitful discussions between users that involve personal attacks and thus violate the rules of the forum.