Characterizing the Twitter network of prominent politicians and SPLC-defined hate groups in the 2016 US presidential election

Twitter, a popular micro-blogging service, provides an observable social network with millions of users (Jansen et al. 2009). Twitter allows users to communicate or update their status in many ways. One can post a message called a tweet that is no more than 140 characters in length (in 2016), follow another Twitter user, and receive the status updates of every user they follow. A tweet that is shared publicly with one’s own followers is known as a retweet.

Our dataset comprises of 21,749,868 communication events in Twitter over a total of 9 weeks centered around the US 2016 election. The tweet IDs are available from https://​tinyurl.​com/​y8lamxzx. The dataset was collected in our designed experiment using Twitter’s streaming and REST APIs (Developers 2018) that were extended for Apache Spark (Zaharia et al. 2016). Each of these communication events was parsed and classified into seven types of events using SparkSQL modules.

Streaming data collector Over 17 million events were collected from Twitter’s public streams by directly tracking communications related to the Twitter accounts of the five political candidates and 52 hate group accounts. Because only 78% of the SPLC identified hate groups had a valid Twitter account that we could track, our study is limited to a further subset of 52 user accounts who were active in the Twitter public streams. Thus, our approach is not exhaustive, in terms of being able to track every account of each hateful ideology, but is nonetheless representative of the public Twitter activity.³

Retrospective data augmentation The remaining 5 million events were obtained retrospectively using Twitter’s REST API as follows. Due to most retweets being an immediate reaction to a tweet that one finds interesting or concurs with Kwak et al. (2010), nearly 7 million of the 10.5 million retweets in the 17 million events collected from the public streams happened within the same day of the original tweet and over 98% of the retweets happened within a week of the original tweet. Furthermore, since our tweet collector is only recording events that are directly related to one of our tracked users, we do not know of any other Twitter interactions by those who retweeted one of our tracked users. Thus, to better understand the recent retweet behavior of at least some of the politically active Twitter users with other non-tracked users on Twitter, we focus on October 19 2016, the day of the 3rd US Presidential debate, and obtain a seed set of users who retweeted either @HillaryClinton or @realDonaldTrump on this day. The communication intensity reached over 120 events per second around the debate in our streaming data collector. Our seed set is made up of a random sample of about a third of all users (including all verified and geo-enabled accounts) who retweeted either Clinton or Trump on October 19 2016 and thus constitutes an evenly represented sample of politically active Twitter users from the two parties. For each user in the seed set, at the end of the 9-week period, we added all the retweets from their 200 most recent status updates that occurred in the 9-week period. This strategy involves a breadth-first expansion about the seed set of users in the much larger retweet network on Twitter as it allows us to expand our 9-week-long retweet network by focusing on the recent retweet timelines of those who retweeted either of the two final presidential candidates during the day of the last debate. Crucially, these augmented data added another 0.3 million users to our network, increased the number of retweet events from 10.5 million to 13.7 million, and made the retweet network into a single connected component.

Retweets are a simple and natural signal of directional concurrence, i.e., concurrence with the user who posted the (original) tweet by the user who retweeted it (i.e., between the tweeter and the retweeter), as they express interest in the message and also endorsement and trust in the communicator (Jansen et al. 2009; Metaxas et al. 2015), especially when retweeted multiple times. Retweet analysis via random network models as done in this study circumvents the ambiguity and uncertainty associated with statistical algorithms (Barberá et al. 2015) in natural language understanding (NLU) that will be required when working with (a) quoted tweets where one is allowed to add a comment to the retweet, (b) reply tweets or (c) reply of quoted tweets where one can reply in possible disagreement.

The number of retweets per day captured from the public stream during the 9-week period reached over 1 million during the day of the third US Presidential debate and the days leading up to the election. The proportion of retweets is known to vary greatly depending on the features of the Twitter subnetwork under study (Kwak et al. 2010). There were over 13.7 million retweets (63.1%) of an original tweet and over 2.7 million (12.6%) original tweets in our dataset. We ignore the remaining 25% of the events that require further NLU and focus our analysis instead on the retweet network obtained from 75% of all events in our dataset as explained in the next section.

We allowed the set of users who tweet and retweet, i.e., tweeters and retweeters, to form the nodes of the retweet network. Each retweet was allowed to represent a directed edge or arc from its tweeter to its retweeter, i.e., the tweeter–retweeter pair, as a signal of directional concurrence in the retweet network. Because one can retweet more than one tweet posted by any user including oneself, we allowed for parallel or multiple edges between the same pair of users including self-loops, i.e., edges from and to the same user. There were more than 4.4 million unique tweeter–retweeter pairs out of 2.5 million unique users representing over 16.4 million tweets and retweets in our dataset with over 21.7 million communication events during the 9-week period around the US presidential election. Mathematically, the retweets in our dataset are represented by a directed multi-edged self-looped network. Thus, the out-degree (number of outgoing edges) and in-degree (number of incoming edges) for a user in the retweet network gives the number of times that the user is retweeted by others and the number of times the user retweets others (including oneself), respectively. Similarly, the number of distinct users who are retweeted by the user is given by their in-nbhd and the number of distinct retweeters of the user is given by their out-nbhd.

The observed retweet network is highly heterogeneous and largely dominated by the two presidential candidates as depicted in Table 1. Due to our Twitter collector’s design with retrospective data augmentation, our network forms a single connected component when viewed with undirected edges.

The configuration model for directed networks (Newman et al. 2001) is a random network model that produces samples uniformly from the set of networks that preserve the in-degree and out-degree of each node in given observed network.

CutPermuteAndRewire is a Monte Carlo algorithm to generate independently sampled networks from the directed multi-edged self-looped random configuration model, i.e., our null random network model for apathetic retweeting. CutPermuteAndRewire is a distributed, scalable, and fault-tolerant version of the standard construction involving random pairings of out-bound and in-bound half-edges (Newman et al. 2001) through the following three steps: (i) cutting the directed edges representing the retweets in our observed retweet network into out-bound and in-bound half-edges, (ii) permuting the in-bound half-edges by sorting them according to pseudo-random numbers that are generated and associated with them, and (iii) rewiring the original out-bound half-edges with the permuted in-bound half-edges using a distributed join. Note that the in-degree and out-degree of each node in the observed retweet network is preserved after the three steps by construction.

By taking advantage of the fastest available distributed sorting and optimized distributed joining algorithms (Zaharia et al. 2016), CutPermuteAndRewire can produce independent sample networks with tens of millions of retweets or edges in a small Apache Spark cluster over six commodity compute nodes. The null distribution of any test statistic of the retweet network can be directly obtained from applying it to each independent Monte Carlo sample from our scalable fault-tolerant randomized algorithm with probability given by:under the null model. By comparing the observed test statistic to Monte Carlo samples from the null distribution, one can directly obtain consistent estimates of the p value in order to attempt to reject the null hypothesis of apathetic retweeting in favor of the alternative hypothesis of non-apathetic retweeting in the framework of generalized Fisher’s exact test.

To gain deeper insights into the empirical retweet network beyond the immediate retweet neighborhood of each user, i.e., those retweeted by a user and those who retweet the user, and explore the alternative hypothesis space if the null hypothesis is rejected, we adapt Milgram’s concept of six degrees of separation (Milgram 1967) that all people in the world are six or fewer steps away from each other so that a sequence of “a friend of a friend” relationships can be made to connect any two people in a maximum of six steps. We adapt the concept in three major ways.

First, we focus on Twitter users and replace the mutual or undirected relationship of being a friend by the directed relationship of being a retweeter. This adaptation accounts for the main difference between various social and technological networks (Watts and Strogatz 1998) as well as other communication networks (Leskovec and Horvitz 2008) that are characterized by mutually reciprocal relationships, and the directed relationships in Twitter where a path from a user to another may follow several distinct sequences while not existing in the reverse direction (Kwak et al. 2010).

Third, we account for the strength of the retweet relationship when defining the most retweeted path by incorporating \(r_{a,b}\), the observed number of retweets between the user who is the source of the original tweets, i.e., the tweeter a, and the user who retweets them, i.e., the retweeter b, through the directed edge-weight given by \({\hat{p}}_{a,b} = 1/(1+r_{a,b})\) that is used to specify the probability of an independent geometric random variable giving the number of retweets of user a by user b.

Let G be such a weighted directed retweet network with nodes as users and directed edges with weights:The collection of independent but non-identical geometric random variables with probability parameters given by these weights is our empirically estimated geometric retweet network model for the joint distribution of the number of retweets of each user by another in out dataset.

We derive our estimated geometric retweet network model by recalling the well-known relationship between Poisson, exponential, and geometric random variables. If the random variable \(R_{a,b}\) giving the total number of retweets of a by b is Poisson distributed with a random mean parameter \(\xi _{a,b}\) that is drawn from the exponentially distributed random variable with rate parameter \(1/\lambda _{a,b}\), then \(R_{a,b}\) is geometrically distributed with probability parameter \(p_{a,b}=1/(1+\lambda _{a,b})\) and expectation \(\lambda _{a,b}\). We can estimate the parameters from the observed number of retweets \(r_{a,b}\) via the moment estimate \({\hat{\lambda }}_{a,b} = r_{a,b}\) and model the number of retweets during the 9-week period according to the geometric random variable with probability parameter \({\hat{p}}_{a,b} = (1+r_{a,b})^{-1}\) for each directed edge between a tweeter and a retweeter in the retweet network.

A small weight \({\hat{p}}_{a,b}=(1+r_{a,b})^{-1}\) corresponds in an inversely proportionate manner to a large number of retweets, and the shortest path from user a to user b through this directed weighted network corresponds to the path with a large number of retweets from user a to user b. Thus, the estimated geometric model interpretation of the weighted retweet network G allows us to use a straightforward distributed vertex program via Pregel API in Apache Spark’s GraphX library to obtain the shortest path that is composed of a sequence of d tweeter–retweeter pairs of edges, say,with the lowest sum of weights given by \(\sum _{(u,v) \in E}{\hat{p}}_{u,v}\) among all possible paths between an influential user of interest a and every other user b in G. Crucially, this \({\hat{p}}\)-weighted shortest path is called the most retweeted path as it is composed of the same sequence of edges with the correspondingly high sum of retweet counts given by \(\sum _{(u,v) \in E}{r}_{u,v}\). The length d of the most retweeted path from a to b, known as the (retweet) degrees of separation from a to b, has a clear interpretation as the length of the sequence of “retweeter of a retweeter” statements along the most retweeted path that is needed to link user a to user b by considering the retweet activities of every user in the network. When we have a set of influential users \(A=\{a_1,a_2,\ldots ,a_k\}\) of interest, say k accounts of a hateful ideology, we define the shortest path from A to any user b as the minimum of the k shortest paths from each user in A to b.