Developing insights from the collective voice of target users in Twitter

We present the details of how to discover a target audience of Twitter users and their collective voice from raw Twitter data. First, in order to identify candidate users that meet certain criteria, we explore available Twitter resources for data collection and existing approaches to user profiling. Next, we discuss enriching user profiles utilizing hashtags in the tweets posted by the target users. Lastly, we present developing topical and social insights from the collective voice of the target users.

Before we go into details, we first present formal modeling of the data space that we analyze in this paper. Our Twitter data space can be noted as \({\mathcal {U}} \times {\mathcal {T}} \times {\mathcal {H}}\), where \({\mathcal {U}}\) is a set of users on Twitter, \({\mathcal {T}}\) is a set of tweets created by the users, and \({\mathcal {H}}\) is a set of hashtags used in the tweets by the users. This implies that a user \(u \in {\mathcal {U}}\) creates a tweet \(t \in {\mathcal {T}}\) using a set of hashtags \({\mathcal {H}}_{u,t} \subset {\mathcal {H}}\).

User profiling is an essential component to our approach, which defines user attributes needed for a study and populates the attribute values for each user. We define the profile of a Twitter user \(u \in U\) as a set of tuples consisting of an attribute and its value where, with respect to user u for an attribute \(a \in A\), its value p(u, a) is computed by a user profiling function p, as in Eq. (1):where A is a set of user attributes. Determining the user profiling function p for each user attribute is the goal of the user profiling phase.

Figure 1 illustrates the flow of our unified scheme for developing social insights from the collective voice of target users. First, attributes of Twitter users are identified in the user profiling stage such as demographic attributes and other personal attributes. When some user attributes are missing due to data availability, researchers can consider developing their own customized solution to a specific user profiling task. A supervised machine learning model can be built by utilizing hashtags as the features for prediction. Second, once this user profiling phase is completed, researchers select only the users of interest based on the identified user attributes. Finally, researchers proceed to develop topical and social insights from the collective voice of these target users.

In general, sampling of Twitter users is less common than sampling of tweets due to the limited functionality of Twitter API for collecting users. For this reason, we begin with a large pool of random tweets, which are known to be much easier to collect via Twitter API mentioned earlier in "Introduction" section. Each tweet collected contains author information describing the user who created the tweet. Some user attributes for the users in the pool are already known or can be easily acquired, while other attributes need to be inferred, are difficult, or impossible to identify. It is worth noting that raw user data collected from Twitter via Twitter API provides surprisingly useful information about users. Table 2 lists native Twitter objects and their fields along with user attributes that can be derived from the fields. Twitter API provides several types of objects encoded in JavaScript Object Notation (JSON), of which User and Tweet objects are the most useful in user profiling.⁴

A User object, which describes an individual user on Twitter, has several fields that can be directly used as user attributes, such as name, location, and url, while the other fields can be analyzed to infer new attributes.⁵ For example, from the description field that has a user-defined description or bio of an account, one can infer many different types of user attributes, such as demographic attributes (e.g., age, education, gender, location, marital status, language, occupation, and race/ethnicity) and other personal attributes (e.g., expertise, hobbies, interests, personality traits, and political orientation), depending on the information included in the text of the field. A wide range of natural language processing (NLP) and text mining techniques can be applied to this field. The other fields in a User object can be good indicators of the account’s popularity, sociability, or activeness. For example, the followers_count and the listed_count fields indicate how popular the account is, while the friends_count field indicates how sociable the account is. One may want to compare the followers_count to the friends_count, to see if there is a large or small gap between the two fields. For example, celebrities tend to have a very large number of followers but a smaller number of friends, whereas spam accounts or bot accounts tend to have many friends but few followers.

The favourites_count and the statuses_count fields can be used to measure how active the account is in terms of posting tweets. The created_at field can be used to calculate the account age in days, months, or years, which can be combined with other fields for normalization. For example, users who have been using Twitter for ten years would probably have more followers or have posted more tweets than those who just began to use Twitter. In this case, one may need to divide the number of followers or number of statuses by the account age, so that the indicators can be normalized for each user.

A profile image from the profile_image_url_https field can be used to identify gender, age, or race/ethnicity of the user by applying state-of-the-art image analysis techniques [13, 61]. The followers field contains the lists of users following the account, while the friends field contains the list of users the account is following, both of which present the relationship network of the user. Note that the two fields, each marked with an asterisk, are not actually linked to the User object as its fields. Twitter API separates these two fields from the User object for some reason. But we link them as fields of the User object, as we believe those fields should also be treated as user attributes.⁶ The two fields provide direct information about who are the followers and friends of a user. The verified field is a unique feature of Twitter, which indicates whether Twitter has verified that the account of public interest is authentic.⁷ A verified account has a blue verified badge on Twitter. This can serve as another indicator of the user’s popularity or authority.

A Tweet object describes an individual tweet posted by a user.⁸ An individual tweet could not be directly used as an attribute of a user due to its limited information. When aggregated, however, they can be a powerful source for a researcher to understand the user. While a Tweet object has a number of fields, the bottom half of Table 2 lists a few of those that can be used to infer user attributes. The text field is the most important one among all fields, as it provides raw tweet text written by the user. It is worth noting that tweet text can have up to only 280 characters (the length limit was increased from 140 to 280 in 2017), which is why Twitter is called a micro-blogging service. The short text has its own pros and cons. In some cases, tweet text might be too short to convey meaningful information from an analysis perspective, while in other cases a single short tweet can have enough information to understand the user. On the other hand, the short text is what has made people freely use Twitter. From a Big Data perspective, the more tweet text we have for a user, the better understanding of the user we will have. The text field can be used to infer most of the demographic attributes and personal attributes mentioned earlier. As with the description field of a User object, this field can benefit from text analysis techniques.

The created_at, coordinates, and place fields can bring a temporal or a geo-spatial aspect to the study. While every tweet has a value in its created_at field, not all tweets have values in the coordinates and place fields. It depends on whether the user had activated location sharing in their applications. It is known that, as already discussed earlier, only a small fraction of tweets are geo-tagged or geo-referenced [20]. The three fields reply_count, retweet_count, favorite_count are considered to be good indicators for the popularity of the tweet, which can also translate into the popularity of the user. The lang field indicates which language the user is primarily using or able to use. It is also worth noting that users can retweet other users’ tweets, and those retweets are considered to be the user’s tweets, although they were originally created by others (users can also add their own comments to the original tweet when retweeting). If we analyze tweets to understand the user, however, those retweets could be of no help, because they were not originally created by the user. In this case, by referring to the retweeted_status field, those retweets can be excluded from any analysis, so that only the normal tweets created by the user are considered.

The Twitter objects and their associated fields listed in Table 2 provide insight into some heuristics for user profiling before attempting to apply advanced methodologies. In particular, the description field of a User object can be directly used to extract various user attributes like gender, location, occupation, and so on. The following description from a Twitter user account, which is open to the public, is a good example:

Senior Narrative Designer @UbiMassive — cats, books, games and scones — Brit in Sweden — opinions all mine — She/her.

This short bio tells much about the user, such as gender, occupation, hobby, nationality, and location. The user is female from the phrase “She/her”; she is a narrative designer at a game company; she likes cats, books, games, and scones; she is British; she lives in Sweden. While not all Twitter users describe themselves in such detail, it is apparent that the description field can serve as a primary source for understanding users. In order to extract the right information from the description text, a string pattern matching technique called regular expression can be employed.

If the approaches relying on some raw user attributes provided by Twitter are too simple to work for a research study, one should consider employing advanced techniques for user profiling listed in Table 1. As described in "Related literature" section, previous works have explored different ways of profiling Twitter users. When applying the advanced methodologies, note again that different methodologies use different data for user profiling, depending on their proposed approaches. For example, to identify the location of a user, some methodologies such as [11, 21‐23] consider only tweet text, whereas other methodologies such as [12, 27‐29] use not only tweet text but also use follow relationship of users or tweet context. Note also that the methodologies targeted at the same user attribute do not always yield exactly the same outcome, as each methodology has its own research questions to address. Depending on objectives of the study, a subset of the user attributes listed in Table 2 can be considered in user profiling. For the market research project example mentioned in "Introduction" section, the researchers should only focus on such attributes of users as age, gender, and interest, and thus examine which methodologies would fit the data they currently have. Again, they should be aware that different methodologies use different data. Once this user profiling task is performed over all users in the data pool, they now can select only the users that meet the criteria they have set for the study. This initial set of selected users can be further analyzed to be selected as the final set of target users.

If the user profiling task was perfectly done and ended up properly populating all user attributes needed, we can move on to selection of target users based on the user attributes. In many cases, however, it is possible that there are no resources available for some user attributes, leaving their values missing. This can happen when (1) there are no available resources at all, (2) the existing resources do not fit the data we have, or (3) the performance of the available resource is not satisfactory.

To resolve this issue, we propose to consider developing a customized solution to a specific user profiling task, especially if it is a supervised machine learning problem. For example, suppose we want to classify each Twitter user by their political orientation, i.e., conservative or liberal. While there are some available resources for political orientation classification, as listed in Table 1, one might find that those existing resources do not work well with the recent Twitter data. This leads us to consider developing our own political orientation classifier as long as we can make labeled data that can be used for training and testing machine learning models. Inspired by the observation that some Twitter users explicitly share their political orientation in their bio, we can collect a set of those users and label them as conservative or liberal. We then can use the labeled data as training data and test data for machine learning by selecting a set of features for prediction. Specifically, we propose to utilize hashtags as the features for political orientation prediction, based on the idea that conservatives and liberals are believed to be interested in different topics to some extent, thereby using somewhat different hashtags. Once a machine learning model is built, one can apply the model to populate the values in the target user attribute. While we cannot say that this approach would work for all user profiling tasks, we believe that it can work for supervised machine learning tasks, such as classification and regression, and that it can be a good complement to the existing user profiling solutions. We call this phase text-based customized user profiling, as opposed to the primary user profiling performed in the first phase, as this customized user profiling task can complement what is missing from the primary user profiling task.

In order to utilize hashtags as features for prediction, we first need to collect the tweets posted by users and mine hashtags from the tweets. The Twitter API allows researchers to retrieve up to 3200 most recent tweets of a user account, as long as the account is set to public.⁹ Alternatively, one can consider web scraping to retrieve more than 3200 tweets from an account, although this option does not provide easy access to the web data in a structured manner unlike using an API. While all words in tweets are meaningful in one way or another, we particularly focus on hashtags in tweets. A hashtag is a word starting with a hash (#) symbol as its prefix such as #metoo, #nowplaying, and #earthday. Hashtags were originally introduced by Twitter and have been used to index keywords or topics on social media, which allow users to easily follow topics of interest. As mentioned by [62], the goal of a hashtag is to facilitate search and aggregation of messages related to the same topic. With the wide adoption of hashtags on Twitter, a number of studies have investigated hashtags on Twitter. Tsur et al. [63] attempt to predict the spread of thoughts and ideas, called memes, using hashtags. Ferragina et al. [64] address hashtag relatedness and classification. Refs. [65‐69] address hashtag recommendation from a personalization perspective, while [70‐74] address hashtag clustering.

One of the reasons why we focus on hashtags, instead of all words or phrases in tweets, is that they are easy to handle. As users explicitly create a hashtag with the hash symbol and a hashtag allows no space in it, they are easy to extract and aggregate from text. In fact, Twitter API provides a list of hashtags identified in a tweet as a Hashtag object, thus API users do not have to extract hashtags themselves, which otherwise should be done with the help of a text analysis technique like regular expression. The main drawback to using hashtags is its sparsity; as pointed out by Godin et al. [66], not all tweets have hashtags and not all users use hashtags. Nevertheless, this sparsity can be overcome when a large number of hashtags are aggregated, mainly because of the fact that a hashtag tends to be adopted by a significant number of users who want to join a virtual community that is interested in a certain topic [75].

Once all hashtags are extracted from tweets, they are aggregated such that the total frequency for each hashtag is calculated. Based on the hashtag frequency, one can have a hashtag popularity ranking sorted by frequency in descending order. This hashtag ranking can be a basis for researchers to manually select top-k popular hashtags that will be used as features for prediction, where k can be determined empirically. When top-k hashtags are selected as features, their frequencies are the values that should be put into the machine learning model. This way, one can build a model that is able to predict the value of a user attribute for a user. Building a machine learning model should always be followed by evaluating the model performance, using commonly used machine learning metrics.

Once the user profiling is completed and all values of the user attributes needed for the study are properly populated, one can now select the target users of interest, using the user attributes. For the market research example mentioned earlier, the researchers can simply select the users in their pool, who are young, female, and interested in fashion. Given that the target users have been identified, researchers can now proceed with in-depth analysis on the collective voice of these targeted users. While this final phase should completely depend on the objectives of the study, i.e., what the researchers want to know about their target audience, we focus on hashtags from a topical perspective to discover popular or rising topics among people and also on relationship networks from a social perspective to identify influencers.

Popular hashtags among the target users can be captured in a similar way that we used earlier to identify popular hashtags for the customized user profiling. A simple frequency ranking from tweets will work for popular hashtags, while one may want to consider advanced techniques to detect a trend over time with hashtags [76‐78]. Influencers in a social network can be identified as well, based on the network structure among the target users. A variety of centrality measures, such as degree centrality, closeness centrality, betweenness centrality, and eigenvector centrality, can be applied, as previously mentioned in "Related literature" section.

Marketers want to know what their customers are currently interested in and who are the influencers among them, so that they can have insights into new business opportunities and focus their marketing effort and resources on specific people who could influence others. In this in-depth case study, we aim to first identify young, female users in Twitter who are interested in fashion and then discover popular topics and influential users among them.

In order to find the target audience of female users interested in fashion, we first begin by searching our random pool of tweets for tweets that have the hashtags #fashion and #style. As mentioned earlier, each Tweet object has a User object that indicates the user who created the tweet, which allows us to identify all users in our pool who have ever used the two hashtags. Here, mentioning the hashtags is assumed to be their interest in the topic. This step can be understood as a simplified implementation of the interests attribute in Table 1. Note that, one can consider adding more hashtags as search terms that are similar to #fashion and #style such as #beauty and #clothing. The search allows us to find 111,913 users in total. Using the Twitter API, we further check if each of these users still has a valid, public account, which leaves 89,437 users.¹⁰ Next, we remove users whose total number of tweets posted is fewer than 100, based on the idea that we would need at least 100 tweets to understand a user by their tweets. This results in 51,276 users in total, i.e., \(|U| = 51276\). We then collect up to 3200 most recent tweets from each user using the Twitter API, which totals 107,002,581 tweets, i.e., \(|T| = 107002581\).

After finding users interested in fashion and collecting their recent tweets, the next step is to identify each user’s gender and age, which will allow us to select young female users. Before applying a gender classification solution, we first remove organization accounts, based on the belief that organizations do not represent our target customers. Note that researchers may want to include organization accounts if they believe organizations are worth being considered in their study. In this case study, we are only interested in individuals, especially young female users. This step can be considered as an implementation of the account type attribute in Table 1. In order to identify organization accounts, we leverage two open source solutions: one is called Humanizr provided by [33] and the other called M3-Inference provided by [13].^11,12 The Humanizr looks into tweets of a user along with user information in the tweets to determine whether the account in question belongs to an individual person or represents an organization, while M3-Inference uses the profile image, name, screen name, and the bio of a user, as already stated in "Related literature" section. In case the two solutions return different outcomes for the same account, in other words, one solution classifies as an organization account, whereas the other does as an individual account, we consider a user to be an organization account when at least one of the two says it is an organization. Otherwise, the account is considered an individual account. In our data, approximately 22% (11,195 out of 51,276) of the accounts turn out to be organization accounts, which is higher than 9.4% reported by [33]. We remove those organization accounts, which leaves 40,081 users who are believed to be individual accounts.

For gender identification, we utilize a Python library called gender-guesser, which employs a statistical approach to gender classification by considering the first name of a person, as well as the M3-Inference solution already used for the account type.¹³ The gender-guesser solution returns one of the six classes: “unknown”, “androgynous”, “male”, “female”, “mostly_male”, or “mostly_female”. Here, we merge “mostly_male” into “male” and “mostly_female” into “female”, for simplicity. As mentioned in "User profiling" section, a User object has the name field that allows users to specify their name. As not all users provide their exact full name, it is possible that there is no first name in the field. Furthermore, even if there is the first name specified by the user, there is no guarantee that the first name is recognized by the solution, which is especially true for non-English names. The M3-Inference solution returns either “female” or “male” for a user. In order to merge the outcomes from the two solutions, we (1) label the users as “conflict“ when one solution returns “female” and the other “male” and (2) label the user as the one predicted by the second solution when the first solution returns “unknown” or “androgynous” and the second solution returns “male” or “female”. This results in 24,886 females, 13,910 males, and 1285 conflicts. We disregard the conflicts in our data.

For the age attribute, we continue to rely on the M3-Inference solution, which returns for each user one of the four age levels: ≤18, (18, 30), [30, 40), [40, 99). From our data, the solution results in 6,011 users for 18 or under, 12,994 for 19 to 29, 10,641 for 30 to 39, and 10,435 for 40 or above.

Now that we know all four user attributes needed for this study, i.e., interest, account type, gender, and age, we can select, from the users interested in fashion, those who are young and female. For the age attribute specifically, we define young women as those in the following two age classes: (18, 30) and [30, 40). This entire selection process of target users results in 16,011 users, who form the final target audience for this study, and 31,506,037 tweets posted by the users, which will be further analyzed. As a reminder, we identify these 16,011 young females out of all 51,276 users.

We now proceed with the last step for gaining insights into popular topics and influential users among the young women interested in fashion. To discover popular topics, we look at popular hashtags used by them in their tweets. When extracting hashtags from tweets, we exclude those hashtags that are exclusively used by a single user. Specifically, a hashtag is excluded if its frequency rate from the most contributing user is higher than or equal to 0.5. We also exclude non-English hashtags. Table 4 presents the top-50 popular hashtag ranking. All the hashtags on this ranking provide us with direct or indirect insights into young female users’ interests in the fashion domain. For example, the first-, second-, and seventh-ranked hashtags #poshmark, #shopmycloset, and #etsy clearly show how popular shopping on Poshmark and Etsy is among young women. Other hashtags on the ranking are also intriguing, such as #handmade, #vintage, #jewelry, #ootd (meaning outfit of the day), #makeup, and #fitness, to name a few. Marketers can get some ideas from these popular hashtags for their marketing strategies.

Regarding the influential actors, we take two approaches. The first one is to simply identify what user accounts are mentioned the most in the tweets, which can be considered to be the popular users in this virtual community. Table 5 presents the top-50 popular user mention ranking from the tweets posted by the same young female users interested in fashion. The user @poshmarkapp is the most mentioned user account, which confirms that shopping on Poshmark is very popular. Note that not all the user accounts listed on this ranking match the young female users in our target audience. They are just the user accounts that were mentioned very frequently by them, some of whom can be outside the target audience.

The second approach to identifying influencers is to leverage two commonly-used measures: eigenvector centrality ([79]) from the network theory and retweet h-index from [80], which is an adaptive version of the traditional Hirsch index to retweets in Twitter data. For the eigenvector centrality measure, we first collect followers and followees data using the Twitter API mentioned in "User profiling" section, identify mutually following pairs of the young female users, and then build an undirected network graph. The network has 9809 nodes, which means that 9809 users out of 16,011 are connected to at least one user. This network is much denser than expected, considering that the users do not share many attributes: they only share the interest in fashion, the gender, and the age class. We finally apply the eigenvector centrality algorithm to the network graph, which basically favors users who are connected with other well-connected users in the network. This results in a centrality score for each user in the graph. It turns out that most users have very low centrality scores, whereas only a few have high centrality scores. We believe that this demonstrates a good example of the existence of influencers in a certain domain. For the retweet h-index measure, we use the Tweet object that contains the information of how many times a tweet has been retweeted by other users. This also results in a retweet h-index value for each user.

Table 6 presents the top-25 influential user ranking sorted by centrality (left side) and h-index (right side), respectively, in descending order. The number one user on the centrality ranking is Jacqueline Line (screen name @JacquelineRLine), who has 367K followers at the time of writing, is a popular user on Poshmark, and her timeline is filled with tweets on various fashion items. On the other hand, the number one user on the retweet h-index ranking is Shaniah (screen name @makeupbyshaniah), who has 115.4K followers at the time of writing, is a popular makeup artist and YouTuber. As shown in the table, the two influencer rankings present completely different users, which implies that the two measures exhibit different perspectives on influence.¹⁴

It is worth further analyzing this case study from a perspective of the Total Twitter Error framework mentioned in "Introduction" section, which helps us to evaluate potential errors in the study. As the study completely relies on the pool of random Twitter users and tweets to identify people interested in fashion, it is not free from the under-coverage error. In other words, it is obvious that the Twitter users found never represent all people in the world interested in fashion. Here, we make a strong assumption that we are only interested in Twitter users and our study is only targeted at those people in a social media world. We do not believe that this assumption is unreasonable, as we are well-aware that many people interested in fashion are using Twitter and having conversation in the cyberspace. Again, this should completely depend on the objectives of the study. On the other hand, the 16,011 young female users found are never small as a sample, as it would be challenging to gather this number of human subjects or respondents in traditional surveys. In addition, we identified and removed organization accounts, which definitely helped to reduce the over-coverage error in our data. In terms of the query error, while we could have added other hashtags than just #fashion and #style when identifying users interested in fashion, we believe that the two hashtags alone are representative of the interest in fashion. Lastly, there is room for the interpretation error, given that the user profiling solutions used are imperfect. In order to minimize the potential interpretation error, we (1) chose the solutions that demonstrate good performances in their papers and also (2) used more than one solutions for the same attribute whenever possible.

One limitation in this case study is that it would be ideal if we could compare the trends observed on Twitter to actual observable indicators coming from out-of-Twitter. To the best of our knowledge, we are unaware of any external data sets that can be mapped to our topic and user rankings for cross-evaluation. This limitation suggests future research in this case study.

The second case study aims to answer the question of whether the political orientation, i.e., conservative vs. liberal, affects people’s reaction to a gender-related issue. We choose the recent Me Too movement as one of the noticeable gender-related topics and attempt to compare how differently conservatives and liberals react to the same issue. To define the target audience for this case study, we take the same approach as the one used in the previous case study on young women interested in fashion: identifying the Twitter users in our pool who have ever used the #metoo hashtag in their tweets. Again, mentioning the hashtag is assumed to be their interest in the topic. From our pool, 68,116 users are identified as those who (1) have ever used the #metoo hashtag, (2) still have valid and public accounts on Twitter, and (3) have posted at least 100 tweets. Formally, \(|U| = 68116\). We then collect up to 3200 recent tweets for each of the users, which totals 188,806,239 tweets, or formally \(|T| = 188806239\).

The next step is to partition the users into two groups: conservatives and liberals. To that end, we opt to develop our own hashtag-based political orientation classifier fitted to our Twitter data for the same reason stated in "Customized user profiling" section. Specifically, we collect another set of users who can be easily labeled as “conservative” or “liberal” and use hashtags of those users as the features for political orientation prediction. We again search our random pool of users and tweets for users who described themselves in their bio as “proud republican”, “proud conservative”, “proud democrat”, or “proud liberal”, based on the observation that these expressions are a common way of expressing one’s political orientation and thus can serve as a strong indicator of their political orientation. In this way, we find the users who have those proud republican or conservative expressions in their bio and label them as “conservative”. Similarly, we label those who describe themselves as proud democratic or liberal as “liberal”. We further check if (1) the users still have valid and public accounts on Twitter and (2) have posted at least 100 tweets. This leaves 5,740 users in total, of which 4717 users are labeled as “liberal” and 1023 users are labeled as “conservative”. We then collect up to 3200 recent tweets of the users, which results in 12,299,722 tweets in total. From the collected tweets, we now extract top-1000 popular hashtags, which will be used as the features for prediction. As in the first case study, hashtags exclusively used by a single user are excluded. Table 7 presents the top-50 popular hashtags. As shown in the table, most of the hashtags are directly or indirectly related to politics, which is a clear indication that the labeled users collected for machine learning are interested in politics. Many of the hashtags on the ranking appear to be discriminative between the two classes, conservative and liberal, such as #trump and #bidenharris2020.

In our data, there are more samples tagged with “liberal” (4717) than those with “conservative” (1023). To avoid any potential bias in the classifier, we transform this unbalanced data set into a balanced data set by undersampling, i.e., selecting the same number of random samples from “liberal” samples as “conservative” samples. Next, we randomly split this data set of equal numbers of “conservative” and “liberal” samples into 80% of training data (1636 samples) and 20% of test data (410 samples). Then, to build a classification model, we apply widely-used classification algorithms to the training data, such as k-Nearest Neighbors, Logistic Regression, Random Forest, XGBoost, Support Vector Machines, Neural Networks, and Deep Neural Networks, for each of which we find the best hyper-parameters that yield the best performance. Lastly, we evaluate each model on the test data.

For model evaluation and selection, we compare the f1-scores, which are the harmonic means of precision and recall. As shown in Fig. 2, the Random Forest model yields the best performance with the f1-score of 0.91, which can be considered a very high accuracy for prediction. Figure 3 presents the Average Precision (AP) curve (left) and the Receiver Operating Characteristic (ROC) curve for the best performing Random Forest model. The Average Precision and Area Under the Curve (AUC) are 0.96 and 0.96, respectively, which confirm the excellent performance of the model. In addition, in order to identify which features (i.e., hashtags) contribute the most to prediction, we list the feature importance scores provided by the Random Forest algorithm. Table 8 presents the top-50 important features and their importance scores. The ranking shows that the #trump2020 hashtag contributes the most in terms of political orientation prediction, followed by #fjb, #moscowmitch, #traitortrump, #oann (meaning One America News Network), #resist, #bidenharris2020, and so on, which all make sense.

As the training data used for political orientation classification are biased toward the users who clearly described themselves as proud liberal/conservative, we further conduct out-of-sample performance evaluation. To create a new data set for out-of-sample evaluation, we randomly select 200 users whose bio has “democrat” or “liberal” with no “proud” and, likewise, 200 users whose bio has “republican” or “conservative” with no “proud”. Next, for each of the group of 200 users, we manually check if the user is actually liberal or conservative by reading their bio, which results in 179 liberal users and 116 conservative users. We then collect up to 3200 most recent tweets from their timelines and extract hashtag frequency features from their tweets. We then apply our political orientation classifier to those users and predict their political orientations. Finally, we compare their predicted political orientations with their actual ones. This results in an f1-score of 0.76. While this performance is lower than the with-in sample performance of 0.91, which is fully expected, the performance is still high enough to be used in real-world Big Data analysis.

In order to prove that hashtag features outperform full-text features in political orientation classification, we utilize BERT ([81]) as the baseline approach to compare, which is known to perform well in text classification. To clarify, our approach uses the frequencies of top-1000 popular hashtags as features, whereas BERT uses the full text of aggregated tweets of users as features for transfer learning. The f1-score we achieve from BERT is 0.61, which is far lower than 0.91 from the best-performing hashtag-based model. Our guess is that the full text of a user’s tweets has too much noise that does not help in identifying their political orientation, whereas hashtags serve as surprisingly good indicators.

Now that we have our own political orientation classifier fitted to tweet data, we apply the classifier to our 68,116 users who are interested in #metoo. This results in 46,037 users labeled as “conservative” and 22,079 users labeled as “liberal”. Unlike the training and test data for modeling the classifier, there are more conservatives than liberals in our Me Too data set.

We now proceed with the final step for comparing the views on the Me Too movement by political orientation. We compare the most popular hashtags that co-occur with the #metoo hashtag in the same tweet, based on the idea that there would be differences between liberals’ interests and conservatives’ interests in the same Me Too context. Table 9 presents the top-50 popular hashtag rankings from the tweets posted by liberals and by conservatives, respectively. Note that, while this table only shows the 50 most popular hashtags, there are much more hashtags following those top-50 hashtags.

In order to measure how different the two entire rankings are, we employ two measures: the cosine similarity and the rank correlation. For the cosine similarity measure, specifically, we transform each entire ranking into a vector of hashtag frequencies and then calculate the cosine similarity between the two vectors, which indicates the angle between the two vectors. The smaller the angle, the more similar the two vectors are. Cosine similarity ranges between 0 and 1, where being close to 1 means very similar and being close 0 means dissimilar. From the two hashtag ranking vectors, we get the cosine similarity of 0.65. For the second rank correlation coefficient measure, we calculate both the Spearman correlation coefficient and the Kendall correlation coefficient on the two entire rankings. A rank correlation coefficient ranges from −1 and 1, where being close to 1 indicates a positive correlation, being close −1 a negative correlation, and being close to 0 no correlation. We achieve −0.24 and −0.23, respectively, which are both closer to 0 than to 1 or −1. The cosine similarity and the rank correlation coefficients indicate the dissimilarity of the two rankings, which implies that the two groups’ interests are not the same.

To get an idea of specifically how the two rankings are different, Figs. 4 and 5 present the top-50 popular hashtag clouds for the liberals and the conservatives, respectively, in which larger hashtags represent more popular ones. Noticeably, the two hashtag clouds present somewhat different hashtags, as they have only 16 hashtags in common.¹⁵ Besides, many of the hashtags do not appear on the other cloud.¹⁶ These all confirm that liberals and conservatives do not equally take the same gender-related issue showing interests in somewhat different topics.

We now evaluate potential errors in this case study from a Total Twitter Error perspective. As with the first case study, this study relies on the pool of random Twitter users and tweets to identify people interested in the Me Too movement, and thus the same argument holds for this study: we assume that the set of 68,116 Twitter users found is sufficient for the study. In terms of the query error, we believe that the #metoo hashtag is the one and only hashtag we can think of and is representative of the interest in the Me Too movement, although there is a possibility that some users did not use the #metoo hashtag in their tweets. In this case, one may consider searching for any other expressions than just hashtags in tweet text that represent Me Too. Lastly, given the very high accuracy of our political orientation classifier, we believe that there is not much room for the interpretation error caused by customized profiling.