Big data meets storytelling: using machine learning to predict popular fanfiction

Our focus is to collect fanfiction about Supernatural in English. Given this language constraint, we cannot tap into the vast Internet literature in other languages such as Chinese, where the largest platform (Cloudary Corporation) officially reports 10 million registered users per day (Lu 2016). Fanfictions in English can be found on several websites (Table 2). While Wattpad has been the subject of prior studies on fanfictions, these studies are either qualitative (Budiarto et al. 2021) or use small sample sizes of a few thousand stories (Pianzola et al. 2020). The main two sources by volume are AO3 and Fanfiction.net. This result is aligned with prior works showing that AO3 is by far the main source and has been rising (McCullough 2023), while Fanfiction.net is a secondary source with a declining market share (Fiesler and Dym 2020). Prior works have used either of these two sources (Table 1), as their terms of service are compatible with the use of computer programs to automatically download content (i.e., data scraping).

On October 2022, we collected Supernatural stories and metadata from both AO3 and Fanfiction.net to achieve a diverse corpus. We used the Python scraper for AO3 created by Jingyi Li and Sarah Sterman (Li et al. 2017). The scraper places one request at most every five seconds. Fanfiction.net uses Cloudflare, which tends to detect web scrapers as hostile bots. We thus used a webdriver (Selenium) to automatize the requests. To avoid creating an undue load on either website and remain within the terms of a fair use,¹ we scrapped 100,310 stories from Fanfiction.net and 72,300 from AO3. The scrapers collected metadata alongside each story. Seven features were obtained for both websites: a unique story ID, the title, URL, rating,² language, publication date, and word count. On AO3, we also obtained the number of ‘kudos’³ (for popularity), number of views, and number of chapters. On Fanfiction.net, we obtained the number of reviews ‘favs’ (for popularity), the number of followers and reviews, the author, and the genre. As exemplified in log-log plots (Fig. 2), there is significant variance in the content of the stories that we collected with respect to the attention that they receive and the size of the story. In Fig. 3, we also note a small correlation between the attention that a story receives (measured by number of reviews) and the extent to which readers endorse it (as measured by ‘like’ or ‘favs’). In the context of online shopping, popular products are defined as “products with many reviews” (Heck et al. 2020), hence we also expect a correlation between measures of popularity in our context.

When using traditional classification methods (detailed in the next subsection), metadata is useful but not sufficient to accurately characterize why a story is popular. We thus need to perform feature engineering to extract additional (potentially) informative features. As emphasized by Minaee and colleagues, this “reliance on the hand-crafted features requires tedious feature engineering and analysis to obtain good performance” (Minaee et al. 2021). Indeed, Sect. 2 showed that many options are available, from sentiment to character networks. It is thus common to start by becoming acquainted with the corpus, which informs the choice and configuration of tools for the ensuing automatic analysis. Four readers independently examined three stories each, with a minimum of 1,000 words, and then collectively synthesized characteristics of stories that readers appeared to like. These characteristics were related to whether the story had a summary, a disclaimer, or author notes; the number of locations, characters, and dialogs; the main character, overall emotion of the story⁴, lexical diversity (i.e., number of unique words) and average word length, and sentiments associated with the two key protagonists of the TV show (Sam and Dean Winchester).

We captured sentiments through seven dimensions for each protagonist (excitement, satisfaction, politeness and lack thereof, sympathetic, frustration, sadness). The distribution of sentiments across stories were similar for the two protagonists and centered on four positive dimensions (excited, satisfied, polite, sympathetic), while the remaining three were much less prevalent (Fig. 4). Note that these seven dimensions are not perfectly orthogonal, as evidenced by the high correlations in Fig. 5; the distributions underlying each correlation are provided online in Supplementary Figures 1 and 2. Overall, our approach added 24 engineered features to the three obtained from web scraping (Table 3). We computed the engineered features using Python libraries including Watson NLP from IBM and Natural Language API from Google. These services start to incur a significant cost at a large scale, hence we created engineered features for a sample of 79,288 stories given a target budget of \(\$5,300\).

Two of the features contain categorical data: the genre provided by the author, and the main character that we detected via NLP. Machine learning algorithms commonly require categorical data to be turned into numerical data. We adopt a common machine learning approach that utilizes “a one-hot encoding technique to convert string labels to numerical labels” (Wanda and Jie 2021). If a given feature had N categorical values, then it is replaced by N binary features which indicate the absence (0) or presence (1) of each possible value.

A classifier is a function that predicts a class given certain features. In our case, we use the features described in the previous subsection to predict whether a fanfiction is popular, hence we perform a binary classification. A longstanding practice in text classification is to use different algorithms to train classifiers (Kadhim 2019), in order to identify the right type of function based on performances. For example, some algorithms are well-suited when the data is linearly separable while others are able to make nonlinear cuts (Fig. 6). In addition, certain algorithms specialize in massive amounts of data or in specific data types (e.g., images). Our data consists of 79,288 rows and 25 columns (24 features and 1 class outcome) structured in a tabular format. Since the complexity and the volume of the data are not aligned with deep learning, we focus on a classic approach and employ two of the most commonly used methods for text classification (Aggarwal and Zain 2012): decision trees and support vector machines. Together, these methods cover both linear and nonlinear function hypotheses (Crutzen and Giabbanelli 2014). Decision trees work well with linear decision boundaries (Kowsari et al. 2019) and they can be trained quickly. Support Vector Machines (SVMs) have been regularly employed for classification with text as they can handle non-linear cases using the ‘kernel trick’ (Fig. 6-right); they resemble a logistic regression when using a linear separation. SVMs are among the most resource-intensive models to train (Kadhim 2019), at the exception of deep learning models. In order to provide a comprehensive set of baseline algorithms for comparison, we also include random forests (i.e., sets of decision trees), logistic regression, and a neural network with 8 layers.

For the decision tree, we optimized two hyper-parameters that limit the cuts that can be made and hence force a simplification of the model. Setting a maximum depth to the tree prevents too many successive cuts, which can arise when the algorithm attempts to isolate a few points and hence causes an overfit. When a decision tree makes a cut, it intuitively divides it into ‘left’ and ‘right’ sides, where different features can be selected for the next cuts. The total number of features used by a tree of depth d thus scales with \(2^d\). If we want each of our 24 features to be used potentially at least once, then we need \(2^d > 24 \times 2\) hence we can pick \(d=5\). If we want to over-provision and potentially use each feature three times, then \(d=7\) would suffice. We thus considered three values of the maximum depth to force a simplification, use each feature once, or over-provision. Raising the minimum number of samples to split also avoids creating cuts in areas that lack data. The impact on the tree was discussed in Rosso and Giabbanelli (2018). For a support vector machine, choosing the right kernel is a notoriously difficult problem (Kowsari et al. 2019) hence we considered four types of kernels. The linear kernel is most useful when data can be linearly separated, which particularly applies to text (Pillutla et al. 2020). We employed the Gaussian Radial Basis Function (RBF) kernel as it is a common alternative to a linear kernel, and the most widely used form of kernels relying on an exponential (aternatives include the Laplace RBF kernel of the Gaussian kernel). We chose the sigmoid kernel as a proxy to small neural networks (i.e., a two-layer perceptron), where other options include the hyperbolic tangent kernel. Although polynomial kernels have known limitations (Steinwart 2001), we included them since they have been used in prior works on text classification (Kalcheva et al. 2020). For each of these four kernels, we optimized multiple kernel-dependent hyper-parameters. Since training an SVM is computationally and memory expensive, a comprehensive optimization process could quickly become prohibitive and force us to use heuristics (Dudzik et al. 2021). We thus focused on binary parameters and limited the number of levels for numerical parameters.

The optimization process for the decision tree, random forest, and SVM used a grid search⁵ to consider all combinations of values listed in Table 4. Since we need to both obtain robust performance estimates and perform a grid search, we divide the data into training, testing, and validation sets. This division is conducted through a 10x10 nested cross validation, also known as a double cross-validation. That is, the data is first split into 10 parts (known as outer folds), nine of which are used for model building and one for testing, until all parts have been involved. For each of the ten instances of model building, this portion of the data is further divided into 10 parts (known as inner folds), nine of which serve to train the model and include a grid search, and the other one serving to measure the initial validation. We used the classic Adam optimizer and gradient descent for the neural network, with Binary Cross Entropy as loss function. Our optimization process provides three metrics (accuracy, precision, recall) for each of the ten outer folds, which allows to compute a confidence interval and thus estimate the robustness of the results vis-á-vis the data.