A Comparative Study of Uncertainty Based Active Learning Strategies for General Purpose Twitter Sentiment Analysis with Deep Neural Networks

The classifier used in these experiments is a convolutional neural network. Its basic architecture is described in Zhang and Wallace (2015). First, the tokenized tweet is transformed into a list of dense word embeddings. The resulting sentence matrix is then convolved with a certain set of filters of potentially varying region sizes. After that, the resulting feature maps, which are vectors describing certain “higher order features” of the tweet, activate a 1-max-pooling layer via a possibly non-linear activation function. Lastly, this pooling layer is densely connected to the output layer using softmax activation and optional dropout regularization. In contrast to Zhang and Wallace (2015), our output layer has three neurons, reflecting the fact that we want to differentiate the three classes positive, negative and uncertain.¹

All weights of the network were initialized randomly except for the embedding layer, where we used word2vec vectors (cf. Mikolov et al. (2013)) of dimension \(d = 100\) trained on a dataset of approximately 33 million tweets collected between June 2012 and August 2013 by Neubauer (2014). After some minimal preprocessing², this dataset contained 624, 015 unique tokens, of which we used the 200, 000 most frequent ones in the network. The parameters were chosen as follows: The model used was the skip-gram model, the window size was 5 words, the subsampling threshold was \(t = 10^{-5}\); negative sampling was used with \(k = 5\) “noise words” and we ran two iterations of the algorithm. Most of these values were recommended by Mikolov et al. (2013), where one can also find explanations for the parameters. The rest of the network’s hyperparameters was found using a search guided by the best practices laid out in Zhang and Wallace (2015): We first evaluated networks with only one region size \(r \in \{2, 3, 4, 5, 6, 8, 10\}\) and \(n \in \{50, 325, 600\}\) filters. The activation function f between the convolution and pooling layers was chosen from the set {id, tanh, RelU³} and the dropout rate (Srivastava et al. 2014) was \(p \in \{0, 0.25, 0.5\}\).

We evaluated all of these combinations based on their average macro- \(F_1\) -score in a tenfold cross-validation using the dataset from Haldenwang and Vornberger (2015). First, each network was trained using a distant supervision procedure with noisy labels based on emoticons in the dataset of Neubauer (2014). Note, that the distant super vision approach only consists of positive and negative tweets, since there is no reliable noisy label for uncertain tweets. Next, the network’s parameters were further refined by using the positive and negative tweets from the datasets of the SemEval competitions (Nakov et al. 2013, Rosenthal et al. 2014, 2015) for training.⁴ The networks were trained using the Adagrad (Duchi et al. 2011) algorithm. Both datasets were presented once (one epoch) in a batch size of 50 tweets.

The best configuration turned out to be \(r = 2\), \(n = 50\), \(f = \tanh \) and \(p = 0.25\) with an average F-score of \(F_1 \approx 0.56\). We also tried adding bigger filters to this configuration in multiple ways, but none of the resulting configurations could significantly surpass the above, so we do not go into further details of this process here. For the following experiments with regard to active learning, we used the version of this network that was only trained on the noisy labels, to properly reflect one of the constraints of this approach: not to have a big supply of manually labeled tweets in advance.

As a strategy to query the best suited tweets to label for the network, we decided to investigate uncertainty sampling, a strategy originally devised by Lewis and Gale (1994) which is both easy to implement and understand and thus commonly used. With this strategy, each tweet is assigned an uncertainty value which defines how uncertain the network is in finding the correct label for the tweet. The most uncertain tweets are then chosen to be labeled.

For a problem with three (or more) classes such as ours, there are different metrics available to calculate uncertainty. These metrics differ in how many of the class probabilities they take into account. In the following a short description for each of the metrics provided. A more thorough introduction and comparison can be found in the literature survey of Settles (2010).

The confidence metric can be used to choose the tweet \(x^{*}_{LC}\) whose label the network is least confident about:The confidence is defined as the probability that the class label \(\hat{y}\) chosen by the network \(\theta \) is correct as considered by the network itself (and as such is the highest of the three probabilities for the three class labels).

The margin metric also takes the second highest probability into account by calculating the difference between the probabilities of the two class labels \(\hat{y_1}\) and \(\hat{y_2}\) the network believes to be most likely correct:A tweet with a smaller margin would be considered more uncertain since the network has difficulties choosing between the labels \(\hat{y_1}\) and \(\hat{y_2}\).

Finally, the entropy metric considers the probability for all class labels \(\hat{y_i}\) to calculate the amount of informativity each tweet has to offer to the network:In our experiment we compare the effect of these metrics to find out which is most helpful for our use case.

To speed up the labeling process, we query and label the tweets in batches of 20. However, since the uncertainty values are not recalculated after picking a tweet for a batch, this could lead to the tweets in the batch being very similar to one another since they all occupy the same uncertain region of the feature space. To avoid this, we introduce diversity as a second criterion to our querying process as described in (Patra and Bruzzone 2012):

For each of the uncertainty metrics described above, the experiment is initialized with a copy of the initial deep neural network that was pretrained with the aforementioned distantly supervised data only. The corpus of unlabled tweets to chose from consisted of 100,000 tweets that were randomly sampled from the 33 million dataset of Neubauer (2014). First, all tweets in the unlabeled corpus are classified by the network and then 20 tweets are chosen to be annotated using the previously mentioned strategy. Next, after the 20 tweets are labeled by the human annotator, 10 training iterations are performed with the newly annotated tweets. This procedure is then repeated until 1, 000 tweets are annotated for each uncertainty metric.

Additionally, we generated a random baseline by training a copy of the initial neural network with randomly selected, manually annotated tweets in batches of 20 with 10 training iterations.

Each generated network was then evaluated using the reliable general purpose Twitter sentiment analysis data set from Haldenwang and Vornberger (2015) as a test set. The resulting macro \(F_1\)-score is reported.