Chinese EmoBank: Building Valence-Arousal Resources for Dimensional Sentiment Analysis

2.1 Dimensional Sentiment Resources

Table 1 presents the language resources for dimensional sentiment analysis. A number of one-dimensional sentiment lexicons provide sentiment intensity or strength of words, including SentiWordNet [Baccianella et al. 2010], SentiFul [Neviarouskaya et al. 2011], SO-CAL [Taboada et al. 2011], AFINN [Nielsen 2011], SentiStrength [Thelwall et al. 2012], and VADER [Hutto and Gilbert 2014]. Specifically, NRC-EIL provides sentiment intensity for eight emotions [Mohammad 2018a]. The SemEval and WASSA shared tasks also released several datasets for single words, multi-word phrases [Rosenthal et al. 2015; Kiritchenko et al. 2016], and sentences [Cortis et al. 2017; Mohammad and Bravo-Marquez 2017; Mohammad et al. 2018]. Stanford Sentiment Treebank [Socher et al. 2013] provided fully labeled parse trees containing sentiment scores at both the phrase- and sentence-levels.

Among multi-dimensional resources, ANEW is the first three-dimensional lexicon providing real-valued scores for the valence, arousal, and dominance dimensions [Bradley and Lang 1999]. ANEW has been extended from 1,034 words to 13,915 words [Warriner et al. 2013]. NRC-VAD provides 20,007 English words with valence, arousal, and dominance ratings [Mohammad 2018b]. In addition to lexicon resources, several multi-dimensional corpora have been proposed. ANET is the first three-dimensional corpus providing valence, arousal, and dominance ratings [Bradley and Lang 2007]. A corpus of 2,895 Facebook posts [Preoţiuc-Pietro et al. 2016] was annotated to provide two-dimensional valence and arousal ratings. EmoBank [Buechel and Hahn 2017] provides 10,062 sentences with valence, arousal, and dominance ratings. All of the above multi-dimensional lexicons and corpora are scored from 1 to 9.

2.2 Dimension Score Prediction

The above dimensional sentiment resources have been used for dimension score prediction at the word-, phrase-, and sentence-levels. These approaches can be categorized as lexicon-based [Paltoglou et al. 2013], regression-based [Wei et al. 2011, Malandrakis et al. 2013; Paltoglou and Thelwall 2013; Amir et al. 2015; Wang et al. 2016a; 2016b], and neural-network-based models [Du and Zhang 2016; Vilares et al. 2016; Wu et al. 2017; Goel et al. 2017; Zhu et al. 2019; Yu et al. 2020; Wang et al. 2020; Huang et al. 2020].

Lexicon-based methods typically determine the sentiment score of a text by averaging the sentiment scores of the words in the text [Paltoglou et al. 2013]. Regression-based methods have been intensively studied for dimension score prediction. Wei et al. [2011] proposed a cross-lingual approach that trained a linear regression model using the dimension scores of a set of English seed words (source) and their translated Chinese seed words (target). Wang et al. [2016a] further extended their work using a locally weighted linear regression model. Malandrakis et al. [2013] built a linear regression model using n-grams with sentiment scores as features. Both Paltoglou and Thelwall [2013] and Amir et al. [2015] used support vector regression (SVR). Wang et al. [2016b] developed a community-based weighted graph model that performed the regression task on a graph using a social network method to predict the dimension scores of words.

Recently, deep neural network models with word embeddings [Mikolov et al. 2013a; 2013b; Pennington et al. 2014; Bojanowski et al. 2017] or sentiment embeddings [Tang et al. 2016; Yu et al. 2018] have been widely applied to dimensional score prediction. Du and Zhang [2016] used a boosted neural network trained on character-enhanced word embeddings to predict the dimension scores of words. Vilares et al. [2016] used a CNN trained on Twitter word embeddings to determine the sentiment of tweets from highly negative to highly positive using a five-point scale. Wu et al. [2017] introduced a densely connected deep LSTM model to concatenate features at different levels to predict the dimension scores of both words and phrases. Goel et al. [2017] presented an ensemble of different neural networks to determine the intensity level for different emotion categories such as anger, fear, joy, and sadness. Zhu et al. [2019] presented an adversarial attention network to predict the dimension scores of short texts. Yu et al. [2020] proposed a pipelined neural network model to sequentially learn word intensity and modifier weights for phrase-level sentiment intensity prediction. Wang et al. [2020] developed a regional CNN-LSTM model that integrates both local (regional) information within sentences and long-distance dependency across sentences to predict the dimension scores of long texts. Huang et al. [2020] incorporated a context-dependent sentiment lexicon into a 3-channel CNN to predict the strength of both words and texts.

3.1 Data Collection

The words in CVAW are collected from two polarity-based sentiment lexicons, C-LIWC [Huang 2012] and NTUSD [Ku and Chen 2007]. These affective words are then combined with a set of modifiers such as negators (e.g., not), degree adverbs (e.g., very), and modals (e.g., should) to form multi-word phrases, i.e., the CVAP. The frequency of each phrase is then retrieved from a large web-based corpus. Only phrases with a frequency greater than or equal to 3 are retained as candidates. To prevent several modifiers from dominating the whole resource, each modifier (or modifier combination) can have at most 50 phrases. In addition, the phrases are selected to maximize the balance between positive and negative words, which helps prevent positive or negative words from dominating the whole resource. The balance is accomplished by, for example, randomly selecting 25 positive words and 25 negative words to constitute the 50 phrases for each modifier. Finally, a total of 2,998 multi-word phrases are selected into the CVAP. To build the CVAS, we first collect Chinese tweets from the social networking service Twitter, selecting tweets that contain the greatest number of affective words found in the CVAW. The tweets are then split into sentences using existing punctuation for a total of 2,582 single sentences excluding emoticons, URLs, and abusive language. For the CVAT, we collect web texts from six different categories: news articles, political discussion forums, car discussion forums, hotel reviews, book reviews, and laptop reviews. Texts containing incomplete semantics and abusive language are excluded. Finally, a total of 2,969 multi-sentence texts containing the greatest number of affective words found in the CVAW were selected for annotation.

3.2 Annotation Details

The annotation of VA ratings is accomplished by crowdsourcing. For the CVAW, each word is randomly assigned to five annotators for rating, and each instance of CVAP, CVAS, and CVAT is randomly assigned to 10 annotators. The number of annotators used for the CVAW is lower than that of CVAP, CVAS, and CVAT because the word-level rating is relatively easier to determine than at the phrase-, sentence-, and multi-sentence levels.

The annotation platform is implemented with the SAM rating scale [Bradley and Lang 1994] using the Google app engine. Figure 2 shows an example of the annotation screen. The top part of Figure. 2 presents an example sentence in the CVAT, followed by the picture-oriented SAM rating scale. Both the valence and arousal dimensions use a nine-degree scale. Value 1 on the valence and arousal dimensions respectively denotes extremely high-negative and low-arousal sentiment, while 9 denotes extremely high-positive and high-arousal sentiment, and 5 denotes a neutral and medium-arousal sentiment. The picture-oriented protocol can help annotators determine the VA ratings more precisely. Volunteer annotators use the annotation screen to provide the VA ratings for each instance in the CVAW, CVAP, CVAS, and CVAT.

Fig. 2.

View Figure

3.3 Corpus Cleanup

Once the annotation process is finished, a cleanup procedure is performed to remove outlier ratings and improper instances (e.g., those containing abusive or vulgar language). Outliers are identified if they do not fall into the interval of the mean plus/minus 1.5 standard deviation (SD). They are then excluded from the calculation of the average VA ratings for each instance in the constructed Chinese EmoBank. Table 2 shows the annotation results of the example sentence presented in Figure 2. For the valence dimension, the rating 8 provided by annotator A10 is marked as an outlier because it exceeds the mean plus 1.5 standard deviation. Similarly, the rating 2 in the arousal dimension provided by annotator A1 is also marked as an outlier. After excluding outlier ratings, the (mean, SD) of the example sentence for the valence dimension is (5.778, 0.416) and that for the arousal dimension is (5.333, 0.667). To use the Chinese EmoBank to train a prediction model, the standard deviation is a useful metric to exclude instances with inconsistent annotations. For example, a previous study suggested excluding instances with a standard deviation higher than 2 [Paltoglou et al. 2013].

Table 2.

• *denotes an outlier.

View Table

Table 2. Example of Corpus Cleanup

• *denotes an outlier.

4.1 Results of the Chinese EmoBank

Table 3 shows the mean and standard deviation of the VA ratings for the Chinese EmoBank. The results show that the standard deviation of the arousal dimension is greater than that of valence, indicating that the arousal ratings are more difficult to be determined by the annotators.

Figure 3 shows the scatter plot of the CVAW. It presents a smile curve, indicating that both high-positive and high-negative words usually have a high arousal value. Table 4 lists several example words for the four quadrants of the VA plane.

Fig. 3.

View Figure

Table 4.

View Table

Table 4. Annotated Examples of the CVAW

For the CVAP, a total of 52 modifiers (including 4 negators, 42 degree adverbs, and 6 modals - see Table 5) are combined with the affective words in the CVAW to form the multi-word phrases. Table 6 shows the distribution of different phrase patterns in the CVAP. Table 7 lists an example phrase for each pattern. Figure 4 shows the scatter plot of the CVAP.

Table 5.

View Table

Table 5. Modifier Set Used in the CVAP

Table 7.

View Table

Table 7. Annotated Examples of the CVAP

Fig. 4.

View Figure

Table 8 shows the distribution of the text categories along with their sentence and word counts in CVAS and CVAT. The CVAS is collected from Twitter alone and the CVAT is collected from web texts based on six different categories where News is the major class (50.83%). Figures 5 and 6 respectively show the scatter plot of single sentences and multi-sentence texts in the CVAS and CVAT. Both scatter plots present a smile curve, which is consistent with that of the CVAW and CVAP. Tables 9 and 10 respectively list several example sentences for the CVAS and CVAT.

Fig. 5.

View Figure

Fig. 6.

View Figure

Table 9.

View Table

Table 9. Annotated Examples of the CVAS

Table 10.

View Table

Table 10. Annotated Examples of the CVAT

4.2 Valence-Arousal Rating Prediction

To demonstrate the application of the constructed affective resources, this section evaluates the performance of the lexicon-based, regression-based, and neural-network-based methods for the valence-arousal rating prediction of the affective corpora.

This experiment used three affective corpora. (i) EmoBank [Buechel and Hahn 2017] contains 10,062 multi-sentence texts. Each text was rated with individual dimensions (valence/arousal/dominance) in the range of (1, 5). (ii) Chinese valence-arousal single-sentences (CVAS) and (iii) Chinese valence-arousal multi-sentence texts (CVAT) are our constructed affective corpora with VA ratings. The former is collected from Twitter. The latter consists of texts collected from six categories, including book reviews, car reviews, hotel reviews, laptop reviews, political commentary, and news.

The following methods were compared to demonstrate their performance. (i) Lexicon-based method [Paltoglou et al. 2013]: For the EmoBank, the Extended ANEW [Warriner et al. 2013] was used to predict the valence (or arousal) ratings of a given sentence by averaging the valence (or arousal) ratings of the words matched in the Extended ANEW. For both CVAS or CVAT, we used the CVAW and CVAP. (ii) Regression-based method: including the linear regression (LR) [Wei et al. 2011] and SVR [Paltoglou and Thelwall 2013; Amir et al. 2015]. (iii) Neural-Network-based method: including CNN, RNN, LSTM, attention LSTM [Yang et al. 2016], BERT [Devlin et al. 2018], and XLNet [Yang et al. 2019].

Experimental settings are described as follows. We used a 5-fold cross validation technique to evaluate the effectiveness of the above methods. In addition, the suggested default parameters shown in Table 11 were selected without further fine-tuning. The word vectors for English and Chinese were trained using the BERT technique [Devlin et al. 2018]. Pre-trained models with whole word masking were downloaded from official BERT GitHub website. For English, we used BERT-Large, Cased (24-layer 1024-hidden, 16-heads, 340 parameters). The dimensionality is 1024. For Chinese, we used BERT-Base, Chinese Simplified, and Traditional (12-layer, 768-hidden, 12-heads, 110 parameters). The dimensionality is 768. The pre-trained XLNet¹ and BERT² models are publicly available online for evaluation.

Performance was evaluated using the mean absolute error (MAE) and Pearson correction coefficient (r), defined as the follows

• Mean Absolute Error (MAE):(1) \[\begin{equation}MAE = \frac{1}{n}\sum\limits_{i = 1}^n {|{a_i} - {p_i}|} \end{equation}\]

• Person Correlation Coefficient (r):(2) \[\begin{equation}r = \frac{1}{{n - 1}}\sum\limits_{i = 1}^n {\left(\frac{{{a_i} - {\mu _A}}}{{{\sigma _A}}}\right)\left(\frac{{{p_i} - {\mu _P}}}{{{\sigma _A}}}\right)} \end{equation}\]

where \({a_i} \in A\) and \({p_i} \in P\) respectively denote the i-th actual value and predicted value, n is the number of test samples, μ_A and σ_A respectively represent the mean value and the standard deviation of A, while μ_P and σ_P respectively represent the mean value and the standard deviation of P. The MAE measures the error rate, and the r measures the linear correlation between the actual values and the predicted values. A lower MAE and a higher r indicate more accurate prediction performance.

Tables 12, 13, and 14 respectively show the prediction results for CVAP, CVAS and CVAT. All three datasets produce nearly consistent findings. The lexicon-based method provides baseline results. For both regression-based methods, the SVR approach outperformed the LR in both the valence or arousal dimensions. The BERT model outperformed the other neural-network-based methods in all dimensions. For CVAP, the Attention model underperformed the LSTM, possibly because phrases are usually very short with one or two modifiers to a word, thus raising challenges for the attention mechanism. Comparing the results achieved by our constructed CVAS and CVAT, all models on the CVAS clearly underperformed the corresponding model results on the CVAT. Based on our observations, the CVAS data containing single sentences from Twitter has more difficulty predicting valence-arousal ratings than multi-sentence texts that provide more information in CVAT. Table 15 also shows the EmoBank results for reference. The consistent conclusions confirm the reliability of our constructed Chinese EmoBank corpus.

Table 12.

View Table

Table 12. Comparative Results of Different Methods in CVAP

Table 13.

View Table

Table 13. Comparative Results of Different Methods in CVAS

Table 14.

View Table

Table 14. Comparative Results of Different Methods in CVAT

Table 15.

View Table

Table 15. Comparative Results of Different Methods in EmoBank