6. Opinion Analysis Corpora Across Languages

NTCIR MOAT was held at NTCIR-6 (Seki et al. 2007), NTCIR-7 (Seki et al. 2008), and NTCIR-8 (Seki et al. 2010). The task definition evolved through the three sessions, as shown in Table 6.1.

The goal of the task is to form a bridge between element technologies such as opinion/polarity sentence classification or opinion holder/target phrase recognition to an application such as (opinion) IR or question answering. The target languages include English, Chinese (both Traditional and Simplified), and Japanese, and the topic set for IR or question answering is shared across languages. We have prepared a document set relevant to the topics retrieved from newspaper articles published in each target language, and have evaluated the system using these document sets annotated with multiple assessors.

Many researchers have focused on a resourceless approach to sentiment analysis (Elming et al. 2014; Le et al. 2016). Blitzer et al. (2007) proposed a domain adaptation approach for sentiment classification. Wan (2009) addressed the Chinese sentiment classification problem by using English sentiment corpora on the Internet. This type of research can be categorized as a semi-supervised approach to opinion/sentiment analysis that aims to solve the resource problem by using small labeled and large unlabeled datasets. We recognize that addressing language resource problems in sentiment analysis for nonnative languages is an important research area. Alternatively, applications such as the Europe Media Monitor (EMM) News Explorer² provide an excellent service by including viewpoints from different countries. We also understand that providing these varied opinions from different countries offers opportunities for better worldwide communications. NTCIR MOAT is the first task to provide opportunities for nonnative researchers to develop a sentiment analysis system for low-resource languages and to bridge cultures by clarifying opinion differences across different languages.

With the broad range of information sources available on the web and in social media, there has been increased interest by both commercial and governmental parties in trying to analyze and monitor the flow of prevailing attitudes from anonymous users automatically. As a result, the research community has given much attention to automatic identification and processing of the following.

Opinion corpus annotation for multiple domains (as in news topics) usually requires expert linguistic knowledge because crowdsourcing annotation (such as the Amazon Mechanical Turk) does not fit the NTCIR MOAT annotation framework. We conducted our evaluation using agreed (intersection) annotations from multiple expert assessors. To check the stability of this evaluation strategy, we compared the evaluation results for agreed (intersection) annotation and selective (union) annotation to arrive at a gold standard for using NTCIR-8 MOAT submission data.

For the English cases in Table 6.2 (the \(\kappa \) coefficient between assessor annotations was 0.73) and the Traditional Chinese cases in Table 6.3 (\(\kappa \) coefficient 0.46), the rank of the participants’ systems is different. Although the rank differences for the English cases were within statistical significance, among the Traditional Chinese cases, the precision-oriented systems (CTL and WIA) tended to be ranked higher for cases of agreed (intersection) annotation, and recall-oriented systems (KLELAB-1 and NTU) tended to be ranked lower. For the Simplified Chinese cases in Table 6.4 (\(\kappa \) coefficient 0.97) and the Japanese cases in Table 6.5 (\(\kappa \) coefficient 0.72), there was no rank difference for the participants’ systems despite the different strategies because of either high \(\kappa \) agreement (Simplified Chinese) or a low number of participants (Japanese). From these observations, we concluded that the \(\kappa \) coefficient between assessor annotations should exceed 0.7 for stable evaluation. We also found that strong opinion definition and online annotation tools were helpful, but using expert linguistic annotators remained necessary to achieve high \(\kappa \) agreement.

We ranked topics by averaging their F1-scores, the harmonic mean of precision and recall, obtained from all NTCIR-8 MOAT raw submissions in the opinionated judgment subtask. The best three (easy) topics and worst three (difficult) topics and the opinion percentage in the source documents are shown in Table 6.6.

Some researchers have used the NTCIR MOAT test collection and presented their work at top-rated conferences, particularly those focused on cross-lingual sentiment analysis. Two representative examples are as follows.

Several opinion corpora involving news have been developed after NTCIR MOAT was published. In this subsection, we introduce the SemEval-2007 Task 14: Affective Corpus (Strapparava and Mihalcea 2007) and the sentiment-annotated quotation set (Balahur and Steinberger 2009; Balahur et al. 2010).

In the SemEval-2007 Affective Corpus, six emotion labels and two polarity labels have been annotated to headlines collected from 1,250 news websites and newspaper articles. The sentiment-annotated quotation set contains a set of 1,590 English language quotations (reported speech), manually annotated by two independent sets of annotators for sentiment (positive, negative, or objective/neutral) expressed toward the entities mentioned inside the quotation. Web crawling for news articles employed the EMM (Steinberger et al. 2009)⁴ developed by the European Commission Joint Research Centre.

After NTCIR MOAT was published, Twitter⁵ and other microblog media came into widespread use by many users. The NLP/IR researchers also focused on tweet sentiment analysis (Martinez-Camara et al. 2013). To improve sentiment classification in Twitter, specific clues were found to be useful because a tweet is much shorter than a news article, including tweet context (Jiang et al. 2011), emoticons and hashtags (Purver and Battersby 2012), lengthened words (Brody and Diakopoulos 2011), and emoji (Felbo et al. 2017).

On the other hand, deep NLP research such as Stanford Sentiment Treebank (Socher et al. 2013)⁶ has become mainstream from a technological point of view. In this research, the learning model builds up a representation of whole sentences based on the sentence structure. An opinion corpus called the Stanford Sentiment Treebank has been developed to estimate compositionality in the sentiment detection task. It includes the fine-grained sentiment labels “very negative”, “negative”, “neutral”, “positive”, and “very positive” for 215,154 phrases in trees parsed with the Stanford Parser from 11,855 sentences extracted from movie reviews (Pang and Lee 2005).

In SemEval 2018 (Mohammad et al. 2018), an opinion corpus has been created from 10,983 English, 4,381 Arabic, and 7.094 Spanish tweets, and used to evaluate the systems. Several tasks are defined that provide annotations for the mental state of the tweeter, including (1) the intensities of the four basic emotions (anger, fear, joy, and sadness), (2) the intensity of sentiment/valence (very negative, moderately negative, slightly negative, neutral or mixed, slightly positive, moderately positive, and very positive), and (3) multi-label emotion classification across 12 emotions (anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust, and neutral). The corpus used best–worst scaling (Louviere et al. 2015), a comparative annotation method in which assessors were asked what was the best (highest in terms of the property) and worst (lowest in terms of the property), given n items (typically n \( = 4\)). Real-valued scores for the association between the items and the property were determined based on the number of times an item was chosen as the best and the worst. The median number of assessors for each tweet was seven. The inter-annotator agreements (Fleiss’s \(\kappa \)) for the multi-label emotion classification were 0.21, 0.29, and 0.28 for the 12 classes, and 0.40, 0.48, and 0.45 for the four basic emotions in English, Arabic, and Spanish. Most of the participants employed SVM/SVR, LSTMs, and Bi-LSTMS as machine learning algorithms, and also took word embedding, affect lexicon features, and word n-grams as features.

Although the document genres being focused on and the annotation properties have changed over time, cross-lingual opinion corpora remain important in current research.