1 Introduction
2 Background and related work
2.1 Focus on sentence-level sentiment analysis
2.2 Existing efforts on comparison of methods
3 Sentiment analysis methods
Name
|
Description
|
L
|
ML
|
---|---|---|---|
Emoticons [20] | Messages containing positive/negative emoticons are positive/negative. Messages without emoticons are not classified. | ✓ | |
Opinion Lexicon [2] | Focus on Product Reviews. Builds a Lexicon to predict polarity of product features phrases that are summarized to provide an overall score to that product feature. | ✓ | |
Performs subjectivity analysis trough a framework with lexical analysis former and a machine learning approach latter. | ✓ | ✓ | |
Construction of a lexical resource for Opinion Mining based on WordNet [26]. The authors grouped adjectives, nouns, etc. in synonym sets (synsets) and associated three polarity scores (positive, negative and neutral) for each one. | ✓ | ✓ | |
LIWC [7] | An acronym for Linguistic Inquiry and Word Count, LIWC is a text analysis paid tool to evaluate emotional, cognitive, and structural components of a given text. It uses a dictionary with words classified into categories (anxiety, health, leisure, etc.). An updated version was launched in 2015. | ✓ | |
Sentiment140 [27] | Sentiment140 (previously known as ‘Twitter Sentiment’) was proposed as an ensemble of three classifiers (Naive Bayes, Maximum Entropy, and SVM) built with a huge amount of tweets containing emoticons collected by the authors. It has been improved and transformed into a paid tool. | ✓ | |
SenticNet [28] | Uses dimensionality reduction to infer the polarity of common sense concepts and hence provide a resource for mining opinions from text at a semantic, rather than just syntactic level. | ✓ | |
AFINN [29] - a new ANEW | Builds a Twitter based sentiment Lexicon including Internet slangs and obscene words. AFINN can be considered as an expansion of ANEW [30], a dictionary created to provides emotional ratings for English words. ANEW dictionary rates words in terms of pleasure, arousal and dominance. | ✓ | |
SO-CAL [31] | Creates a new Lexicon with unigrams (verbs, adverbs, nouns and adjectives) and multi-grams (phrasal verbs and intensifiers) hand ranked with scale +5 (strongly positive) to −5 (strongly negative). Authors also included part of speech processing, negation and intensifiers. | ✓ | |
Emoticons DS (Distant Supervision) [32] | Creates a scored lexicon based on a large dataset of tweets. Its based on the frequency each lexicon occurs with positive or negative emotions. | ✓ | |
NRC Hashtag [33] | Builds a lexicon dictionary using a Distant Supervised Approach. In a nutshell it uses known hashtags (i.e. #joy, #happy, etc.) to ‘classify’ the tweet. Afterwards, it verifies frequency each specific n-gram occurs in a emotion and calculates its Strong of Association with that emotion. | ✓ | |
Pattern.en [34] | Python Programming Package (toolkit) to deal with NLP, Web Mining and Sentiment Analysis. Sentiment analysis is provided through averaging scores from adjectives in the sentence according to a bundle lexicon of adjective. | ✓ | |
SASA [35] | Detects public sentiments on Twitter during the 2012 U.S. presidential election. It is based on the statistical model obtained from the classifier Naive Bayes on unigram features. It also explores emoticons and exclamations. | ✓ | |
PANAS-t [8] | Detects mood fluctuations of users on Twitter. The method consists of an adapted version (PANAS) Positive Affect Negative Affect Scale [36], well-known method in psychology with a large set of words, each of them associated with one from eleven moods such as surprise, fear, guilt, etc. | ✓ | |
Emolex [37] | Builds a general sentiment Lexicon crowdsourcing supported. Each entry lists the association of a token with 8 basic sentiments: joy, sadness, anger, etc. defined by [38]. Proposed Lexicon includes unigrams and bigrams from Macquarie Thesaurus and also words from GI and WordNet. | ✓ | |
USent [39] | Infer additional reviews user ratings by performing sentiment analysis (SA) of user comments and integrating its output in a nearest neighbor (NN) model that provides multimedia recommendations over TED talks. | ✓ | ✓ |
Sentiment140 Lexicon [40] | A lexicon dictionary based on the same dataset used to train the Sentiment140 Method. The lexicon was built in a similar way to [33] but authors used the occurrence of emoticons to classify the tweet as positive or negative. Then, the n-gram score was calculated based on the frequency of occurrence in each class of tweets. | ✓ | |
SentiStrength [11] | Builds a lexicon dictionary annotated by humans and improved with the use of Machine Learning. | ✓ | ✓ |
Stanford Recursive Deep Model [41] | Proposes a model called Recursive Neural Tensor Network (RNTN) that processes all sentences dealing with their structures and compute the interactions between them. This approach is interesting since RNTN take into account the order of words in a sentence, which is ignored in most of methods. | ✓ | ✓ |
Umigon [18] | Disambiguates tweets using lexicon with heuristics to detect negations plus elongated words and hashtags evaluation. | ✓ | |
ANEW_SUB [42] | ✓ | ||
VADER [15] | It is a human-validated sentiment analysis method developed for Twitter and social media contexts. VADER was created from a generalizable, valence-based, human-curated gold standard sentiment lexicon. | ✓ | |
Semantria [44] | It is a paid tool that employs multi-level analysis of sentences. Basically it has four levels: part of speech, assignment of previous scores from dictionaries, application of intensifiers and finally machine learning techniques to delivery a final weight to the sentence. | ✓ | ✓ |
Name
|
Output
|
Validation
|
Compared to
|
Lexicon size
|
---|---|---|---|---|
Emoticons |
,
| - | - | 79 |
Opinion Lexicon | Provides polarities for lexicons | Product Reviews from Amazon and CNet | - | 6,787 |
Opinion Finder (MPQA) |
, Objective,
| MPQA [45] | Compared to itself in different versions | 20,611 |
SentiWordNet | Provides positive, negative and objective scores for each word (0.0 to 1.0) | - | General Inquirer (GI) [46] | 117,658 |
Sentiment140 |
, 2,
| Their own datasets - 359 tweets (Tweets_STF, presented at Table 3) | Naive Bayes, Maximum Entropy, and SVM classifiers as described in [6] | - |
LIWC15 |
,
| - | Their previous dictionary (2001) | 4,500 |
SenticNet |
,
| Patient Opinions (Unavailable) | SentiStrength [11] | 15,000 |
AFINN | Provides polarity score for lexicons (−5 to 5) | Twitter [47] | 2,477 | |
SO-CAL |
, 0,
| 9,928 | ||
Emoticons DS (Distant Supervision) | Provides polarity score for lexicons | Validation with unlabeled Twitter data [51] | - | 1,162,894 |
NRC Hashtag | Provides polarities for lexicons | Twitter (SemEval-2007 Affective Text Corpus) [52] | WordNet Affect [52] | 679,468 |
Pattern.en |
Objective,
,
| Product Reviews, but the source was not specified | - | 2,973 |
SASA [35] |
, Neutral, Unsure,
| ‘Political’ tweets labeled by ‘turkers’ (AMT) (unavailable) | - | - |
PANAS-t | Provides association for each word with eleven moods (joviality, attentiveness, fear, etc.) | Validation with unlabeled Twitter data [51] | - | 50 |
Emolex | Provides polarities for lexicons | - | Compared with existing gold standard data but it was not specified | 141,820 |
USent |
, neu,
| Their own dataset - TED talks | Comparison with other multimedia recommendation approaches | MPQA (8,226)/Their own (9,176) |
Sentiment140 Lexicon | Provides polarity scores for lexicon | Twitter and SMS from SemEval 2013, task 2 [53] | Other SemEval 2013, task 2 approaches | 1,220,176 |
SentiStrength |
, 0,
| Their own datasets - Twitter, Youtube, Digg, Myspace, BBC Forums and Runners World | The best of nine Machine Learning techniques for each test | 2,698 |
Stanford Recursive Deep Model |
,
, neutral,
,
| Movie Reviews [54] | Naive Bayes and SVM with bag of words features and bag of bigram features | 227,009 |
Umigon |
, Neutral,
| Twitter and SMS from SemEval 2013, task 2 [53] | [40] | 1,053 |
ANEW_WKB | Provides ratings for words in terms of Valence, Arousal and Dominance. Results can also be grouped by gender, age and education | - | Compared to similar works, including cross-language studies, by means of correlations between emotional dimensions | 13,915 |
VADER |
, (−0.05,…,0.05),
| Their own datasets - Twitter, Movie Reviews, Technical Product Reviews, NYT User’s Opinions | 7,517 | |
LIWC15 |
,
| - | Their previous dictionary (2007) | 6,400 |
Semantria |
, neutral,
| Not available | Not available | Not available |
3.1 Adapting lexicons for the sentence level task
3.2 Output adaptations
3.3 Paid softwares
3.4 Methods not included
3.5 Datasets and comparison among methods
4 Gold standard data
Dataset
|
Nomenclature
|
# Msgs
|
# Pos
|
# Neg
|
# Neu
|
Average # of phrases
|
Average # of words
|
Annotators expertise
|
# of annotators
|
CK
|
---|---|---|---|---|---|---|---|---|---|---|
Comments (BBC) [11] |
Comments_BBC
| 1,000 | 99 | 653 | 248 | 3.98 | 64.39 | Non expert | 3 | 0.427 |
Comments (Digg) [11] |
Comments_Digg
| 1,077 | 210 | 572 | 295 | 2.50 | 33.97 | Non expert | 3 | 0.607 |
Comments (NYT) [15] |
Comments_NYT
| 5,190 | 2,204 | 2,742 | 244 | 1.01 | 17.76 | AMT | 20 | 0.628 |
Comments (TED) [65] |
Comments_TED
| 839 | 318 | 409 | 112 | 1 | 16.95 | Non expert | 6 | 0.617 |
Comments (Youtube) [11] |
Comments_YTB
| 3,407 | 1,665 | 767 | 975 | 1.78 | 17.68 | Non expert | 3 | 0.724 |
Movie Reviews [54] |
Reviews_I
| 10,662 | 5,331 | 5,331 | - | 1.15 | 18.99 | User rating | - | 0.719 |
Movie Reviews [15] |
Reviews_II
| 10,605 | 5,242 | 5,326 | 37 | 1.12 | 19.33 | AMT | 20 | 0.555 |
Myspace posts [11] |
Myspace
| 1,041 | 702 | 132 | 207 | 2.22 | 21.12 | Non expert | 3 | 0.647 |
Product Reviews [15] |
Amazon
| 3,708 | 2,128 | 1,482 | 98 | 1.03 | 16.59 | AMT | 20 | 0.822 |
Tweets (debate) [66] |
Tweets_DBT
| 3,238 | 730 | 1,249 | 1,259 | 1.86 | 14.86 | AMT+expert | Undef. | 0.419 |
Tweets (random) [11] |
Tweets_RND_I
| 4,242 | 1,340 | 949 | 1,953 | 1.77 | 15.81 | Non expert | 3 | 0.683 |
Tweets (random) [15] |
Tweets_RND_II
| 4,200 | 2,897 | 1,299 | 4 | 1.87 | 14.10 | AMT | 20 | 0.800 |
Tweets (random) [67] |
Tweets_RND_III
| 3,771 | 739 | 488 | 2,536 | 1.54 | 14.32 | AMT | 3 | 0.824 |
Tweets (random) [68] |
Tweets_RND_IV
| 500 | 139 | 119 | 222 | 1.90 | 15.44 | Expert | Undef. | 0.643 |
Tweets (specific domains w/emot.) [27] |
Tweets_STF
| 359 | 182 | 177 | - | 1.0 | 15.1 | Non expert | Undef. | 1.000 |
Tweets (specific topics) [69] |
Tweets_SAN
| 3,737 | 580 | 654 | 2,503 | 1.60 | 15.03 | Expert | 1 | 0.404 |
Tweets (SemEval2013 task 2) [53] |
Tweets_Semeval
| 6,087 | 2,223 | 837 | 3,027 | 1.86 | 20.05 | AMT | 5 | 0.617 |
Runners World forum [11] |
RW
| 1,046 | 484 | 221 | 341 | 4.79 | 66.12 | Non expert | 3 | 0.615 |
5 Comparison results
5.1 Experimental details
5.2 Comparison metrics
Predicted
| ||||
---|---|---|---|---|
Positive
|
Neutral
|
Negative
| ||
Actual
| Positive |
a
|
b
|
c
|
Neutral |
d
|
e
|
f
| |
Negative |
g
|
h
|
i
|
Predicted
| |||
---|---|---|---|
Positive
|
Negative
| ||
Actual
| Positive |
a
|
b
|
Negative |
c
|
d
|
5.3 Comparing prediction performance
Dataset
|
Method
|
Accur.
|
Posit. sentiment
|
Negat. sentiment
|
Macro-
F
1
|
Coverage
| ||||
---|---|---|---|---|---|---|---|---|---|---|
P
|
R
|
F
1
|
P
|
R
|
F
1
| |||||
Tweets_RND_II
| AFINN | 96.37 | 97.66 | 96.94 | 97.30 | 93.75 | 95.19 | 94.47 | 95.88 | 80.77 |
ANEW_SUB | 81.36 | 80.52 | 96.38 | 87.74 | 85.44 | 47.64 | 61.17 | 74.45 | 93.35 | |
Emolex | 86.06 | 89.82 | 89.11 | 89.47 | 78.77 | 80.00 | 79.38 | 84.42 | 63.58 | |
Emoticons | 97.75 | 97.90 | 99.42 | 98.65 | 96.97 | 89.72 | 93.20 | 95.93 | 14.82 | |
Emoticons DS | 71.04 | 70.61 | 99.90 | 82.74 | 95.83 | 5.43 | 10.28 | 46.51 | 99.09 | |
NRC Hashtag | 67.37 | 83.76 | 65.43 | 73.47 | 48.17 | 71.69 | 57.62 | 65.55 | 91.94 | |
LIWC07 | 66.47 | 74.46 | 78.81 | 76.58 | 44.20 | 38.31 | 41.04 | 58.81 | 73.93 | |
LIWC15 | 96.44 | 97.09 | 98.04 | 97.56 | 94.68 | 92.23 | 93.44 | 95.50 | 77.05 | |
Opinion Finder | 78.32 | 93.86 | 71.11 | 80.92 | 63.42 | 91.50 | 74.92 | 77.92 | 41.23 | |
Opinion Lexicon | 93.45 | 97.03 | 93.14 | 95.04 | 86.93 | 94.11 | 90.38 | 92.71 | 70.64 | |
PANAS-t | 90.71 | 96.95 | 88.19 | 92.36 | 82.11 | 95.12 | 88.14 | 90.25 | 5.39 | |
Pattern.en | 91.76 | 92.94 | 96.19 | 94.54 | 87.86 | 79.06 | 83.23 | 88.88 | 70.85 | |
SASA | 70.06 | 82.81 | 72.81 | 77.49 | 49.05 | 63.39 | 55.30 | 66.40 | 63.04 | |
Semantria | 91.61 | 96.94 | 90.55 | 93.64 | 82.25 | 93.88 | 87.68 | 90.66 | 63.61 | |
SenticNet | 73.64 | 90.74 | 68.45 | 78.03 | 55.41 | 84.88 | 67.05 | 72.54 | 82.82 | |
Sentiment140 | 94.75 | 97.10 | 95.71 | 96.40 | 88.64 | 92.13 | 90.35 | 93.37 | 49.95 | |
Sentiment140_L | 78.05 | 88.68 | 78.31 | 83.17 | 61.32 | 77.47 | 68.45 | 75.81 | 93.28 | |
SentiStrength | 96.97 | 98.92 | 96.43 | 97.66 | 93.54 | 98.01 | 95.72 | 96.69 | 34.65 | |
SentiWordNet | 78.57 | 87.88 | 80.91 | 84.25 | 61.09 | 72.87 | 66.46 | 75.36 | 61.49 | |
SO-CAL | 87.76 | 94.25 | 86.99 | 90.47 | 77.34 | 89.32 | 82.90 | 86.68 | 67.18 | |
Stanford DM | 60.46 | 94.48 | 44.87 | 60.84 | 44.06 | 94.30 | 60.06 | 60.45 | 88.89 | |
Umigon | 88.63 | 97.73 | 85.92 | 91.45 | 73.64 | 95.17 | 83.03 | 87.24 | 70.83 | |
USent | 84.46 | 89.28 | 87.67 | 88.47 | 74.77 | 77.63 | 76.17 | 82.32 | 38.94 | |
VADER | 99.04 | 99.16 | 99.45 | 99.31 | 98.77 | 98.12 | 98.45 | 98.88 | 94.40 | |
Tweets_STF
| AFINN | 84.42 | 80.62 | 91.49 | 85.71 | 89.66 | 77.04 | 82.87 | 84.29 | 76.88 |
ANEW_SUB | 68.05 | 63.08 | 93.18 | 75.23 | 84.62 | 40.74 | 55.00 | 65.11 | 94.15 | |
Emolex | 79.65 | 76.09 | 88.98 | 82.03 | 85.23 | 69.44 | 76.53 | 79.28 | 62.95 | |
Emoticons | 85.42 | 80.65 | 96.15 | 87.72 | 94.12 | 72.73 | 82.05 | 84.89 | 13.37 | |
Emoticons DS | 51.96 | 51.41 | 100.00 | 67.91 | 100.00 | 2.27 | 4.44 | 36.18 | 99.72 | |
NRC Hashtag | 71.30 | 73.05 | 70.93 | 71.98 | 69.51 | 71.70 | 70.59 | 71.28 | 92.20 | |
LIWC07 | 64.29 | 63.75 | 76.12 | 69.39 | 65.22 | 50.85 | 57.14 | 63.27 | 70.39 | |
LIWC15 | 89.22 | 84.18 | 97.08 | 90.17 | 96.40 | 81.06 | 88.07 | 89.12 | 74.93 | |
Opinion Finder | 80.77 | 81.16 | 76.71 | 78.87 | 80.46 | 84.34 | 82.35 | 80.61 | 43.45 | |
Opinion Lexicon | 86.10 | 83.67 | 91.11 | 87.23 | 89.29 | 80.65 | 84.75 | 85.99 | 72.14 | |
PANAS-t | 94.12 | 88.89 | 100.00 | 94.12 | 100.00 | 88.89 | 94.12 | 94.12 | 4.74 | |
Pattern.en | 79.55 | 74.86 | 94.48 | 83.54 | 90.12 | 61.34 | 73.00 | 78.27 | 73.54 | |
SASA | 68.52 | 65.65 | 78.90 | 71.67 | 72.94 | 57.94 | 64.58 | 68.12 | 60.17 | |
Semantria | 88.45 | 89.15 | 88.46 | 88.80 | 87.70 | 88.43 | 88.07 | 88.43 | 69.92 | |
SenticNet | 70.49 | 71.31 | 63.50 | 67.18 | 69.88 | 76.82 | 73.19 | 70.18 | 80.22 | |
Sentiment140 | 93.29 | 91.36 | 94.87 | 93.08 | 95.18 | 91.86 | 93.49 | 93.29 | 45.68 | |
Sentiment140_L | 79.12 | 81.48 | 76.30 | 78.81 | 76.97 | 82.04 | 79.42 | 79.11 | 94.71 | |
SentiStrength | 95.33 | 95.18 | 96.34 | 95.76 | 95.52 | 94.12 | 94.81 | 95.29 | 41.78 | |
SentiWordNet | 72.99 | 73.17 | 78.95 | 75.95 | 72.73 | 65.98 | 69.19 | 72.57 | 58.77 | |
SO-CAL | 87.36 | 82.89 | 93.33 | 87.80 | 92.80 | 81.69 | 86.89 | 87.35 | 77.16 | |
Stanford DM | 66.56 | 87.69 | 36.31 | 51.35 | 61.24 | 95.18 | 74.53 | 62.94 | 89.97 | |
Umigon | 86.99 | 91.73 | 81.88 | 86.52 | 83.02 | 92.31 | 87.42 | 86.97 | 81.34 | |
USent | 73.21 | 69.35 | 82.69 | 75.44 | 78.82 | 63.81 | 70.53 | 72.98 | 58.22 | |
VADER | 84.44 | 80.23 | 92.21 | 85.80 | 90.40 | 76.35 | 82.78 | 84.29 | 84.12 | |
Comments_Digg
| AFINN | 70.94 | 47.01 | 81.82 | 59.72 | 91.17 | 67.05 | 77.27 | 68.49 | 74.81 |
ANEW_SUB | 43.25 | 30.98 | 92.31 | 46.39 | 90.13 | 25.46 | 39.71 | 43.05 | 93.73 | |
Emolex | 61.71 | 34.60 | 75.83 | 47.52 | 88.93 | 57.53 | 69.87 | 58.69 | 67.14 | |
Emoticons | 73.08 | 72.22 | 86.67 | 78.79 | 75.00 | 54.55 | 63.16 | 70.97 | 3.32 | |
Emoticons DS | 28.24 | 27.30 | 100.00 | 42.89 | 100.00 | 1.77 | 3.48 | 23.19 | 98.72 | |
NRC Hashtag | 74.69 | 51.01 | 40.64 | 45.24 | 80.80 | 86.48 | 83.54 | 64.39 | 92.97 | |
LIWC07 | 46.15 | 27.44 | 58.40 | 37.34 | 72.49 | 41.52 | 52.79 | 45.07 | 58.18 | |
LIWC15 | 70.67 | 49.81 | 90.91 | 64.36 | 94.35 | 62.36 | 75.09 | 69.72 | 62.79 | |
Opinion Finder | 71.14 | 43.04 | 64.76 | 51.71 | 86.88 | 73.13 | 79.42 | 65.56 | 56.27 | |
Opinion Lexicon | 71.82 | 47.45 | 86.43 | 61.27 | 93.40 | 66.75 | 77.86 | 69.56 | 69.44 | |
PANAS-t | 68.00 | 12.50 | 50.00 | 20.00 | 94.12 | 69.57 | 80.00 | 50.00 | 3.20 | |
Pattern.en | 60.05 | 43.73 | 92.14 | 59.31 | 92.57 | 45.21 | 60.75 | 60.03 | 56.65 | |
SASA | 65.54 | 40.26 | 66.91 | 50.27 | 84.82 | 65.06 | 73.64 | 61.95 | 68.29 | |
Semantria | 82.46 | 62.72 | 88.33 | 73.36 | 94.81 | 80.25 | 86.93 | 80.14 | 56.14 | |
SenticNet | 69.40 | 46.30 | 72.46 | 56.50 | 86.77 | 68.25 | 76.40 | 66.45 | 96.55 | |
Sentiment140 | 85.06 | 62.50 | 78.95 | 69.77 | 93.65 | 86.76 | 90.08 | 79.92 | 33.38 | |
Sentiment140_L | 67.76 | 42.07 | 73.45 | 53.50 | 88.01 | 65.84 | 75.33 | 64.41 | 89.64 | |
SentiStrength | 92.09 | 78.69 | 92.31 | 84.96 | 97.40 | 92.02 | 94.64 | 89.80 | 27.49 | |
SentiWordNet | 62.17 | 36.86 | 77.68 | 50.00 | 88.84 | 57.18 | 69.58 | 59.79 | 58.82 | |
SO-CAL | 76.55 | 52.86 | 77.08 | 62.71 | 90.65 | 76.37 | 82.90 | 72.81 | 71.99 | |
Stanford DM | 69.16 | 35.29 | 20.27 | 25.75 | 75.21 | 86.68 | 80.54 | 53.15 | 78.90 | |
Umigon | 83.37 | 66.22 | 75.38 | 70.50 | 90.72 | 86.23 | 88.42 | 79.46 | 63.04 | |
USent | 55.98 | 36.06 | 80.65 | 49.83 | 86.67 | 46.80 | 60.78 | 55.31 | 43.86 | |
VADER | 69.05 | 45.48 | 85.88 | 59.47 | 92.55 | 63.00 | 74.97 | 67.22 | 82.23 | |
Comments_BBC
| AFINN | 66.56 | 23.08 | 81.08 | 35.93 | 96.32 | 64.66 | 77.38 | 56.65 | 85.11 |
ANEW_SUB | 31.37 | 15.48 | 95.79 | 26.65 | 97.18 | 21.73 | 35.52 | 31.08 | 97.07 | |
Emolex | 59.64 | 21.52 | 89.04 | 34.67 | 97.38 | 55.62 | 70.80 | 52.73 | 80.72 | |
Emoticons | 33.33 | 0.00 | 0.00 | 0.00 | 100.00 | 33.33 | 50.00 | 25.00 | 0.40 | |
Emoticons DS | 13.33 | 13.10 | 100.00 | 23.17 | 100.00 | 0.31 | 0.61 | 11.89 | 99.73 | |
NRC Hashtag | 84.45 | 33.33 | 25.27 | 28.75 | 89.76 | 92.83 | 91.27 | 60.01 | 97.47 | |
LIWC07 | 50.10 | 15.38 | 58.33 | 24.35 | 88.00 | 48.78 | 62.77 | 43.56 | 69.55 | |
LIWC15 | 63.21 | 25.86 | 90.67 | 40.24 | 97.55 | 58.86 | 73.42 | 56.83 | 73.01 | |
Opinion Finder | 74.43 | 21.74 | 62.50 | 32.26 | 94.93 | 75.72 | 84.24 | 58.25 | 76.46 | |
Opinion Lexicon | 74.14 | 29.81 | 84.93 | 44.13 | 97.24 | 72.66 | 83.17 | 63.65 | 80.72 | |
PANAS-t | 58.73 | 20.00 | 75.00 | 31.58 | 93.94 | 56.36 | 70.45 | 51.02 | 8.38 | |
Pattern.en | 41.75 | 19.73 | 93.55 | 32.58 | 96.61 | 32.57 | 48.72 | 40.65 | 54.79 | |
SASA | 61.61 | 23.50 | 66.20 | 34.69 | 90.80 | 60.77 | 72.81 | 53.75 | 61.30 | |
Semantria | 83.43 | 40.00 | 84.75 | 54.35 | 97.64 | 83.26 | 89.88 | 72.11 | 67.42 | |
SenticNet | 66.07 | 24.44 | 74.16 | 36.77 | 94.24 | 64.83 | 76.81 | 56.79 | 88.96 | |
Sentiment140 | 68.51 | 24.00 | 69.77 | 35.71 | 94.04 | 68.33 | 79.15 | 57.43 | 45.61 | |
Sentiment140_L | 56.85 | 18.52 | 69.15 | 29.21 | 92.35 | 55.03 | 68.97 | 49.09 | 97.07 | |
SentiStrength | 93.93 | 64.29 | 78.26 | 70.59 | 97.72 | 95.54 | 96.61 | 83.60 | 32.85 | |
SentiWordNet | 57.49 | 20.00 | 88.06 | 32.60 | 97.13 | 53.45 | 68.96 | 50.78 | 76.33 | |
SO-CAL | 75.28 | 28.93 | 80.28 | 42.54 | 96.71 | 74.64 | 84.25 | 63.40 | 82.85 | |
Stanford DM | 89.45 | 63.16 | 40.91 | 49.66 | 91.81 | 96.52 | 94.11 | 71.88 | 92.02 | |
Umigon | 79.37 | 39.13 | 61.02 | 47.68 | 92.10 | 82.72 | 87.15 | 67.42 | 50.93 | |
USent | 52.60 | 18.33 | 80.49 | 29.86 | 94.56 | 48.60 | 64.20 | 47.03 | 43.48 | |
VADER | 62.76 | 22.68 | 85.54 | 35.86 | 96.75 | 59.60 | 73.76 | 54.81 | 90.69 |
Dataset
|
Method
|
Accur.
|
Posit. sentiment
|
Negat. sentiment
|
Neut. sentiment
|
Macro-
F
1
| ||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
P
|
R
|
F
1
|
P
|
R
|
F
1
|
P
|
R
|
F
1
| ||||
Tweets_Semeval
| AFINN | 62.36 | 61.10 | 70.09 | 65.28 | 44.08 | 55.56 | 49.15 | 71.43 | 58.57 | 64.37 | 59.60 |
ANEW_SUB | 39.51 | 38.79 | 96.31 | 55.31 | 43.50 | 23.18 | 30.24 | 57.38 | 2.31 | 4.45 | 30.00 | |
Emolex | 48.74 | 48.15 | 62.71 | 54.47 | 31.27 | 38.59 | 34.55 | 57.90 | 41.30 | 48.21 | 45.74 | |
Emoticons | 52.88 | 72.83 | 11.34 | 19.62 | 55.56 | 5.38 | 9.80 | 34.05 | 96.53 | 50.34 | 26.59 | |
Emoticons DS | 36.59 | 36.55 | 100.00 | 53.53 | 75.00 | 0.36 | 0.71 | 100.00 | 0.03 | 0.07 | 18.10 | |
NRC Hashtag | 36.95 | 42.04 | 75.03 | 53.88 | 24.57 | 56.03 | 34.16 | 53.33 | 3.70 | 6.92 | 31.65 | |
LIWC07 | 39.54 | 36.52 | 42.33 | 39.21 | 15.14 | 13.02 | 14.00 | 48.64 | 44.83 | 46.66 | 33.29 | |
LIWC15 | 62.56 | 59.77 | 71.03 | 64.91 | 49.04 | 42.65 | 45.62 | 68.90 | 61.84 | 65.18 | 58.57 | |
Opinion Finder | 57.63 | 67.57 | 27.94 | 39.53 | 40.75 | 33.69 | 36.89 | 58.20 | 86.06 | 69.44 | 48.62 | |
Opinion Lexicon | 60.37 | 62.09 | 62.71 | 62.40 | 41.19 | 52.81 | 46.28 | 66.41 | 60.75 | 63.46 | 57.38 | |
PANAS-t | 53.08 | 90.95 | 9.04 | 16.45 | 51.56 | 3.94 | 7.33 | 51.65 | 99.01 | 67.89 | 30.55 | |
Pattern.en | 57.99 | 57.97 | 68.74 | 62.89 | 34.83 | 35.24 | 35.04 | 65.55 | 56.39 | 60.63 | 52.85 | |
SASA | 50.63 | 46.34 | 47.77 | 47.04 | 33.07 | 20.31 | 25.17 | 56.39 | 61.12 | 58.66 | 43.62 | |
Semantria | 61.54 | 67.28 | 57.35 | 61.92 | 39.57 | 52.81 | 45.24 | 65.98 | 67.03 | 66.50 | 57.89 | |
SenticNet | 49.68 | 51.85 | 1.26 | 2.46 | 29.79 | 1.67 | 3.17 | 49.82 | 98.51 | 66.17 | 23.93 | |
Sentiment140 | 60.42 | 63.87 | 51.37 | 56.94 | 50.96 | 37.87 | 43.45 | 60.35 | 73.31 | 66.20 | 55.53 | |
Sentiment140_L | 39.44 | 43.52 | 74.72 | 55.00 | 27.67 | 65.35 | 38.88 | 65.87 | 6.38 | 11.63 | 35.17 | |
SentiStrength | 57.83 | 78.01 | 27.13 | 40.25 | 47.80 | 23.42 | 31.44 | 55.49 | 89.89 | 68.62 | 46.77 | |
SentiWordNet | 48.33 | 55.54 | 53.44 | 54.47 | 19.67 | 37.51 | 25.81 | 61.22 | 47.57 | 53.54 | 44.61 | |
SO-CAL | 58.83 | 58.89 | 59.02 | 58.95 | 40.39 | 54.24 | 46.30 | 39.89 | 59.96 | 47.91 | 51.05 | |
Stanford DM | 22.54 | 72.14 | 18.17 | 29.03 | 14.92 | 90.56 | 25.61 | 47.19 | 6.94 | 12.10 | 22.25 | |
Umigon | 65.88 | 75.18 | 56.14 | 64.28 | 39.66 | 55.91 | 46.41 | 70.65 | 75.78 | 73.13 | 61.27 | |
USent | 52.13 | 49.86 | 32.88 | 39.63 | 39.96 | 22.82 | 29.05 | 54.33 | 74.36 | 62.79 | 43.82 | |
VADER | 60.21 | 56.46 | 79.04 | 65.87 | 44.30 | 59.02 | 50.61 | 76.02 | 46.71 | 57.87 | 58.12 | |
Tweets_RND_III
| AFINN | 64.41 | 40.81 | 72.12 | 52.13 | 49.67 | 62.50 | 55.35 | 85.95 | 62.54 | 72.40 | 59.96 |
ANEW_SUB | 28.03 | 21.89 | 92.29 | 35.38 | 44.30 | 34.22 | 38.61 | 74.82 | 8.18 | 14.74 | 29.58 | |
Emolex | 54.76 | 31.67 | 59.95 | 41.44 | 40.14 | 47.54 | 43.53 | 77.48 | 54.64 | 64.08 | 49.68 | |
Emoticons | 70.22 | 70.06 | 16.78 | 27.07 | 65.62 | 8.61 | 15.22 | 41.29 | 97.56 | 58.02 | 33.44 | |
Emoticons DS | 20.34 | 19.78 | 99.46 | 33.00 | 62.07 | 3.69 | 6.96 | 53.85 | 0.55 | 1.09 | 13.68 | |
NRC Hashtag | 30.47 | 28.25 | 77.40 | 41.39 | 24.18 | 72.54 | 36.27 | 79.08 | 8.77 | 15.78 | 31.15 | |
LIWC | 46.88 | 21.85 | 38.43 | 27.86 | 19.18 | 18.24 | 18.70 | 69.51 | 54.83 | 61.31 | 35.95 | |
LIWC15 | 67.75 | 44.78 | 78.35 | 56.99 | 57.49 | 57.38 | 57.44 | 85.18 | 66.67 | 74.80 | 63.07 | |
Opinion Finder | 71.55 | 57.48 | 32.75 | 41.72 | 49.85 | 34.63 | 40.87 | 75.95 | 89.90 | 82.34 | 54.98 | |
Opinion Lexicon | 63.86 | 40.65 | 66.17 | 50.36 | 48.84 | 56.15 | 52.24 | 81.96 | 64.66 | 72.29 | 58.30 | |
PANAS-t | 68.79 | 79.49 | 8.39 | 15.18 | 48.57 | 3.48 | 6.50 | 68.75 | 98.86 | 81.10 | 34.26 | |
Pattern.en | 59.56 | 36.20 | 77.00 | 49.24 | 52.87 | 45.29 | 48.79 | 81.75 | 57.23 | 67.33 | 55.12 | |
SASA | 55.37 | 29.42 | 54.53 | 38.22 | 42.46 | 47.34 | 44.77 | 78.30 | 57.15 | 66.08 | 49.69 | |
Semantria | 68.89 | 48.86 | 63.73 | 55.31 | 49.82 | 55.53 | 52.52 | 82.02 | 72.96 | 77.22 | 61.68 | |
SenticNet | 29.97 | 31.08 | 74.83 | 43.92 | 20.98 | 73.98 | 32.68 | 79.70 | 8.49 | 15.35 | 30.65 | |
Sentiment140 | 76.40 | 64.42 | 51.69 | 57.36 | 74.75 | 45.49 | 56.56 | 79.04 | 89.50 | 83.94 | 65.95 | |
Sentiment140_L | 31.32 | 25.83 | 77.13 | 38.70 | 30.05 | 78.69 | 43.49 | 79.37 | 8.92 | 16.04 | 32.74 | |
SentiStrength | 73.80 | 70.94 | 41.95 | 52.72 | 57.53 | 25.82 | 35.64 | 75.35 | 92.26 | 82.95 | 57.10 | |
SentiWordNet | 55.85 | 37.42 | 58.19 | 45.55 | 24.04 | 35.86 | 28.78 | 79.25 | 59.00 | 67.64 | 47.33 | |
SO-CAL | 66.51 | 43.06 | 68.88 | 52.99 | 51.84 | 60.66 | 55.90 | 45.77 | 66.94 | 54.37 | 54.42 | |
Stanford DM | 31.90 | 64.48 | 38.57 | 48.26 | 15.58 | 85.04 | 26.33 | 75.64 | 19.77 | 31.35 | 35.32 | |
Umigon | 74.12 | 57.67 | 70.23 | 63.33 | 48.83 | 68.44 | 57.00 | 88.80 | 76.34 | 82.10 | 67.47 | |
USent | 66.06 | 40.60 | 36.81 | 38.61 | 44.87 | 28.69 | 35.00 | 74.54 | 81.72 | 77.97 | 50.53 | |
VADER | 60.14 | 37.69 | 81.60 | 51.56 | 48.56 | 65.57 | 55.80 | 88.96 | 52.87 | 66.32 | 57.89 | |
Comments_BBC
| AFINN | 50.10 | 16.22 | 60.61 | 25.59 | 82.62 | 56.05 | 66.79 | 40.11 | 30.24 | 34.48 | 42.29 |
ANEW_SUB | 24.30 | 11.38 | 91.92 | 20.24 | 84.15 | 21.13 | 33.78 | 38.89 | 5.65 | 9.86 | 21.30 | |
Emolex | 44.10 | 15.51 | 65.66 | 25.10 | 83.19 | 45.48 | 58.81 | 35.27 | 31.85 | 33.47 | 39.13 | |
Emoticons | 24.60 | 0.00 | 0.00 | 0.00 | 33.33 | 0.15 | 0.30 | 19.77 | 98.79 | 32.95 | 11.09 | |
Emoticons DS | 10.00 | 9.85 | 98.99 | 17.92 | 66.67 | 0.31 | 0.61 | 0.00 | 0.00 | 0.00 | 6.18 | |
NRC Hashtag | 64.00 | 20.72 | 23.23 | 21.90 | 70.20 | 91.27 | 79.36 | 52.50 | 8.47 | 14.58 | 38.62 | |
LIWC07 | 33.00 | 11.11 | 42.42 | 17.61 | 67.69 | 33.69 | 44.99 | 22.90 | 27.42 | 24.95 | 29.18 | |
LIWC15 | 43.70 | 17.94 | 68.69 | 28.45 | 85.06 | 42.73 | 56.88 | 30.72 | 36.29 | 33.27 | 39.53 | |
Opinion Finder | 51.80 | 14.96 | 35.35 | 21.02 | 78.76 | 60.18 | 68.23 | 33.71 | 36.29 | 34.95 | 41.40 | |
Opinion Lexicon | 55.00 | 20.67 | 62.63 | 31.08 | 85.27 | 59.42 | 70.04 | 40.82 | 40.32 | 40.57 | 47.23 | |
PANAS-t | 27.10 | 16.67 | 6.06 | 8.89 | 75.61 | 4.75 | 8.93 | 25.35 | 94.35 | 39.97 | 19.26 | |
Pattern.en | 28.70 | 14.25 | 58.59 | 22.92 | 82.61 | 17.46 | 28.82 | 25.27 | 46.37 | 32.72 | 28.16 | |
SASA | 38.20 | 17.03 | 47.47 | 25.07 | 70.75 | 36.29 | 47.98 | 25.19 | 39.52 | 30.77 | 34.60 | |
Semantria | 56.00 | 28.90 | 50.51 | 36.76 | 83.82 | 57.12 | 67.94 | 35.86 | 55.24 | 43.49 | 49.40 | |
SenticNet | 47.10 | 17.74 | 66.67 | 28.03 | 72.87 | 57.58 | 64.33 | 25.89 | 11.69 | 16.11 | 36.16 | |
Sentiment140 | 40.00 | 17.75 | 30.30 | 22.39 | 79.77 | 31.39 | 45.05 | 28.75 | 66.53 | 40.15 | 35.86 | |
Sentiment140_L | 43.10 | 13.32 | 65.66 | 22.15 | 73.84 | 53.60 | 62.11 | 42.11 | 6.45 | 11.19 | 31.82 | |
SentiStrength | 44.20 | 47.37 | 18.18 | 26.28 | 86.64 | 32.77 | 47.56 | 29.37 | 84.68 | 43.61 | 39.15 | |
SentiWordNet | 42.40 | 14.90 | 59.60 | 23.84 | 81.63 | 41.50 | 55.03 | 34.56 | 37.90 | 36.15 | 38.34 | |
SO-CAL | 55.50 | 20.88 | 57.58 | 30.65 | 80.47 | 63.09 | 70.73 | 28.57 | 34.68 | 31.33 | 44.23 | |
Stanford DM | 65.50 | 43.37 | 36.36 | 39.56 | 71.01 | 89.28 | 79.10 | 37.50 | 14.52 | 20.93 | 46.53 | |
Umigon | 45.70 | 28.35 | 36.36 | 31.86 | 76.35 | 41.04 | 53.39 | 29.31 | 61.69 | 39.74 | 41.66 | |
USent | 33.80 | 13.75 | 33.33 | 19.47 | 82.25 | 21.29 | 33.82 | 28.09 | 66.94 | 39.57 | 30.95 | |
VADER | 49.40 | 16.36 | 71.72 | 26.64 | 83.02 | 54.67 | 65.93 | 48.53 | 26.61 | 34.38 | 42.31 | |
Comments_NYT
| AFINN | 42.45 | 64.81 | 41.79 | 50.81 | 80.29 | 39.82 | 53.24 | 7.89 | 77.87 | 14.32 | 39.46 |
ANEW_SUB | 51.12 | 48.35 | 88.57 | 62.55 | 79.65 | 24.69 | 37.69 | 7.92 | 9.84 | 8.78 | 36.34 | |
Emolex | 42.97 | 55.12 | 53.72 | 54.41 | 75.35 | 33.33 | 46.22 | 7.22 | 54.10 | 12.74 | 37.79 | |
Emoticons | 4.68 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 4.47 | 99.59 | 8.56 | 2.85 | |
Emoticons DS | 42.58 | 42.55 | 99.77 | 59.66 | 78.57 | 0.40 | 0.80 | 0.00 | 0.00 | 0.00 | 20.15 | |
NRC Hashtag | 54.84 | 55.38 | 45.74 | 50.10 | 61.55 | 65.68 | 63.55 | 8.33 | 15.16 | 10.76 | 41.47 | |
LIWC07 | 24.35 | 42.88 | 27.72 | 33.67 | 53.42 | 19.07 | 28.11 | 4.67 | 53.28 | 8.58 | 23.45 | |
LIWC15 | 36.49 | 65.29 | 40.29 | 49.83 | 81.50 | 29.25 | 43.05 | 7.17 | 83.61 | 13.20 | 35.36 | |
Opinion Finder | 29.38 | 68.77 | 18.78 | 29.51 | 76.52 | 32.68 | 45.80 | 6.29 | 88.11 | 11.75 | 29.02 | |
Opinion Lexicon | 44.57 | 65.95 | 43.15 | 52.17 | 79.81 | 43.11 | 55.98 | 7.94 | 73.77 | 14.34 | 40.83 | |
PANAS-t | 5.88 | 69.23 | 1.23 | 2.41 | 62.07 | 1.31 | 2.57 | 4.75 | 99.18 | 9.07 | 4.68 | |
Pattern.en | 31.60 | 55.23 | 45.05 | 49.63 | 72.80 | 17.76 | 28.55 | 5.88 | 65.57 | 10.79 | 29.66 | |
SASA | 30.04 | 49.92 | 30.13 | 37.58 | 59.11 | 27.21 | 37.26 | 5.74 | 61.07 | 10.49 | 28.44 | |
Semantria | 44.59 | 70.60 | 41.83 | 52.54 | 80.54 | 44.24 | 57.11 | 7.53 | 73.36 | 13.65 | 41.10 | |
SenticNet | 61.85 | 58.19 | 59.48 | 58.83 | 65.01 | 69.26 | 67.07 | 0.00 | 0.00 | 0.00 | 41.97 | |
Sentiment140 | 13.58 | 77.32 | 6.81 | 12.51 | 75.40 | 11.96 | 20.65 | 4.98 | 93.03 | 9.45 | 14.20 | |
Sentiment140_L | 54.61 | 54.72 | 59.12 | 56.84 | 67.00 | 54.41 | 60.05 | 6.70 | 15.98 | 9.44 | 42.11 | |
SentiStrength | 18.17 | 78.51 | 8.62 | 15.54 | 81.12 | 18.96 | 30.74 | 5.41 | 95.49 | 10.24 | 18.84 | |
SentiWordNet | 32.20 | 57.35 | 34.53 | 43.10 | 70.31 | 26.95 | 38.97 | 6.08 | 70.08 | 11.19 | 31.09 | |
SO-CAL | 50.79 | 64.36 | 51.13 | 56.99 | 77.25 | 49.16 | 60.08 | 8.68 | 65.98 | 15.34 | 44.14 | |
Stanford DM | 51.93 | 73.39 | 21.14 | 32.83 | 59.48 | 77.90 | 67.46 | 9.65 | 38.11 | 15.40 | 38.56 | |
Umigon | 24.08 | 68.76 | 16.38 | 26.46 | 68.78 | 24.51 | 36.14 | 5.88 | 88.93 | 11.04 | 24.54 | |
USent | 27.44 | 56.61 | 28.95 | 38.31 | 77.69 | 21.59 | 33.79 | 5.88 | 79.51 | 10.94 | 27.68 | |
VADER | 48.03 | 62.67 | 51.63 | 56.62 | 79.91 | 43.07 | 55.97 | 9.18 | 71.31 | 16.26 | 42.95 |
3-classes
|
2-classes
| |||||
---|---|---|---|---|---|---|
Pos
|
Method
|
Mean Rank
|
Pos
|
Method
|
Mean Rank
|
Coverage (%)
|
1 | VADER | 4.00 (4.17) | 1 | SentiStrength | 2.33 (3.00) | 29.30 (28.91) |
2 | LIWC15 | 4.62 | 2 | Sentiment140 | 3.44 | 39.29 |
3 | AFINN | 4.69 | 3 | Semantria | 4.61 | 62.34 |
4 | Opinion Lexicon | 5.00 | 4 | Opinion Lexicon | 6.72 | 69.50 |
5 | Semantria | 5.31 | 5 | LIWC15 | 7.33 | 68.28 |
6 | Umigon | 5.77 | 6 | SO-CAL | 7.61 | 72.64 |
7 | SO-CAL | 7.23 | 7 | AFINN | 8.11 | 73.05 |
8 | Pattern.en | 9.92 | 8 | VADER | 9.17 (9.79) | 82.20 (83.18) |
9 | Sentiment140 | 10.92 | 9 | Umigon | 9.39 | 64.11 |
10 | Emolex | 11.38 | 10 | PANAS-t | 10.17 | 5.10 |
11 | Opinion Finder | 13.08 | 11 | Emoticons | 10.39 | 10.69 |
12 | SentiWordNet | 13.38 | 12 | Pattern.en | 12.61 | 65.02 |
13 | Sentiment140_L | 13.54 | 13 | SenticNet | 13.61 | 84.00 |
14 | SenticNet | 13.62 | 14 | Emolex | 14.50 | 66.12 |
15 | SentiStrength | 13.69 (13.71) | 15 | Opinion Finder | 14.72 | 46.63 |
16 | SASA | 14.77 | 16 | USent | 14.89 | 44.00 |
17 | Stanford DM | 15.85 | 17 | Sentiment140_L | 14.94 | 93.36 |
18 | USent | 15.92 | 18 | NRC Hashtag | 17.17 | 93.52 |
19 | NRC Hashtag | 16.31 | 19 | Stanford DM | 17.39 | 87.32 |
20 | LIWC | 16.46 | 20 | SentiWordNet | 17.50 | 61.77 |
21 | ANEW_SUB | 18.54 | 21 | SASA | 18.94 | 60.12 |
22 | Emoticons | 21.00 | 22 | LIWC | 19.67 | 61.82 |
23 | PANAS-t | 21.77 | 23 | ANEW_SUB | 21.17 | 94.20 |
24 | Emoticons DS | 23.23 | 24 | Emoticons DS | 23.61 | 99.36 |
2-class experiments
|
3-class experiments
| ||
---|---|---|---|
FR | 275.59 | FR | 197.52 |
Critical value | 35.17 | Critical value | 35.17 |
Reject null hypothesis
|
Reject null hypothesis
|
Dataset
|
Method
|
F
1-Pos
|
F
1-Neg
|
Macro-
F
1
|
Coverage
|
---|---|---|---|---|---|
Comments_BBC | SentiStrength | 70.59 | 96.61 | 83.60 | 32.85 |
Comments_Digg | SentiStrength | 84.96 | 94.64 | 89.80 | 27.49 |
Comments_NYT | SentiStrength | 70.11 | 86.52 | 78.32 | 17.63 |
Comments_TED | Emoticons | 85.71 | 94.12 | 89.92 | 1.65 |
Comments_YTB | SentiStrength | 96.94 | 89.62 | 93.28 | 38.24 |
Reviews_I | SenticNet | 97.39 | 93.66 | 95.52 | 69.41 |
Reviews_II | SenticNet | 94.15 | 93.87 | 94.01 | 94.25 |
Myspace | SentiStrength | 98.73 | 88.46 | 93.6 | 31.53 |
Amazon | SentiStrength | 93.85 | 79.38 | 86.62 | 19.58 |
Tweets_DBT | Sentiment140 | 72.86 | 83.55 | 78.2 | 18.75 |
Tweets_RND_I | SentiStrength | 95.28 | 90.6 | 92.94 | 27.13 |
Tweets_RND_II | VADER | 99.31 | 98.45 | 98.88 | 94.4 |
Tweets_RND_III | Sentiment140 | 97.57 | 95.9 | 96.73 | 50.77 |
Tweets_RND_IV | Emoticons | 94.74 | 86.76 | 88.6 | 58.27 |
Tweets_STF | SentiStrength | 95.76 | 94.81 | 95.29 | 41.78 |
Tweets_SAN | SentiStrength | 90.23 | 88.59 | 89.41 | 29.61 |
Tweets_Semeval | SentiStrength | 93.93 | 83.4 | 88.66 | 28.66 |
RW | SentiStrength | 90.04 | 75.79 | 82.92 | 23.12 |
Dataset
|
Method
|
F
1-Pos
|
F
1-Neg
|
F
1-Neu
|
Macro-
F
1
|
---|---|---|---|---|---|
Comments_BBC | Semantria | 36.76 | 67.94 | 43.49 | 49.40 |
Comments_Digg | Umigon | 49.62 | 62.04 | 44.27 | 51.98 |
Comments_NYT | SO-CAL | 56.99 | 60.08 | 15.34 | 44.14 |
Comments_TED | Opinion Lexicon | 64.95 | 56.59 | 30.77 | 50.77 |
Comments_YTB | LIWC15 | 73.68 | 49.72 | 48.79 | 57.4 |
Myspace | LIWC15 | 78.83 | 41.74 | 43.76 | 54.78 |
Tweets_DBT | Opinion Lexicon | 43.44 | 47.71 | 48.84 | 46.66 |
Tweets_RND_I | Umigon | 60.53 | 51.39 | 65.22 | 59.05 |
Tweets_RND_III | Umigon | 63.33 | 57.00 | 82.10 | 67.47 |
Tweets_RND_IV | Umigon | 75.86 | 76.33 | 71.54 | 74.58 |
Tweets_SAN | Umigon | 44.16 | 45.95 | 70.45 | 53.52 |
Tweets_Semeval | Umigon | 64.28 | 46.41 | 73.13 | 61.27 |
RW | Sentiment140 | 62.24 | 51.17 | 42.66 | 52.02 |
Context groups
| |
---|---|
Social Networks | Myspace, Tweets_DBT, Tweets_RND_I, Tweets_RND_ II, Tweets_RND_III, Tweets_RND_IV, Tweets_STF, Tweets_SAN, Tweets_Semeval |
Comments | Comments_BBC, Comments_DIGG, Comments_NYT, Comments_ TED, Comments_YTB, RW |
Reviews | Reviews_I, Reviews_I, Amazon |
3-classes
|
2-classes
| |||||
---|---|---|---|---|---|---|
Pos
|
Method
|
Mean Rank
|
Pos
|
Method
|
Mean Rank
|
Coverage (%)
|
1 | Umigon | 2.57 | 1 | SentiStrength | 2.22 (2.57) | 31.54 (32.18) |
2 | LIWC15 | 3.29 | 2 | Sentiment140 | 3.00 | 46.98 |
3 | VADER | 4.57 (4.57) | 3 | Emoticons | 5.11 | 18.04 |
4 | AFINN | 5.00 | 4 | LIWC15 | 5.67 | 71.73 |
5 | Opinion Lexicon | 5.57 | 5 | Semantria | 5.89 | 61.98 |
6 | Semantria | 6.00 | 6 | PANAS-t | 6.33 | 5.87 |
7 | Sentiment140 | 7.00 | 7 | Opinion Lexicon | 7.56 | 66.56 |
8 | Pattern.en | 7.57 | 8 | Umigon | 8.00 | 71.67 |
9 | SO-CAL | 9.00 | 9 | AFINN | 8.67 | 73.37 |
10 | Emolex | 12.29 | 10 | SO-CAL | 8.78 | 67.81 |
11 | SentiStrength | 12.43 (11.60) | 11 | VADER | 8.78 (9.75) | 83.29 (81.90) |
12 | Opinion Finder | 13.00 | 12 | Pattern.en | 11.22 | 69.47 |
13 | SentiWordNet | 13.57 | 13 | Sentiment140_L | 14.00 | 94.61 |
14 | SenticNet | 14.14 | 14 | Opinion Finder | 14.33 | 39.58 |
15 | SASA | 14.86 | 15 | Emolex | 14.56 | 62.63 |
16 | LIWC | 15.43 | 16 | USent | 15.22 | 38.60 |
17 | Sentiment140_L | 15.43 | 17 | SenticNet | 17.22 | 75.46 |
18 | USent | 16.00 | 18 | SentiWordNet | 18.44 | 61.41 |
19 | ANEW_SUB | 19.14 | 19 | NRC Hashtag | 19.11 | 94.20 |
20 | Emoticons | 19.14 | 20 | SASA | 19.44 | 58.57 |
21 | Stanford DM | 19.43 | 21 | LIWC | 19.56 | 61.24 |
22 | NRC Hashtag | 20.00 | 22 | ANEW_SUB | 20.56 | 93.51 |
23 | PANAS-t | 20.86 | 23 | Stanford DM | 22.56 | 89.06 |
24 | Emoticons DS | 23.71 | 24 | Emoticons DS | 23.78 | 99.28 |
3-classes
|
2-classes
| |||||
---|---|---|---|---|---|---|
Pos
|
Method
|
Mean Rank
|
Pos
|
Method
|
Mean Rank
|
Coverage (%)
|
1 | VADER | 3.33 (3.60) | 1 | SentiStrength | 1.17 (1.50) | 28.29 (24.02) |
2 | AFINN | 4.33 | 2 | Semantria | 2.83 | 61.02 |
3 | Opinion Lexicon | 4.33 | 3 | Sentiment140 | 4.17 | 36.49 |
4 | Semantria | 4.50 | 4 | Opinion Lexicon | 6.50 | 71.59 |
5 | SO-CAL | 5.17 | 5 | LIWC15 | 6.67 | 65.80 |
6 | LIWC15 | 6.17 | 6 | AFINN | 7.00 | 74.21 |
7 | Umigon | 9.50 | 7 | SO-CAL | 7.50 | 74.59 |
8 | Emolex | 10.33 | 8 | VADER | 9.50 (9.60) | 81.98 (85.34) |
9 | Sentiment140_L | 11.33 | 9 | Umigon | 10.50 | 57.87 |
10 | Stanford DM | 11.67 | 10 | Emoticons | 11.83 | 4.99 |
11 | NRC Hashtag | 12.00 | 11 | Opinion Finder | 13.00 | 55.66 |
12 | Pattern.en | 12.67 | 12 | SenticNet | 13.00 | 95.28 |
13 | SenticNet | 13.00 | 13 | USent | 14.00 | 45.66 |
14 | Opinion Finder | 13.17 | 14 | NRC Hashtag | 14.67 | 93.43 |
15 | SentiWordNet | 13.17 | 15 | Emolex | 15.00 | 69.69 |
16 | SASA | 14.67 | 16 | PANAS-t | 15.50 | 5.10 |
17 | SentiStrength | 15.17 (19.00) | 17 | Stanford DM | 15.67 | 84.43 |
18 | Sentiment140 | 15.50 | 18 | Pattern.en | 15.83 | 59.00 |
19 | USent | 15.83 | 19 | Sentiment140_L | 15.83 | 92.30 |
20 | LIWC | 17.67 | 20 | SentiWordNet | 17.00 | 63.32 |
21 | ANEW_SUB | 17.83 | 21 | SASA | 17.50 | 61.91 |
22 | Emoticons DS | 22.67 | 22 | LIWC | 19.67 | 62.24 |
23 | PANAS-t | 22.83 | 23 | ANEW_SUB | 22.00 | 94.31 |
24 | Emoticons | 23.17 | 24 | Emoticons DS | 23.67 | 99.31 |
3-classes
|
2-classes
| |||||
---|---|---|---|---|---|---|
Pos
|
Method
|
Mean Rank
|
Pos
|
Method
|
Mean Rank
|
Coverage (%)
|
1 | - | - | 1 | Sentiment140 | 3.33 | 21.82 |
2 | - | - | 2 | SenticNet | 4.00 | 87.05 |
3 | - | - | 3 | Semantria | 4.33 | 66.04 |
4 | - | - | 4 | SO-CAL | 4.33 | 83.20 |
5 | - | - | 5 | Opinion Lexicon | 4.67 | 74.14 |
6 | - | - | 6 | SentiStrength | 5.00 (5.00) | 24.56 (24.56) |
7 | - | - | 7 | Stanford DM | 5.33 | 87.89 |
8 | - | - | 8 | AFINN | 8.67 | 69.77 |
9 | - | - | 9 | VADER | 9.67 (11.00) | 79.39 (82.70) |
10 | - | - | 10 | Pattern.en | 10.33 | 63.70 |
11 | - | - | 11 | PANAS-t | 11.00 | 2.80 |
12 | - | - | 12 | Umigon | 11.33 | 53.90 |
13 | - | - | 13 | Emolex | 13.33 | 69.47 |
14 | - | - | 14 | LIWC15 | 13.67 | 62.90 |
15 | - | - | 15 | USent | 15.67 | 56.85 |
16 | - | - | 16 | SentiWordNet | 15.67 | 59.73 |
17 | - | - | 17 | Sentiment140_L | 16.00 | 91.71 |
18 | - | - | 18 | NRC Hashtag | 16.33 | 91.64 |
19 | - | - | 19 | Opinion Finder | 19.33 | 49.73 |
20 | - | - | 20 | LIWC | 20.00 | 62.75 |
21 | - | - | 21 | SASA | 20.33 | 61.22 |
22 | - | - | 22 | ANEW_SUB | 21.33 | 96.05 |
23 | - | - | 23 | Emoticons DS | 23.00 | 99.71 |
24 | - | - | 24 | Emoticons | 23.33 | 0.04 |
2-class experiments
|
3-class experiments
| ||
---|---|---|---|
Context: Social Networks | |||
FR | 175.94 | FR | 124.16 |
Critical value | 35.17 | Critical value | 35.17 |
Reject null hypothesis
|
Reject null hypothesis
| ||
Context: Comments | |||
FR | 95.59 | FR | 96.41 |
Critical value | 35.17 | Critical value | 35.17 |
Reject null hypothesis
|
Reject null hypothesis
| ||
Context: Reviews | |||
FR | 60.52 | FR | - |
Critical value | 35.17 | Critical value | - |
Reject null hypothesis
|
Reject null hypothesis
|