1 Introduction
1.1 Motivation and Purpose
1.2 Approach
1.2.1 CSNN
1.2.2 IP Learning
1.3 Contribution
- We proposed a novel NN architecture called CSNN that can explain its sentiment analysis process in a form that humans find natural and agreeable.
- To realize the interpretability of CSNN, we proposed a novel learning strategy called IP learning.
- We experimentally demonstrated the high interpretability and high predictability of the proposed CSNN.
2 CSNN
2.1 Structure of CSNN
2.1.1 WOSL
2.1.2 SSL
2.1.3 GIL
2.1.4 WCSL
2.1.5 CCSL
2.1.6 Output
2.2 Key Idea in IP learning
2.3 Initialization and Propagation (IP) Learning
2.3.1 Update
2.3.2 Init
3 Pre-experimental Evaluation for IP Learning
3.1 Dataset
3.1.1 Text Corpus
3.1.2 Annotated Dataset
3.2 CSNN Development Setting
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | |
---|---|---|---|---|
Training | ||||
Positive reviews | 20,000 | 35,000 | 30,612 | 650,000 |
Negative reviews | 20,000 | 35,000 | 9388 | 650,000 |
Validation | ||||
Positive reviews | 2000 | 2000 | 3387 | 50,000 |
Negative reviews | 2000 | 2000 | 1613 | 50,000 |
Test | ||||
Positive reviews | 4000 | 4000 | 7538 | 100,000 |
Negative reviews | 4000 | 4000 | 2462 | 100,000 |
Vocabulary size v | 8071 | 11,130 | 33,080 | 71,316 |
(i) Word polarity list | |||||
---|---|---|---|---|---|
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | ||
Positive | 348 | 337 | 422 | 1843 | |
Negative | 391 | 387 | 372 | 947 |
(ii) Sentiment shift tags | |||||
---|---|---|---|---|---|
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | ||
Shifted tags | 872 | 859 | 378 | 429 | |
Non-shifted tags | 3762 | 3740 | 2391 | 4504 |
(iii) Word-level global important point tags | |||||
---|---|---|---|---|---|
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | ||
Important tags (1) | 6632 | 6631 | 1526 | - | |
Unimportant tags (0) | 62,652 | 62,652 | 48,890 | - |
(iv) word-level and phrase-level contextual polarity tags | |||||
---|---|---|---|---|---|
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | ||
Level | Word | Word | Word | Word | Phrase |
Shifted negative | 776 | 756 | 227 | 169 | – |
Non-shifted negative | 1491 | 1483 | 1187 | 1294 | – |
Shifted positive | 96 | 96 | 151 | 260 | – |
Non-shifted positive | 2271 | 2179 | 1204 | 3210 | – |
Negative (total) | 2267 | 2239 | 1414 | 1463 | 3634 |
Positive (total) | 2367 | 2275 | 1355 | 3470 | 5907 |
3.3 Evaluation Metrics in Explanation ability
3.3.1 Validity of WOSL
3.3.2 Validity of SSL
3.3.3 Validity of GIL
3.3.4 Validity of WCSL
- \(CSNN^{Base}\) is developed using the general backpropagation and without Update or Init strategy.
- \(CSNN^{Random}\) is developed with only Update strategy.
- \(CSNN^{NoUp}\) is developed with only Init strategy.
4 Experimental Evaluation for CSNN
4.1 Evaluation Metrics in Predictability
4.2 Result
4.2.1 Explanation ability and Predictability
4.2.2 Effect of IP Learning
4.3 Discussion
4.3.1 Predictability
4.3.2 Effect of IP Learning
4.3.3 Sentiment Shift Detection Performance in Yahoo Dataset
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | |
---|---|---|---|---|
PMI | 0.734 | 0.745 | 0.793 | 0.733 |
LFW | 0.715 | 0.740 | 0.766 | 0.725 |
SONN | 0.702 | 0.724 | 0.725 | 0.705 |
GINN | 0.723 | 0.755 | 0.754 | 0.735 |
\(CSNN^{Base}\) | 0.417 | 0.381 | 0.499 | 0.373 |
\(CSNN^{NoUp}\) | 0.832 | 0.846 | 0.798 | 0.754 |
\(CSNN^{Rand}\) | 0.452 | 0.543 | 0.460 | 0.430 |
CSNN (200) | 0.837 | 0.865 | 0.825 | 0.742 |
CSNN (100) | 0.838 | 0.851 | 0.817 | 0.744 |
CSNN (50) | 0.843 | 0.865 | 0.805 | 0.743 |
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | |
---|---|---|---|---|
Baseline | 0.660 | 0.712 | 0.579 | 0.560 |
NegRNN | 0.536 | 0.626 | 0.564 | 0.558 |
\(CSNN^{Base}\) | 0.661 | 0.311 | 0.244 | 0.314 |
\(CSNN^{NoUp}\) | 0.374 | 0.246 | 0.360 | 0.417 |
\(CSNN^{Rand}\) | 0.263 | 0.531 | 0.315 | 0.293 |
CSNN (200) | 0.777 | 0.804 | 0.691 | 0.743 |
CSNN (100) | 0.780 | 0.816 | 0.681 | 0.751 |
CSNN (50) | 0.784 | 0.809 | 0.675 | 0.762 |
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | |
---|---|---|---|---|
ATT | \(-\) 0.015 | \(-\) 0.081 | 0.062 | – |
HN-ATT | 0.108 | 0.188 | 0.262 | – |
SNNN | 0.281 | 0.456 | 0.192 | – |
LBSA | 0.333 | 0.344 | 0.405 | – |
\(CSNN^{Base}\) | 0.014 | 0.170 | 0.171 | – |
\(CSNN^{NoUp}\) | 0.607 | 0.590 | 0.329 | – |
\(CSNN^{Rand}\) | 0.207 | 0.224 | 0.164 | – |
CSNN (200) | 0.595 | 0.580 | 0.325 | – |
CSNN (100) | 0.584 | 0.567 | 0.308 | |
CSNN (50) | 0.585 | 0.562 | 0.321 | – |
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | ||
---|---|---|---|---|---|
Level | Word | Word | Word | Word | Phrase |
PMI | 0.578 | 0.548 | 0.575 | 0.631 | 0.822 |
Grad + RNN | 0.578 | .621 | .601 | 0.681 | 0.743 |
IntGrad + RNN | 0.607 | 0.621 | 0.625 | 0.679 | 0.796 |
LRP + RNN | 0.597 | 0.518 | 0.579 | 0.638 | 0.808 |
LFW | 0.549 | 0.545 | 0.578 | 0.587 | 0.749 |
SONN | 0.555 | 0.542 | 0.566 | 0.600 | 0.787 |
GINN | 0.569 | 0.555 | 0.577 | 0.623 | 0.831 |
\(CSNN^{Base}\) | 0.355 | 0.521 | 0.490 | 0.575 | 0.595 |
\(CSNN^{NoUp}\) | 0.416 | 0.316 | 0.526 | 0.509 | 0.512 |
\(CSNN^{Rand}\) | 0.606 | 0.621 | 0.516 | 0.794 | 0.748 |
CSNN (200) | 0.676 | 0.711 | 0.669 | 0.788 | 0.858 |
CSNN (100) | 0.679 | 0.723 | 0.675 | 0.784 | 0.862 |
CSNN (50) | 0.692 | 0.719 | 0.670 | 0.788 | 0.857 |
EcoRev I | EcoRev II | Yahoo | Sentiment 140 | |
---|---|---|---|---|
LR | 0.878 | 0.879 | 0.741 | 0.785 |
LFW | 0.876 | 0.840 | 0.751 | 0.745 |
SONN | 0.863 | 0.876 | 0.717 | 0.776 |
GINN | 0.860 | 0.859 | 0.740 | 0.782 |
CNN | 0.894 | 0.911 | 0.757 | 0.820 |
RNN | 0.922 | 0.932 | 0.749 | 0.837 |
ATT | 0.924 | 0.937 | 0.750 | 0.835 |
HN-ATT | 0.927 | 0.940 | 0.750 | 0.837 |
SNNN | 0.918 | 0.928 | 0.752 | 0.827 |
LBSA | 0.922 | 0.941 | 0.762 | 0.832 |
CSNN (200) | 0.921 | 0.938 | 0.768 | 0.833 |
CSNN (100) | 0.914 | 0.937 | 0.762 | 0.835 |
CSNN (50) | 0.916 | 0.939 | 0.765 | 0.833 |