Skip to main content
Erschienen in: Soft Computing 22/2022

01.07.2022 | Application of soft computing

Multi-channel word embeddings for sentiment analysis

verfasst von: Jhe-Wei Lin, Tran Duy Thanh, Rong-Guey Chang

Erschienen in: Soft Computing | Ausgabe 22/2022

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Sentiment analysis (SA) is widely applied in practical applications and known as emotion AI or opinion mining. In fact, SA tasks are really hard even for human; there are many factors that can affect the overall performance such as the length of the text, idiom, metaphor, or slang. In this paper, mainly proposed multi-channel word embeddings for SA consists of three parts: improving word representation, applying attention mechanism, and applying the state-of-the-art deep learning models. Particularly, a better representation, first, allows our method to achieve accurate and higher performance. Second, the attention mechanism enhances the model ability in focusing on the place from where the useful and important information can be extracted. In final, in order to benchmark our proposed method, we implement deep learning models such as CNN, RNN, CNN variants. The experimental results show that our method achieved a higher performance compared to baseline methods. The experiment results highlighted the main contributions based on better word representation of multi-channel pre-trained word embeddings were shown. In addition, this model focuses on the words that contain useful and important information, and a higher accuracy was found compared to the previous models.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literatur
Zurück zum Zitat Bahdanau D et al. (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Bahdanau D et al. (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:​1409.​0473.
Zurück zum Zitat Bengio Y et al (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166CrossRef Bengio Y et al (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166CrossRef
Zurück zum Zitat Chen Y (2015) Convolutional neural network for sentence classification (Master's thesis, University of Waterloo). Chen Y (2015) Convolutional neural network for sentence classification (Master's thesis, University of Waterloo).
Zurück zum Zitat Cho K et al (2015) Describing multimedia content using attention-based encoder-decoder networks. IEEE Trans Multimedia 17(11):1875–1886CrossRef Cho K et al (2015) Describing multimedia content using attention-based encoder-decoder networks. IEEE Trans Multimedia 17(11):1875–1886CrossRef
Zurück zum Zitat Chung J et al. (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. Chung J et al. (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:​1412.​3555.
Zurück zum Zitat Da’u A, Salim N (2019) Aspect extraction on user textual reviews using multi-channel convolutional neural network. PeerJ Comp Sci 5:e191CrossRef Da’u A, Salim N (2019) Aspect extraction on user textual reviews using multi-channel convolutional neural network. PeerJ Comp Sci 5:e191CrossRef
Zurück zum Zitat Dey R and FM Salemt (2017). Gate-variants of Gated Recurrent Unit (GRU) neural networks. In: Circuits and systems (MWSCAS), 2017 IEEE 60th international midwest symposium on, IEEE. Dey R and FM Salemt (2017). Gate-variants of Gated Recurrent Unit (GRU) neural networks. In: Circuits and systems (MWSCAS), 2017 IEEE 60th international midwest symposium on, IEEE.
Zurück zum Zitat Show A (2015) Tell: Neural image caption generation with visual attention. Kelvin Xu et. al. arXiv Pre-Print 83: 89. Show A (2015) Tell: Neural image caption generation with visual attention. Kelvin Xu et. al. arXiv Pre-Print 83: 89.
Zurück zum Zitat Gao J et al. (2014) Modeling interestingness with deep neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Gao J et al. (2014) Modeling interestingness with deep neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP).
Zurück zum Zitat Ghassemi MM et al. (2015) A visualization of evolving clinical sentiment using vector representations of clinical notes. Computing in Cardiology Conference (CinC), 2015, IEEE. Ghassemi MM et al. (2015) A visualization of evolving clinical sentiment using vector representations of clinical notes. Computing in Cardiology Conference (CinC), 2015, IEEE.
Zurück zum Zitat Goldberg Y and O Levy (2014) word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. Goldberg Y and O Levy (2014) word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method. arXiv preprint arXiv:​1402.​3722.
Zurück zum Zitat Guo B, Zhang C, Liu J, Ma X (2019) Improving text classification with weighted word embeddings via a multi-channel TextCNN model. Neurocomputing 363:366–374CrossRef Guo B, Zhang C, Liu J, Ma X (2019) Improving text classification with weighted word embeddings via a multi-channel TextCNN model. Neurocomputing 363:366–374CrossRef
Zurück zum Zitat Guthrie D et al. (2006). A closer look at skip-gram modelling. In: Proceedings of the 5th international Conference on Language Resources and Evaluation (LREC-2006). Guthrie D et al. (2006). A closer look at skip-gram modelling. In: Proceedings of the 5th international Conference on Language Resources and Evaluation (LREC-2006).
Zurück zum Zitat Hermann, K. M., et al. (2015). Teaching machines to read and comprehend. Adv Neural Inf Process Syst. Hermann, K. M., et al. (2015). Teaching machines to read and comprehend. Adv Neural Inf Process Syst.
Zurück zum Zitat Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRef Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRef
Zurück zum Zitat Hofmann T (2001) Unsupervised learning by probabilistic latent semantic analysis. Mach Learn 42(1–2):177–196CrossRef Hofmann T (2001) Unsupervised learning by probabilistic latent semantic analysis. Mach Learn 42(1–2):177–196CrossRef
Zurück zum Zitat Huang A (2008) Similarity measures for text document clustering. In: Proceedings of the sixth new zealand computer science research student conference (NZCSRSC2008), Christchurch, New Zealand. Huang A (2008) Similarity measures for text document clustering. In: Proceedings of the sixth new zealand computer science research student conference (NZCSRSC2008), Christchurch, New Zealand.
Zurück zum Zitat Huang EH et al. (2012) Improving word representations via global context and multiple word prototypes. In: Proceedings of the 50th annual meeting of the association for computational linguistics: long papers-volume 1, Association for computational linguistics. Huang EH et al. (2012) Improving word representations via global context and multiple word prototypes. In: Proceedings of the 50th annual meeting of the association for computational linguistics: long papers-volume 1, Association for computational linguistics.
Zurück zum Zitat Johnson R and T Zhang (2015) Semi-supervised convolutional neural networks for text categorization via region embedding. Adv Neural Inf Process Syst. Johnson R and T Zhang (2015) Semi-supervised convolutional neural networks for text categorization via region embedding. Adv Neural Inf Process Syst.
Zurück zum Zitat Joulin, A. and T. Mikolov (2015). Inferring algorithmic patterns with stack-augmented recurrent nets. Adv Neural Inf Process Syst. Joulin, A. and T. Mikolov (2015). Inferring algorithmic patterns with stack-augmented recurrent nets. Adv Neural Inf Process Syst.
Zurück zum Zitat Kim Zeng D et al. (2014). Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers. Kim Zeng D et al. (2014). Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers.
Zurück zum Zitat Kiros R et al. (2015) Skip-thought vectors. Adv Neural Inf Process Syst. Kiros R et al. (2015) Skip-thought vectors. Adv Neural Inf Process Syst.
Zurück zum Zitat Landauer TK et al (1998) An introduction to latent semantic analysis. Discourse Process 25(2–3):259–284CrossRef Landauer TK et al (1998) An introduction to latent semantic analysis. Discourse Process 25(2–3):259–284CrossRef
Zurück zum Zitat Lee JY and F Dernoncourt (2016) Sequential short-text classification with recurrent and convolutional neural networks. arXiv preprint arXiv:1603.03827. Lee JY and F Dernoncourt (2016) Sequential short-text classification with recurrent and convolutional neural networks. arXiv preprint arXiv:​1603.​03827.
Zurück zum Zitat Levy O and Y Goldberg (2014a) Dependency-based word embeddings. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 2: Short Papers). Levy O and Y Goldberg (2014a) Dependency-based word embeddings. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 2: Short Papers).
Zurück zum Zitat Levy O and Y Goldberg (2014b) Neural word embedding as implicit matrix factorization. Adv Neural Inf Process Syst. Levy O and Y Goldberg (2014b) Neural word embedding as implicit matrix factorization. Adv Neural Inf Process Syst.
Zurück zum Zitat Liu B (2012) Sentiment analysis and opinion mining. Synth Lectures Human Lang Technol 5(1):1–167CrossRef Liu B (2012) Sentiment analysis and opinion mining. Synth Lectures Human Lang Technol 5(1):1–167CrossRef
Zurück zum Zitat Liu Y et al (2018) Modelling context with neural networks for recommending idioms in essay writing. Neurocomputing 275:2287–2293CrossRef Liu Y et al (2018) Modelling context with neural networks for recommending idioms in essay writing. Neurocomputing 275:2287–2293CrossRef
Zurück zum Zitat Liu, P., et al. (2015). Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model. IJCAI. Liu, P., et al. (2015). Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model. IJCAI.
Zurück zum Zitat Mikolov T et al. (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst. Mikolov T et al. (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst.
Zurück zum Zitat Mikolov, T., et al. (2010). Recurrent neural network based language model. In: Eleventh annual conference of the international speech communication association. Mikolov, T., et al. (2010). Recurrent neural network based language model. In: Eleventh annual conference of the international speech communication association.
Zurück zum Zitat Mikolov T et al. (2013b) Linguistic regularities in continuous space word representations. In: Proceedings of the 2013b conference of the north american chapter of the association for computational linguistics: human language technologies. Mikolov T et al. (2013b) Linguistic regularities in continuous space word representations. In: Proceedings of the 2013b conference of the north american chapter of the association for computational linguistics: human language technologies.
Zurück zum Zitat Mnih V et al. (2014) Recurrent models of visual attention. Adv Neural Inf Process Syst. Mnih V et al. (2014) Recurrent models of visual attention. Adv Neural Inf Process Syst.
Zurück zum Zitat Mohammad SM et al. (2013) NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:1308.6242. Mohammad SM et al. (2013) NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. arXiv preprint arXiv:​1308.​6242.
Zurück zum Zitat Mullen T and N Collier (2004) Sentiment analysis using support vector machines with diverse information sources. In: Proceedings of the 2004 conference on empirical methods in natural language processing. Mullen T and N Collier (2004) Sentiment analysis using support vector machines with diverse information sources. In: Proceedings of the 2004 conference on empirical methods in natural language processing.
Zurück zum Zitat Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends® Inf Retr 2(1–2):1–135CrossRef Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends® Inf Retr 2(1–2):1–135CrossRef
Zurück zum Zitat Pang B et al. (2002) Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, Association for computational linguistics. Pang B et al. (2002) Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, Association for computational linguistics.
Zurück zum Zitat Pennington J et al. (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Pennington J et al. (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP).
Zurück zum Zitat Qin P et al (2016) An empirical convolutional neural network approach for semantic relation classification. Neurocomputing 190:1–9CrossRef Qin P et al (2016) An empirical convolutional neural network approach for semantic relation classification. Neurocomputing 190:1–9CrossRef
Zurück zum Zitat Raffel C and D Ellis (2015) Feed-forward networks with attention can solve some long-term memory problems. arXiv preprint arXiv:1512.08756. Raffel C and D Ellis (2015) Feed-forward networks with attention can solve some long-term memory problems. arXiv preprint arXiv:​1512.​08756.
Zurück zum Zitat Rodriguez P et al (1999) A recurrent neural network that learns to count. Connect Sci 11(1):5–40CrossRef Rodriguez P et al (1999) A recurrent neural network that learns to count. Connect Sci 11(1):5–40CrossRef
Zurück zum Zitat Rosenthal S et al. (2017) SemEval-2017 task 4: Sentiment analysis in Twitter. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017). Rosenthal S et al. (2017) SemEval-2017 task 4: Sentiment analysis in Twitter. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017).
Zurück zum Zitat Severyn A and A Moschitti (2015) Twitter sentiment analysis with deep convolutional neural networks. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, ACM. Severyn A and A Moschitti (2015) Twitter sentiment analysis with deep convolutional neural networks. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, ACM.
Zurück zum Zitat Shen Y et al. (2014) A latent semantic model with convolutional-pooling structure for information retrieval. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management, ACM. Shen Y et al. (2014) A latent semantic model with convolutional-pooling structure for information retrieval. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management, ACM.
Zurück zum Zitat Socher R et al. (2013) Parsing with compositional vector grammars. In: Proceedings of the 51st annual meeting of the association for computational linguistics (Volume 1: Long Papers). Socher R et al. (2013) Parsing with compositional vector grammars. In: Proceedings of the 51st annual meeting of the association for computational linguistics (Volume 1: Long Papers).
Zurück zum Zitat Sriram B et al. (2010) Short text classification in twitter to improve information filtering. In: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, ACM. Sriram B et al. (2010) Short text classification in twitter to improve information filtering. In: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, ACM.
Zurück zum Zitat Sun, Y., et al. (2015). Modeling mention, context and entity with neural networks for entity disambiguation. In: IJCAI. Sun, Y., et al. (2015). Modeling mention, context and entity with neural networks for entity disambiguation. In: IJCAI.
Zurück zum Zitat Sundermeyer M et al. (2012) LSTM neural networks for language modeling. In: Thirteenth annual conference of the international speech communication association. Sundermeyer M et al. (2012) LSTM neural networks for language modeling. In: Thirteenth annual conference of the international speech communication association.
Zurück zum Zitat Tang D et al. (2014) Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers). Tang D et al. (2014) Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers).
Zurück zum Zitat Tissier J et al. (2017) Dict2vec: learning word embeddings using lexical dictionaries. In: Conference on empirical methods in natural language processing (EMNLP 2017). Tissier J et al. (2017) Dict2vec: learning word embeddings using lexical dictionaries. In: Conference on empirical methods in natural language processing (EMNLP 2017).
Zurück zum Zitat Turian J et al. (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics, Association for computational linguistics. Turian J et al. (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics, Association for computational linguistics.
Zurück zum Zitat Turney PD (2002) Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In: Proceedings of the 40th annual meeting on association for computational linguistics, Association for computational linguistics. Turney PD (2002) Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In: Proceedings of the 40th annual meeting on association for computational linguistics, Association for computational linguistics.
Zurück zum Zitat Wallach HM (2006) Topic modeling: beyond bag-of-words. In: Proceedings of the 23rd international conference on Machine learning, ACM. Wallach HM (2006) Topic modeling: beyond bag-of-words. In: Proceedings of the 23rd international conference on Machine learning, ACM.
Zurück zum Zitat Wang X et al. (2016a) Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In: Proceedings of COLING 2016a, the 26th international conference on computational linguistics: technical papers. Wang X et al. (2016a) Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In: Proceedings of COLING 2016a, the 26th international conference on computational linguistics: technical papers.
Zurück zum Zitat Wang Y et al. (2016b) Attention-based lstm for aspect-level sentiment classification. In: Proceedings of the 2016b conference on empirical methods in natural language processing. Wang Y et al. (2016b) Attention-based lstm for aspect-level sentiment classification. In: Proceedings of the 2016b conference on empirical methods in natural language processing.
Zurück zum Zitat Wang J et al. (2016c) Dimensional sentiment analysis using a regional CNN-LSTM model. In: Proceedings of the 54th annual meeting of the association for computational linguistics (Volume 2: Short Papers). Wang J et al. (2016c) Dimensional sentiment analysis using a regional CNN-LSTM model. In: Proceedings of the 54th annual meeting of the association for computational linguistics (Volume 2: Short Papers).
Zurück zum Zitat Yoon J and Kim H (2017, November) Multi-channel lexicon integrated CNN-BiLSTM models for sentiment analysis. In Proceedings of the 29th conference on computational linguistics and speech processing (ROCLING 2017) (pp. 244–253). Yoon J and Kim H (2017, November) Multi-channel lexicon integrated CNN-BiLSTM models for sentiment analysis. In Proceedings of the 29th conference on computational linguistics and speech processing (ROCLING 2017) (pp. 244–253).
Zurück zum Zitat Zhang Y et al (2010) Understanding bag-of-words model: a statistical framework. Int J Mach Learn Cybern 1(1–4):43–52CrossRef Zhang Y et al (2010) Understanding bag-of-words model: a statistical framework. Int J Mach Learn Cybern 1(1–4):43–52CrossRef
Zurück zum Zitat Zhang Y and B Wallace (2015) A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820. Zhang Y and B Wallace (2015) A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:​1510.​03820.
Metadaten
Titel
Multi-channel word embeddings for sentiment analysis
verfasst von
Jhe-Wei Lin
Tran Duy Thanh
Rong-Guey Chang
Publikationsdatum
01.07.2022
Verlag
Springer Berlin Heidelberg
Erschienen in
Soft Computing / Ausgabe 22/2022
Print ISSN: 1432-7643
Elektronische ISSN: 1433-7479
DOI
https://doi.org/10.1007/s00500-022-07267-6

Weitere Artikel der Ausgabe 22/2022

Soft Computing 22/2022 Zur Ausgabe