Louis-Philippe Morency
ABSTRACT
With more than 10,000 new videos posted online every day on social websites such as YouTube and Facebook, the internet is becoming an almost infinite source of information. One crucial challenge for the coming decade is to be able to harvest relevant information from this constant flow of multimodal data. This paper addresses the task of multimodal sentiment analysis, and conducts proof-of-concept experiments that demonstrate that a joint model that integrates visual, audio, and textual features can be effectively used to identify sentiment in Web videos. This paper makes three important contributions. First, it addresses for the first time the task of tri-modal sentiment analysis, and shows that it is a feasible task that can benefit from the joint exploitation of visual, audio and textual modalities. Second, it identifies a subset of audio-visual features relevant to sentiment analysis and present guidelines on how to integrate these features. Finally, it introduces a new dataset consisting of real online data, which will be useful for future research in this area.
- C. Alm, D. Roth, and R. Sproat. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 347--354, Vancouver, Canada, 2005. Google Scholar
Digital Library
- J. Arguello and C. Rose. Topic segmentation of dialogue. In HLT-NAACL Workshop on Analyzing Conversations in Text and Speech, 2009. Google ScholarDigital Library
- K. Balog, G. Mishne, and M. de Rijke. Why are they excited? identifying and explaining spikes in blog mood levels. In Proceedings of the 11th Meeting of the European Chapter of the As sociation for Computational Linguistics (EACL-2006), 2006. Google ScholarDigital Library
- T. Banziger and K. R. Scherer. Introducing the geneva multimodal emotion portrayal (gemep) corpus. Oxford University Press, 2010.Google Scholar
- J. Blitzer, M. Dredze, and F. Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics, 2007.Google Scholar
- C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Journal of Language Resources and Evaluation, 42(4):335--359, December 2008.Google ScholarCross Ref
- C. Busso and S. Narayanan. Interrelation between speech and facial gestures in emotional utterances: a single subject study. IEEE Transactions on Audio, Speech and Language Processing, 15(8):2331--2347, November 2007. Google ScholarDigital Library
- G. Carenini, R. Ng, and X. Zhou. Summarizing emails with conversational cohesion and subjectivity. In Proceedings of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT 2008), Columbus, Ohio, 2008.Google Scholar
- P. Ekman. Facial expression of emotion. American Psychologist, 48:384--392, 1993.Google ScholarCross Ref
- A. Esuli and F. Sebastiani. SentiWordNet: A publicly available lexical resource for opinion mining. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC 2006), Genova, IT, 2006.Google Scholar
- G.McKeown, M. Valstar, R. Cowie, and M. Pantic. The semaine corpus of emotionally coloured character interactions. In ICME, pages 1079--1084, 2010.Google ScholarCross Ref
- N. Godbole, M. Srinivasaiah, and S. Sekine. Large-scale sentiment analysis for news and blogs. In International Conference on Weblogs and Social Media, Denver, CO, 2007.Google Scholar
- J. A. Harrigan, T. E. Oxman, and R. Rosenthal. Rapport expressed through nonverbal behavior. Journal of Nonverbal Behavior, 9(2), 1985.Google ScholarCross Ref
- M. Hu and B. Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, Seattle, Washington, 2004. Google ScholarDigital Library
- N. Kaji and M. Kitsuregawa. Building lexicon for sentiment analysis from massive collection of html documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Prague, Czech Republic, 2007.Google Scholar
- R. Mihalcea, C. Banea, and J. Wiebe. Learning multilingual subjective language via cross-lingual projections. In Proceedings of the Association for Computational Linguistics, Prague, Czech Republic, 2007.Google Scholar
- L.-P. Morency, A. Quattoni, and T. Darrell. Latent-dynamic discriminative models for continuous gesture recognition. In CVPR, June 2007.Google ScholarCross Ref
- B. Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, Barcelona, Spain, July 2004. Google ScholarDigital Library
- T. Plotz and G. A. Fink. Markov models for offline handwriting recognition: a survey. International Journal on Document Analysis and Recognition, 12(4), 2009. Google ScholarDigital Library
- S. Raaijmakers, K. Truong, and T. Wilson. Multimodal subjectivity analysis of multiparty conversation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 466--474, Honolulu, Hawaii, 2008. Google ScholarDigital Library
- L. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 1989.Google ScholarCross Ref
- F. E. M. W. B. Schuller. Openear introducing the munich open-source emotion and affect recognition toolkit. In ACII, 2009.Google Scholar
- N. Sebe, I. Cohen, T. Gevers, and T. Huang. Emotion recognition based on joint visual and audio cues. In ICPR, 2006. Google ScholarDigital Library
- P. Stone. General Inquirer: Computer Approach to Content Analysis. MIT Press, 1968.Google Scholar
- C. Strapparava and R. Mihalcea. Semeval-2007 task 14: Affective text. In Proceedings of the 4th International Workshop on the Semantic Evaluations (SemEval 2007), Prague, Czech Republic, 2007. Google ScholarDigital Library
- M. Taboada, J. Brooke, M. Tofiloski, K. Voli, and M. Stede. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(3), 2011. Google ScholarDigital Library
- J. Wiebe and R. Mihalcea. Word sense and subjectivity. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 2006. Google ScholarDigital Library
- J. Wiebe and E. Riloff. Creating subjective and objective sentence classifiers from unannotated texts. In Proceedings of the 6th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2005) (invited paper), Mexico City, Mexico, 2005. Google ScholarDigital Library
- J. Wiebe, T. Wilson, and C. Cardie. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2--3):165--210, 2005.Google Scholar
- M. Wollmer, B. Schuller, F. Eyben, and G. Rigoll. Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening. IEEE Journal of Selected Topics in Signal Processing, 4(5), October 2010.Google ScholarCross Ref
- H. Yu and V. Hatzivassiloglou. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Conference on Empirical Methods in Natural Language Processing (EMNLP-03), pages 129--136, Sapporo, Japan, 2003. Google ScholarDigital Library
- Z. Zhihong, M. P. G. Roisman, and T. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. PAMI, 31(1), 2009. Google ScholarDigital Library
Index Terms
- Towards multimodal sentiment analysis: harvesting opinions from the web
Recommendations
Joint sentiment/topic model for sentiment analysis
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementSentiment analysis or opinion mining aims to use automated tools to detect subjective information such as opinions, attitudes, and feelings expressed in text. This paper proposes a novel probabilistic modeling framework based on Latent Dirichlet ...
Topic sentiment change analysis
MLDM'11: Proceedings of the 7th international conference on Machine learning and data mining in pattern recognitionPublic opinions on a topic may change over time. Topic Sentiment change analysis is a new research problem consisting of two main components: (a) mining opinions on a certain topic, and (b) detect significant changes of sentiment of the opinions on the ...
Towards jointly extracting aspects and aspect-specific sentiment knowledge
CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge managementIn this paper, we aim to jointly extract aspects and aspect-specific sentiment knowledge from online reviews, where the sentiment knowledge refers to the aspect-specific opinion words along with their aspect-aware sentiment polarities. To this end, we ...
Comments