Damian Borth
ABSTRACT
We address the challenge of sentiment analysis from visual content. In contrast to existing methods which infer sentiment or emotion directly from visual low-level features, we propose a novel approach based on understanding of the visual concepts that are strongly related to sentiments. Our key contribution is two-fold: first, we present a method built upon psychological theories and web mining to automatically construct a large-scale Visual Sentiment Ontology (VSO) consisting of more than 3,000 Adjective Noun Pairs (ANP). Second, we propose SentiBank, a novel visual concept detector library that can be used to detect the presence of 1,200 ANPs in an image. The VSO and SentiBank are distinct from existing work and will open a gate towards various applications enabled by automatic sentiment analysis. Experiments on detecting sentiment of image tweets demonstrate significant improvement in detection accuracy when comparing the proposed SentiBank based predictors with the text-based approaches. The effort also leads to a large publicly available resource consisting of a visual sentiment ontology, a large detector library, and the training/testing benchmark for visual sentiment analysis.
- H. Aradhye, G. Toderici, and J. Yagnik. Video2Text: Learning to Annotate Video Content. Internet Multimedia Mining, 2009. Google Scholar
Digital Library
- D. Borth, A. Ulges, and T.M. Breuel. Lookapp - Interactive Construction of web-based Concept Detectors. ICMR, 2011. Google ScholarDigital Library
- D. Borth and S-F. Chang. Constructing Structures and Relations in SentiBank Visual Sentiment Ontology. Technical Report#CUCS-020--13, Columbia University, Computer Science Dep., 2013.Google Scholar
- E. Dan-Glauser et al. The Geneva Affective Picture Database (GAPED): a new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 2011.Google Scholar
- Charles Darwin. The Expression of the Emotions in Man and Animals. Oxford University Press, USA, 1872 / 1998.Google Scholar
- R. Datta, D. Joshi, J. Li, and J. Wang. Studying Aesthetics in Photographic Images using a Computational Approach. ECCV, 2006. Google ScholarDigital Library
- J. Deng et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.Google ScholarCross Ref
- P. Ekman et al. Facial Expression and Emotion. American Psychologist, 48:384--384, 1993.Google ScholarCross Ref
- A. Esuli and F. Sebastiani. SentiWordnet: A publicly available Lexical Resource for Opinion Mining. LREC, 2006.Google Scholar
- M. Everingham, et al. The Pascal Visual Object Classes (VOC) Challenge. Int. J. of Computer Vision, 88(2):303--338, 2010. Google ScholarDigital Library
- A. Hanjalic, C. Kofler, and M. Larson. Intent and its Discontents: the User at the Wheel of the Online Video Search Engine. ACM MM, 2012. Google ScholarDigital Library
- P. Isola, J. Xiao, A. Torralba, and A. Oliva. What makes an Image Memorable? CVPR, 2011. Google ScholarDigital Library
- J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, and J. Tang. Can we understand van Gogh's Mood?: Learning to infer Affects from Images in Social Networks. ACM MM, 2012. Google ScholarDigital Library
- Y.-G. Jiang, G. Ye, S.-F. Chang, D. Ellis, and A. Loui. Consumer Video Understand.: Benchmark Database and an Eval. of Human and Machine Performance. ICMR, 2011. Google ScholarDigital Library
- D. Joshi, R. Datta, E. Fedorovskaya, Q. Luong, J. Wang, J. Li, and J. Luo. Aesthetics and Emotions in Images. Signal Processing Magazine, 28(5):94--115, 2011.Google ScholarCross Ref
- L. Kennedy, S.-F. Chang, and I. Kozintsev. To Search or to Label?: Predicting the Performance of Search-based Automatic Image Classifiers. MIR Workshop, 2006. Google ScholarDigital Library
- P. Lang, M. Bradley, and B. Cuthbert. International Affective Picture System (IAPS): Technical Manual and Affective Ratings, 1999.Google Scholar
- B. Li, et al. Scaring or Pleasing: Exploit Emotional Impact of an Image. ACM MM, 2012. Google ScholarDigital Library
- X. Li, C. Snoek, M. Worring, and A. Smeulders. Harvesting Social Images for Bi-Concept Search. IEEE Transactions on Multimedia, 14(4):1091--1104, 2012.Google ScholarDigital Library
- N. Codella et al. IBM Research and Columbia University TRECVID-2011 Multimedia Event Detection (med) System. NIST TRECVID Workshop, 2011.Google Scholar
- J. Machajdik and A. Hanbury. Affective Image Classification using Features inspired by Psychology and Art Theory. ACM MM, 2010. Google ScholarDigital Library
- L. Marchesotti, F. Perronnin, D. Larlus, and G. Csurka. Assessing the Aesthetic Quality of Photographs using Generic Image Descriptors. ICCV, 2011. Google ScholarDigital Library
- M. Naphade, J. Smith, J. Tesic, S. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis. Large-Scale Concept Ontology for Multimedia. IEEE MultiMedia, 13(3):86--91, 2006. Google ScholarDigital Library
- C. Osgood, G. Suci, and P. Tannenbaum. The Measurement of Meaning, volume 47. University of Illinois Press, 1957.Google Scholar
- P. Over et al. Trecvid 2012 -- An Overview of the Goals, Tasks, Data, Evaluation Mechanisms and Metrics. TRECVID Workshop, 2012.Google Scholar
- B. Pang and L. Lee. Opinion Mining and Sentiment Analysis. Information Retrieval, 2(1--2):1--135, 2008. Google ScholarDigital Library
- Robert Plutchik. Emotion: A Psychoevolutionary Synthesis. Harper & Row, Publishers, 1980.Google Scholar
- C. Snoek and M. Worring. Concept-based Video Retrieval. Foundations and Trends in Inf. Retrieval, 4(2), 2009. Google ScholarDigital Library
- S. Strassel et al. Creating HAVIC: Heterogeneous Audio Visual Internet Collection. LREC, 2012.Google Scholar
- M. Thelwall et al. Sentiment Strength Detection in Short Informal Text. J. of the American Soc. for Information Science and Tech., 61(12):2544--2558, 2010. Google ScholarDigital Library
- A. Ulges, C. Schulze, M. Koch, and T. Breuel. Learning Automatic Concept Detectors from Online Video. Journal on Comp. Vis. Img. Underst., 114(4):429--438, 2010. Google ScholarDigital Library
- V. Vonikakis and S. Winkler. Emotion-based Sequence of Family Photos. ACM MM, 2012. Google ScholarDigital Library
- W. Wang and Q. He. A Survey on Emotional Semantic Image Retrieval. IEEE ICIP, 2008.Google ScholarCross Ref
- X. Wang, J. Jia, P. Hu, S. Wu, J. Tang, and L. Cai. Understanding the Emotional Impact of Images. ACM MM, 2012. Google ScholarDigital Library
- T. Wilson et al. Recognizing Contextual Polarity in phrase-level Sentiment Analysis. HLT/EMNLP, 2005. Google ScholarDigital Library
- V. Yanulevskaya et al. In the Eye of the Beholder: Employing Statistical Analysis and Eye Tracking for Analyzing Abstract Paintings. ACM MM, 2012. Google ScholarDigital Library
- V. Yanulevskaya et al. Emotional Valence Categorization using Holistic Image Features. IEEE ICIP, 2008.Google ScholarCross Ref
- Li et al. ObjectBank: A high-level Image Rep. for Scene Classification and Semantic Feature Sparsification. NIPS, 2010.Google Scholar
- Torresani et al. Efficient Object Category Recognition using Classemes. ECCV, 2010. Google ScholarDigital Library
- A. Olivai and A. Torralba. Modeling the Shape of the Scene: a Holistic Representation of the Spatial Envelope. Int. J. of Computer Vision, 42(3):145--175, 2001. Google ScholarDigital Library
- S. Bhattacharya and R. Sukthankar and M. Shah. A holistic Approach to Aesthetic Enhancement of Photographs. TOMCCAP, 7(1), 2011. Google ScholarDigital Library
Index Terms
- Large-scale visual sentiment ontology and detectors using adjective noun pairs
Recommendations
SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content
MM '13: Proceedings of the 21st ACM international conference on MultimediaA picture is worth one thousand words, but what words should be used to describe the sentiment and emotions conveyed in the increasingly popular social multimedia? We demonstrate a novel system which combines sound structures from psychology and the ...
Image sentiment prediction based on textual descriptions with adjective noun pairs
We aim to predict the sentiment related information reflected in images based on SentiBank, which is a library including Adjective Noun Pair (ANP) concept detectors for image sentiment analysis. Instead of using only ANP responses in images as mid-level ...
Multilingual Visual Sentiment Concept Matching
ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia RetrievalThe impact of culture in visual emotion perception has recently captured the attention of multimedia research. In this study, we provide powerful computational linguistics tools to explore, retrieve and browse a dataset of 16K multilingual affective ...
Comments