Victor Campos
Xavier Giro-i-Nieto
ABSTRACT
Visual media are powerful means of expressing emotions and sentiments. The constant generation of new content in social networks highlights the need of automated visual sentiment analysis tools. While Convolutional Neural Networks (CNNs) have established a new state-of-the-art in several vision problems, their application to the task of sentiment analysis is mostly unexplored and there are few studies regarding how to design CNNs for this purpose. In this work, we study the suitability of fine-tuning a CNN for visual sentiment prediction as well as explore performance boosting techniques within this deep learning setting. Finally, we provide a deep-dive analysis into a benchmark, state-of-the-art network architecture to gain insight about how to design patterns for CNNs on the task of visual sentiment prediction.
- D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM MM, 2013. Google Scholar
Digital Library
- K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In British Machine Vision Conference, 2014.Google ScholarCross Ref
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248--255. IEEE, 2009.Google ScholarCross Ref
- K. Dinakar, B. Jones, C. Havasi, and H. Lieberman. Common sense reasoning for detection, prevention, and mitigation of cyberbullying.Google Scholar
- K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. arXiv:abs/1502.01852 {cs.CV}, 2015.Google Scholar
- J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, and J. Tang. Can we understand van Gogh's mood?: Learning to infer affects from images in social networks. In ACM MM, 2012. Google ScholarDigital Library
- Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, 2014. Google ScholarDigital Library
- Y.-G. Jiang, B. Xu, and X. Xue. Predicting emotions in user-generated videos. In AAAI, 2014.Google ScholarDigital Library
- X. Jin, A. Gallagher, L. Cao, J. Luo, and J. Han. The wisdom of social multimedia: Using Flickr for prediction and forecast. In ACM MM, 2010. Google ScholarDigital Library
- B. Jou, S. Bhattacharya, and S.-F. Chang. Predicting viewer perceived emotions in animated GIFs. In ACM MM, 2014. Google ScholarDigital Library
- Y. Kim, H. Lee, and E. M. Provost. Deep learning for robust feature generation in audiovisual emotion recognition. In ICASSP, 2013.Google ScholarCross Ref
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.Google ScholarDigital Library
- P. Lang, M. Bradley, and B. Cuthbert. International Affective Picture System (IAPS): Technical manual and affective ratings. Technical report, NIMH CSEA, 1997.Google Scholar
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proc. of the IEEE, 1998.Google Scholar
- J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In ACM MM, 2010. Google ScholarDigital Library
- D. McDuff, R. Kaliouby, J. Cohn, and R. Picard. Predicting ad liking and purchase intent: Large-scale analysis of facial responses to ads.Google Scholar
- M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1717--1724. IEEE, 2014. Google ScholarDigital Library
- K.-C. Peng, T. Chen, A. Sadovnik, and A. Gallagher. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In CVPR, 2015.Google ScholarCross Ref
- K.-C. Peng, K. Karlsson, T. Chen, D.-Q. Zhang, and H. Yu. A framework of changing image emotion using emotion prediction.Google Scholar
- R. Plutchik. Emotion: A Psychoevolutionary Synthesis. Harper & Row, 1980.Google Scholar
- A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: An astounding baseline for recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 512--519. IEEE, 2014. Google ScholarDigital Library
- A. Salvador, M. Zeppelzauer, D. Manchon-Vizuete, A. Calafell, and X. Giro-i Nieto. Cultural event recognition with visual convnets and temporal models. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2015 IEEE Conference on. IEEE, 2015.Google ScholarCross Ref
- S. Siersdorfer, E. Minack, F. Deng, and J. Hare. Analyzing and predicting sentiment of images on the social web. In Proceedings of the international conference on Multimedia, pages 715--718. ACM, 2010. Google ScholarDigital Library
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.Google Scholar
- C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014.Google Scholar
- Y. Tang. Deep learning using linear support vector machines. In ICML Workshop on Challenges in Representation Learning, 2013.Google Scholar
- X. Wang, J. Jia, and L. Cai. Affective image adjustment with a single word. The Visual Computer, 29(11):1121--1133, 2013. Google ScholarDigital Library
- C. Xu, S. Cetintas, K.-C. Lee, and L.-J. Li. Visual sentiment prediction with deep convolutional neural networks. arXiv preprint arXiv:1411.5731, 2014.Google Scholar
- V. Yanulevskaya, J. van Gemert, K. Roth, A. Herbold, N. Sebe, and J. M. Geusebroek. Emotional valence categorization using holistic image features. In ICIP, 2008.Google ScholarCross Ref
- Q. You, J. Luo, H. Jin, and J. Yang. Robust image sentiment analysis using progressively trained and domain transferred deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), 2015.Google ScholarDigital Library
- M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014, pages 818--833. Springer, 2014.Google ScholarCross Ref
Index Terms
- Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction
Recommendations
The Role of Visual Attention in Sentiment Prediction
MM '17: Proceedings of the 25th ACM international conference on MultimediaAutomated assessment of visual sentiment has many applications, such as monitoring social media and facilitating online advertising. In current research on automated visual sentiment assessment, images are mainly input and processed as a whole. However, ...
A query-based multi-document sentiment summarizer
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementReview websites, such as Epinions.com, which offer users a platform to share their opinions on diverse products and services, provide a valuable source of opinion-rich information. Browsing through archived reviews to locate different opinions on a ...
Intelligence system for sentiment classification with deep topic embedding using N-gram based topic modeling
Multi-modal information outbreak is consistently increasing in social media. Classification of tweet sentiments using various information modalities will help the recommender systems to achieve success in digital marketing. Moreover, aspect-level ...
Comments