Top

Neural Processing Letters

Published in:

04-04-2019

Visual Sentiment Analysis by Combining Global and Local Information

Authors: Lifang Wu, Mingchao Qi, Meng Jian, Heng Zhang

Published in: Neural Processing Letters | Issue 3/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

With the development of visual social networks, the sentiment analysis of images has quickly emerged for opinion mining. Based on the observation that the sentiments conveyed by some images are related to salient objects in them, we propose a scheme for visual sentiment analysis that combines global and local information. First, the sentiment is predicted from the entire images. Second, it is judged whether there are salient objects in an image or not. If there are, sub-images are cropped from the entire image based on the detection window of the salient objects. Moreover, a CNN model is trained for the set of sub-images. Predictions of sentiments from entire images and sub-images are then fused together to obtain the final results. If no salient object is detected in the images, the sentiment predicted directly from entire images is used as the final result. The compared experimental results show that the proposed approach is superior to state-of-the-art algorithms. It also demonstrates that reasonably utilizing the local information could improve the performance for visual sentiment analysis.

previous article Learning Multi-level Deep Representations for Image Emotion Classification

next article Deep Transfer Learning for Image Emotion Analysis: Reducing Marginal and Joint Distribution Discrepancies Together

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Zhao S, Yao H, Yue G, Ding G, Chua TS (2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9:526–540. https://doi.org/10.1109/TAFFC.2016.2628787 CrossRef

Yang J, Sun M, Sun X (2017) Learning visual sentiment distributions via augmented conditional probability neural network. In: AAAI Conference on Artificial Intelligence, North America, February 2017. https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14506. Accessed 03 Apr 2019

Kosti R, Alvarez JM, Recasens A, Lapedriza A (2017) Emotion recognition in context. In: IEEE conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2017.212

Zhao S, Yue G, Ding G, Chua TS (2017) Real-time multimedia social event detection in microblog. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2017.2762344 CrossRef

Zhao S, Ding G, Huang Q, Chua TS, Bjorn WS, Kurt K (2018) Affective image content analysis: a comprehensive survey. In: IJCAI

Joshi D, Datta R, Fedorovskaya E, Luong QT (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28:94–115. https://doi.org/10.1109/MSP.2011.941851 CrossRef

Zhao S, Gao Y, Jiang X, Yao H, Chua TS, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: ACM int. conf. multimedia, pp 47–56. https://doi.org/10.1145/2647868.2654930

Zhang H, Augilius E, Honkela T, Laaksonen J, Gamper H, Alene H (2011) Analyzing emotional semantics of abstract art using low-level image features. In: International conference on advances in intelligent data analysis X, LNCS, vol 7014, pp 413–423. https://doi.org/10.1007/978-3-642-24800-9_38

Li B, Feng S, Xiong W, Hu W (2012) Scaring or pleasing: exploit emotional impact of an image. In: ACM international conference on multimedia, pp 1365–1366. https://doi.org/10.1145/2393347.2396487

10.

Liu N, Dellandra E, Tellez B, Chen L (2011) Associating textual features with visual ones to improve affective image classification. Springer, Berlin, pp 229–238. https://doi.org/10.1007/978-3-642-24600-5_23 CrossRef

11.

Lu X, Suryanarayan P, Li J, Newman MG, Wang JZ (2012) On shape and the computability of emotions. In: ACM international conference on multimedia, pp 229–238. https://doi.org/10.1145/2393347.2393384

12.

Yuan J, Mcdonough S, You Q, Luo J (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the second international workshop on issues of sentiment discovery and opinion mining. https://doi.org/10.1145/2502069.2502079

13.

Rao T, Xu M, Liu H, Wang J, Burnett I (2016) Multi-scale blocks based image emotion classification using multiple instance learning. IEEE Int Conf Image Process. https://doi.org/10.1109/ICIP.2016.7532434 CrossRef

14.

Borth D, Ji R, Chen T, Breuel T, Chang SF (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: 21st ACM international conference on Multimedia, pp 223–232. https://doi.org/10.1145/2502081.2502282

15.

Zhao S, Yao H, Yue G, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multi-task shared sparse regression. IEEE Trans Multimed 19:632–645. https://doi.org/10.1109/TMM.2016.2617741 CrossRef

16.

Zhao S, Ding G, Yue G, Xin Z, Tang Y, Han J, Yao H, Huang Q (2018) Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2018.2818685 CrossRef

17.

Xu C, Cetintas S, Lee KC, Li LJ (2014) Visual sentiment prediction with deep convolutional neural networks. Eprint ArXiv arXiv:1411.5731

18.

You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Twenty-ninth AAAI conference on artificial intelligence, pp 381–388. arXiv:1509.06041

19.

Wu L, Liu S, Jian M, Luo J, Zhang X, Qi M (2018) Reducing noisy labels in weakly labeled data for visual sentiment analysis. In: IEEE international conference on image processing, pp 1322–1326. https://doi.org/10.1109/ICIP.2017.8296496

20.

Zheng H, Chen T, Luo J (2016) When saliency meets sentiment: understanding how image content invokes emotion and sentiment, pp 630–634. arXiv:1611.04636

21.

Fan S, Jiang M, Shen Z et al (2017) The role of visual attention in sentiment prediction. In: Proceedings of the 2017 ACM on multimedia conference. ACM, pp 217–225. https://doi.org/10.1145/3123266.3123445

22.

Fan S, Shen Z, Jiang M, Koenig B, Xu J, Kankanhalli M, Zhao Q (2018) Emotional attention: a study of image sentiment and visual attention. In: The IEEE conference on computer vision and pattern recognition (CVPR)

23.

You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions. In: AAAI conf. artif. intell, pp 231–237

24.

Yang J, She D, Sun M, Cheng MM, Rosin P, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimed 20:2513–2525. https://doi.org/10.1109/TMM.2018.2803520 CrossRef

25.

Zhang J, Sclaroff S, Lin Z, Shen X, Price B, Mech R (2016) Unconstrained salient object detection via proposal subset optimization. In: Computer vision and pattern recognition, pp 5733–5742. https://doi.org/10.1109/CVPR.2016.618

26.

You Q, Luo J, Jin H, Yang J (2016) Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: AAAI conf. artif. intel, pp 308–314. arXiv:1605.02677

27.

Simonyan K, Zisserman (2014) A very deep convolutional networks for large-scale image recognition. Comput Sci. arXiv:1409.1556v6

28.

Peng KC, Sadovnik A, Gallagher A, Chen T (2016) Where do emotions come from? Predicting the emotion stimuli map. In: IEEE international conference on image processing. https://doi.org/10.1109/ICIP.2016.7532430

29.

Chen T, Borth D, Darrell T, Chang SF (2014) Deep SentiBank: visual sentiment concept classification with deep convolutional neural networks. arXiv:1410.8586

Title: Visual Sentiment Analysis by Combining Global and Local Information
Authors: Lifang Wu
Mingchao Qi
Meng Jian
Heng Zhang
Publication date: 04-04-2019
Publisher: Springer US
Published in: Neural Processing Letters / Issue 3/2020
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI: https://doi.org/10.1007/s11063-019-10027-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2020

Visual Sentiment Prediction with Attribute Augmentation and Multi-attention Mechanism

Multi-weighted Complex Structure on Fractional Order Coupled Neural Networks with Linear Coupling Delay: A Robust Synchronization Problem

Inferring Personality Traits from Attentive Regions of User Liked Images Via Weakly Supervised Dual Convolutional Network

On Impulsive Synchronization Control for Coupled Inertial Neural Networks with Pinning Control

Legendre Neural Network Method for Several Classes of Singularly Perturbed Differential Equations Based on Mapping and Piecewise Optimization Technology

Selective Embedding with Gated Fusion for 6D Object Pose Estimation