Skip to main content
SpringerLink
Log in

Salient object based visual sentiment analysis by combining deep features and handcrafted features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the rapid growth of social networks, the visual sentiment analysis has quickly emerged for opinion mining. Recent study reveals that the sentiments conveyed by some images are related to salient objects in them, we propose a scheme for visual sentiment analysis that combines deep and handcrafted features. First, the salient objects are identified from the entire images. Then a pre-trained model such as VGG16 is used to extract deep features from the salient objects. In addition, hand-crafted features such as Visual texture, Colourfulness, Complexity and Fourier Sigma are extracted from all the salient objects. Deep features are combined individually with all the handcrafted features and the performance is measured. The sentiment is predicted using Convolutional Neural Network Classifier. The proposed method is tested on ArtPhoto, Emotion6, Abstract, IAPS datasets, Flickr and Flickr & Instagram datasets. The experimental results substantially proved that the proposed method achieves higher accuracy than other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ali AR, Shahid U, Ali M, Ho J (2017)High-level concepts for affective understanding of images. In 2017 IEEE winter conference on applications of computer vision (WACV) (pp. 679-687). IEEE.

  2. Andrienko YA, Brilliantov NV, Kurths J (2000) Complexity of two dimensional patterns. Eur Phys J B 15:539–546. https://doi.org/10.1007/s100510051157

    Article  Google Scholar 

  3. Barla, A., Franceschi, E., Odone, F., and Verri, A. (2002). “Image Kernels,” in Pattern Recognition with Support Vector Machines: First International Workshop, SVM 2002 Niagara Falls, Canada, August 10, 2002 Proceedings, eds S.-W. Lee and A. Verri (Berlin: Springer), 83–96. https://doi.org/10.1007/3-540-45665-1_7

  4. Birkhoff G (1933) Aesthetic measure. Harvard University Press, Cambridge

    Book  Google Scholar 

  5. D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, “Large-scale visual sentiment ontology and detectors using adjective noun pairs,” in Proceedings of the 21st ACM international conference on Multimedia. ACM, 2013, pp. 223–232.

  6. Borth D, Ji R, Chen T, Breuel T, Chang SF (2013)Large-scale visual sentiment ontology and detectors using adjective noun pairs. In proceedings of the 21st ACM international conference on multimedia (pp. 223-232).

  7. Braun J, Amirshahi SA, Denzler J, Redies C (2013) Statistical image properties of print advertisements, visual artworks and images of architecture. Front Psychol 4:808. https://doi.org/10.3389/fpsyg.2013.00808

    Article  Google Scholar 

  8. V. Campos, A. Salvador, X. Giro-i Nieto, and B. Jou, “Diving deep into sentiment: Understanding fine-tuned cnns for visual sentiment prediction,” in Proceedings of the 1st International Workshop on Affect & Sentiment in Multimedia. ACM, 2015, pp. 57–62.

  9. T. Chen, D. Borth, T. Darrell, and S.-F. Chang, “Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks,” arXiv preprint arXiv:1410.8586, 2014.

  10. Z. Cheng, Q. Yang, and B. Sheng, “Deep colorization,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 415–423.

  11. Dalal, N., and Triggs, B. (2005). “Histograms of oriented gradients for human detection,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (Piscataway, NJ: IEEE), 886–893. https://doi.org/10.1109/CVPR.2005.177

  12. Fu K, Gu IYH, Yang J (2018) Spectral salient object detection. Neurocomputing 275:788–803

    Article  Google Scholar 

  13. Hanjalic A (2006) Extracting moods from pictures and sounds: towards truly personalized tv. IEEE Signal Process Mag 23(2):90–100

    Article  Google Scholar 

  14. Hasler, D., and Süsstrunk, S. E. (2003). “Measuring colorfulness in natural images,” in Human Vision and Electronic Imaging VIII, eds B. E. Rogowitz and N. P. Thrasyvoulos (Santa Clara, CA: The International Society for Optical Engineering), 87–95. https://doi.org/10.1117/12.477378

  15. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conf. Comput. Vis. Pattern Recog., 2016.

  16. He X, Zhang H, Li N, Feng L, Zheng F (2019, July) A multi-attentive pyramidal model for visual sentiment analysis. In 2019 international joint conference on neural networks (IJCNN) (pp. 1-8). IEEE.

  17. Joshi D, Datta R, Fedorovskaya E, Luong Q-T, Wang JZ, Li J, Luo J (2011) Aesthetics and emotions in images. IEEE Signal Proc Mag 28(5):94–115

    Article  Google Scholar 

  18. B. Jou, S. Bhattacharya, and S.-F. Chang, “Predicting viewer perceived emotions in animated GIFs,” in ACM Int. Conf. Multimedia, 2014.

  19. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.

  20. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.

  21. X. Lu, P. Suryanarayan, R. B. Adams Jr, J. Li, M. G. Newman, and J. Z. Wang, “On shape and the computability of emotions,” in Proceedings of the 20th ACM international conference on Multimedia. ACM, 2012, pp. 229–238.

  22. J. Machajdik and A. Hanbury, “Affective image classification using features inspired by psychology and art theory,” in Proceedings of the 18th ACM international conference on Multimedia. ACM, 2010, pp. 83–92.

  23. Mikels JA, Fredrickson BL, Larkin GR, Lindberg CM, Maglio SJ, Reuter-Lorenz PA (2005) Emotional category data on images from the international affective picture system. Behav Res Methods 37(4):626–630

    Article  Google Scholar 

  24. M. A. Nicolaou, H. Gunes, and M. Pantic, “A multi-layer hybrid framework for dimensional emotion classification,” in ACM Int. Conf. Multimedia, 2011.

  25. K.-C. Peng, T. Chen, A. Sadovnik, and A. Gallagher (2005) A Mixed Bag of Emotions: Model, Predict, and Transfer Emotion Distributions. In 2015 IEEE conference on computer vision and pattern recognition (CVPR), pages 860–868. IEEE.

  26. Proulx R, Parrott L (2008) Measures of structural complexity in digital images for monitoring the ecological signature of an old-growth forest ecosystem. Ecol Indic 8:270–284. https://doi.org/10.1016/j.ecolind.2007.02.005

    Article  Google Scholar 

  27. Redies, C., Amirshahi, S. A., Koch, M., and Denzler, J. (2012). “PHOG-Derived aesthetic measures applied to color photographs of artworks, natural scenes and objects,” in Computer Vision – ECCV 2012. Workshops and Demonstrations: Florence, Italy, October 7–13, 2012, Proceedings, Part I, eds A. Fusiello, V. Murino, and R. Cucchiara (Berlin: Springer), 522–531. https://doi.org/10.1007/978-3-642-33863-2_54

  28. Rosenholtz R, Li Y, Nakano T (2007) Measuring visual clutter. J Vis 7:1–22. https://doi.org/10.1167/7.2.17

    Article  Google Scholar 

  29. M. Solli and R. Lenz, “Color based bags-of-emotions,” in Int. Conf. Comput. Anal. Images Patterns, 2009.

  30. Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “HCP: a flexible CNN framework for multi-label image classification,” vol. 38, no. 9, pp. 1901–1907, 2016.

  31. Wu L, Qi M, Jian M, Zhang H (2019) Visual sentiment analysis by combining global and local information. Neural Processing Letters, pp:1–13

  32. Xiong H, Liu Q, Song S, Cai Y (2019)Region-based convolutional neural network using group sparse regularization for image sentiment classification. EURASIP J Image Video Processing 2019(1):1–9

    Article  Google Scholar 

  33. Yang J, She D, Sun M (2017) Joint image emotion classification and distribution learning via deep convolutional neural network. In IJCAI (pp. 3266-3272).

  34. Yang J, She D, Sun M, Cheng MM, Rosin PL, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimedia 20(9):2513–2525

    Article  Google Scholar 

  35. Yang J, She D, Lai YK, Rosin PL, Yang MH (2018) Weakly supervised coupled networks for visual sentiment analysis. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7584-7592).

  36. You, Q., Luo, J., Jin, H., & Yang, J. (2016). Building a large scale dataset for image emotion recognition: the fine print and the benchmark. arXiv preprint arXiv:1605.02677.

  37. Zhan, C., She, D., Zhao, S., Cheng, M.M. and Yang, J., 2019. Zero-shot emotion recognition via affective structural embedding. In proceedings of the IEEE international conference on computer vision (pp. 1151-1160).

  38. S. Zhao, Y. Gao, X. Jiang, H. Yao, T.-S. Chua, and X. Sun, “Exploring principles-of-art features for image emotion recognition,” in Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2014, pp. 47–56.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, St. Mary’s College (Autonomous), Thoothukudi, Tamilnadu, India

    S. Sowmyayani

  2. Department of Computer Science and Engineering, Manonmaniam Sundaranar University, Tirunelveli, Tamilnadu, India

    P. Arockia Jansi Rani

Authors
  1. S. Sowmyayani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Arockia Jansi Rani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Sowmyayani.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sowmyayani, S., Rani, P.A.J. Salient object based visual sentiment analysis by combining deep features and handcrafted features. Multimed Tools Appl 81, 7941–7955 (2022). https://doi.org/10.1007/s11042-022-11982-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-11982-5

Keywords

Search

Navigation