nach oben

Erschienen in:

2018 | OriginalPaper | Buchkapitel

Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias

verfasst von : Rameswar Panda, Jianming Zhang, Haoxiang Li, Joon-Young Lee, Xin Lu, Amit K. Roy-Chowdhury

Erschienen in: Computer Vision – ECCV 2018

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

While machine learning approaches to visual emotion recognition offer great promise, current methods consider training and testing models on small scale datasets covering limited visual emotion concepts. Our analysis identifies an important but long overlooked issue of existing visual emotion benchmarks in the form of dataset biases. We design a series of tests to show and measure how such dataset biases obstruct learning a generalizable emotion recognition model. Based on our analysis, we propose a webly supervised approach by leveraging a large quantity of stock image data. Our approach uses a simple yet effective curriculum guided training strategy for learning discriminative emotion features. We discover that the models learned using our large scale stock image dataset exhibit significantly better generalization ability than the existing datasets without the manual collection of even a single label. Moreover, visual representation learned using our approach holds a lot of promise across a variety of tasks on different image and video datasets.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Textual Explanations for Self-Driving Vehicles

Nächstes Kapitel Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks

Nur mit Berechtigung zugänglich

The image is taken from Google Images with the search keyword sad amusement park. Source: https://goo.gl/AUwoPZ.

All our datasets, models and supplementary material are publicly available on our project page: https://rpand002.github.io/emotion.html.

For example, see: https://tinyurl.com/yazvkjmv.

Alameda-Pineda, X., Ricci, E., Yan, Y., Sebe, N.: Recognizing emotions from abstract paintings using non-linear matrix completion. In: CVPR (2016)

Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: ICML (2009)

Cesa-Bianchi, N., Gentile, C., Zaniboni, L.: Incremental algorithms for hierarchical classification. JMLR 7, 31–54 (2006)MathSciNetMATH

Chen, T., Borth, D., Darrell, T., Chang, S.F.: Deepsentibank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586 (2014)

Chen, X., Gupta, A.: Webly supervised learning of convolutional networks. In: ICCV (2015)

Chu, W.S., Song, Y., Jaimes, A.: Video co-summarization: video summarization by visual co-occurrence. In: CVPR (2015)

Chu, W.S., De la Torre, F., Cohn, J.F.: Selective transfer machine for personalized facial expression analysis. TPAMI 39, 529–545 (2017)CrossRef

Deng, J., Krause, J., Berg, A.C., Fei-Fei, L.: Hedging your bets: optimizing accuracy-specificity trade-offs in large scale visual recognition. In: CVPR (2012)

Deng, J., Satheesh, S., Berg, A.C., Li, F.: Fast and balanced: efficient label tree learning for large scale object recognition. In: NIPS (2011)

10.

Divvala, S.K., Farhadi, A., Guestrin, C.: Learning everything about anything: webly-supervised visual concept learning. In: CVPR (2014)

11.

Dong, Q., Gong, S., Zhu, X.: Multi-task curriculum transfer deep learning of clothing attributes. In: WACV (2017)

12.

Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. In: Proceedings of the National Academy of Sciences (2014)

13.

Ekman, P.: An argument for basic emotions. Cogn. Emot. 6, 169–200 (1992)CrossRef

14.

Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. In: TIP (2015)

15.

Eleftheriadis, S., Rudovic, O., Pantic, M.: Joint facial action unit detection and feature fusion: a multi-conditional learning approach. In: TIP (2016)

16.

Fabian Benitez-Quiroz, C., Srinivasan, R., Martinez, A.M.: Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: CVPR (2016)

17.

Gan, C., Sun, C., Duan, L., Gong, B.: Webly-supervised video recognition by mutually voting for relevant web images and web video frames. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 849–866. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_52CrossRef

18.

Gan, C., Yao, T., Yang, K., Yang, Y., Mei, T.: You lead, we exceed: labor-free video concept learning by jointly exploiting web videos and images. In: CVPR (2016)

19.

Gao, R., Grauman, K.: On-demand learning for deep image restoration. In: ICCV (2017)

20.

Griffin, G., Perona, P.: Learning and using taxonomies for fast visual categorization. In: CVPR, pp. 1–8 (2008)

21.

Gygli, M., Grabner, H., Riemenschneider, H., Van Gool, L.: Creating summaries from user videos. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 505–520. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_33CrossRef

22.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

23.

Hussain, Z., et al.: Automatic understanding of image and video advertisements. In: CVPR (2017)

24.

Jia, Y., Abbott, J.T., Austerweil, J.L., Griffiths, T., Darrell, T.: Visual concept learning: combining machine vision and Bayesian generalization on concept hierarchies. In: NIPS (2013)

25.

Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: MM (2014)

26.

Joo, J., Li, W., Steen, F.F., Zhu, S.C.: Visual persuasion: inferring communicative intents of images. In: CVPR (2014)

27.

Jou, B., Bhattacharya, S., Chang, S.F.: Predicting viewer perceived emotions in animated gifs. In: MM (2014)

28.

Joulin, A., van der Maaten, L., Jabri, A., Vasilache, N.: Learning visual features from large weakly supervised data. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 67–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_5CrossRef

29.

Kahou, S.E., et al.: Combining modality specific deep neural networks for emotion recognition in video. In: ICMI (2013)

30.

Kim, H.R., Kim, S.J., Lee, I.K.: Building emotional machines: recognizing image emotions through deep neural networks. arXiv preprint arXiv:1705.07543 (2017)

31.

Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: CVPR (2017)

32.

Krause, J., et al.: The unreasonable effectiveness of noisy data for fine-grained recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 301–320. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_19CrossRef

33.

Lee, Y.J., Grauman, K.: Learning the easy things first: self-paced visual category discovery. In: CVPR (2011)

34.

Li, L.J., Wang, C., Lim, Y., Blei, D.M., Fei-Fei, L.: Building and using a semantivisual image hierarchy. In: CVPR (2010)

35.

Li, W., Wang, L., Li, W., Agustsson, E., Van Gool, L.: Webvision database: visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 (2017)

36.

Liang, J., Jiang, L., Meng, D., Hauptmann, A.: Exploiting multi-modal curriculum in noisy web data for large-scale concept learning. arXiv preprint arXiv:1607.04780 (2016)

37.

Liang, J., Jiang, L., Meng, D., Hauptmann, A.G.: Learning to detect concepts from webly-labeled video data. In: IJCAI (2016)

38.

Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: MM (2010)

39.

Marszalek, M., Schmid, C.: Semantic hierarchies for visual object recognition. In: CVPR (2007)

40.

Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: ICMI (2015)

41.

Panda, R., Roy-Chowdhury, A.K.: Collaborative summarization of topic-related videos. In: CVPR (2017)

42.

Parrott, W.G.: Emotions in Social Psychology: Essential Readings. Psychology Press, Hove (2001)

43.

Peng, K.C., Chen, T., Sadovnik, A., Gallagher, A.C.: A mixed bag of emotions: model, predict, and transfer emotion distributions. In: CVPR (2015)

44.

Pentina, A., Sharmanska, V., Lampert, C.H.: Curriculum learning of multiple tasks. In: CVPR (2015)

45.

Plutchik, R., Kellerman, H.: Emotion, Theory, Research, and Experience: Theory, Research and Experience. Academic Press, San Diego (1980)

46.

Silla Jr., C.N., Freitas, A.A.: A survey of hierarchical classification across different application domains. Data Min. Knowl. Discov. 22, 31–72 (2011)MathSciNetCrossRef

47.

Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of EEG signals and facial expressions for continuous emotion detection. In: TAC (2016)

48.

Srivastava, N., Salakhutdinov, R.R.: Discriminative transfer learning with tree-based priors. In: NIPS (2013)

49.

Sukhbaatar, S., Fergus, R.: Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080 (2014)

50.

Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: ICCV (2017)

51.

Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR (2011)

52.

Tousch, A.M., Herbin, S., Audibert, J.Y.: Semantic hierarchies for image annotation: a survey. Pattern Recognit. 45, 333–345 (2012)CrossRef

53.

Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: ICCV (2015)

54.

Verma, N., Mahajan, D., Sellamanickam, S., Nair, V.: Learning hierarchical similarity metrics. In: CVPR (2012)

55.

Wang, D., Shen, Z., Shao, J., Zhang, W., Xue, X., Zhang, Z.: Multiple granularity descriptors for fine-grained categorization. In: ICCV (2015)

56.

Wang, X., Jia, J., Tang, J., Wu, B., Cai, L., Xie, L.: Modeling emotion influence in image social networks. TAC 6, 286–297 (2015)

57.

Wu, B., Jia, J., Yang, Y., Zhao, P., Tang, J., Tian, Q.: Inferring emotional tags from social images with user demographics. TMM 19, 1670–1684 (2017)

58.

Xiao, T., Zhang, J., Yang, K., Peng, Y., Zhang, Z.: Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In: MM (2014)

59.

Xu, B., Fu, Y., Jiang, Y.G., Li, B., Sigal, L.: Heterogeneous knowledge transfer in video emotion recognition, attribution and summarization. TAC 9, 255–270 (2016)

60.

Xu, B., Fu, Y., Jiang, Y.G., Li, B., Sigal, L.: Video emotion recognition with ferred deep feature encodings. In: ICMR (2016)

61.

Yan, Z., et al.: HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In: ICCV (2015)

62.

You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: AAAI (2015)

63.

You, Q., Luo, J., Jin, H., Yang, J.: Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: AAAI (2016)

64.

Zhang, Y., David, P., Gong, B.: Curriculum domain adaptation for semantic segmentation of urban scenes. arXiv preprint arXiv:1707.09465 (2017)

65.

Zhao, S., Gao, Y., Jiang, X., Yao, H., Chua, T.S., Sun, X.: Exploring principles-of-art features for image emotion recognition. In: MM (2014)

Titel: Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias
verfasst von: Rameswar Panda
Jianming Zhang
Haoxiang Li
Joon-Young Lee
Xin Lu
Amit K. Roy-Chowdhury
Verlag: Springer International Publishing
Buch: Computer Vision – ECCV 2018
Print ISBN: 978-3-030-01215-1

Electronic ISBN: 978-3-030-01216-8

Copyright-Jahr: 2018
DOI: https://doi.org/10.1007/978-3-030-01216-8_36

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"