Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
ATZ-Fachkongress

Chassis.tech plus 2024


4 Kongresse in einer Veranstaltung

Treffen Sie in München die Fachleute von Chassis, Lenkung, Bremsen sowie Reifen/Räder.
Erweiterte Suche
Anmelden
nach oben
Journal on Multimodal User Interfaces
Erschienen in: Journal on Multimodal User Interfaces 1/2014

01.03.2014 | Original Paper

Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements

verfasst von: Ingo Siegert, Ronald Böck, Andreas Wendemuth

Erschienen in: Journal on Multimodal User Interfaces | Ausgabe 1/2014

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

To enable a naturalistic human–computer interaction the recognition of emotions and intentions experiences increased attention and several modalities are comprised to cover all human communication abilities. For this reason, naturalistic material is recorded, where the subjects are guided through an interaction with crucial points, but with the freedom to react individually. This material captures realistic user reactions but lacks of clear labels. So, a good transcription and annotation of the given material is essential. For that, the assignment of human annotators has become widely accepted. A good measurement for the reliability of labelled material is the inter-rater agreement. In this paper we investigate the achieved inter-rater agreement utilizing Krippendorff’s alpha for emotional annotated interaction corpora and present methods to improve the reliability, we show that the reliabilities obtained with different methods does not differ much, so a choice could rely on other aspects. Furthermore, a multimodal presentation of the items in their natural order increases the reliability.
Vorheriger Artikel Using unlabeled data to improve classification of emotional states in human computer interaction
Nächster Artikel An audio-visual dataset of human–human interactions in stressful situations

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Literatur
1.
Altman DG (1991) Practical statistics for medical research. Chapman & Hall, London
2.
Artstein R, Poesio M (2008) Inter-coder agreement for computational linguistics. Comput Linguist 34(4):555–596
3.
Batliner A, Hacker C, Steidl S, Nöth E, Russell M, Wong M (2004) “You stupid tin box”-children interacting with the AIBO robot: a cross-linguistic emotional speech corpus. In: Proceedings of LREC, pp 865–868
4.
Böck R, Siegert I, Vlasenko B, Wendemuth A, Haase M, Lange J (2011) A processing tool for emotionally coloured speech. In: Proceedings of ICME, s.p.
5.
Bradley M, Lang P (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psy 25(1):49–59
6.
Burger S, MacLaren V, Yu H (2002) The ISL meeting corpus: the impact of meeting type on speech style. In: Proceedings of the international conference on spoken language processing, pp 301–304
7.
Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of german emotional speech. In: Proceedings of interspeech, pp 1517–1520
8.
Callejas Z, Lpez-Czar R (2008) Influence of contextual information in emotion annotation for spoken dialogue systems. Speech Commun 50(5):416–433
9.
Cauldwell RT (2000) Where did the anger go? The role of context in interpreting emotion in speech. In: Proceedings of ITRW on speech and, emotion, pp 127–131
10.
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 24(1):37–46CrossRef
11.
Cowie R, Cornelius RR (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1–2):5–32CrossRefMATH
12.
Crawford JR, Henry JD (2004) The positive and negative affect schedule (PANAS): construct validity, measurement properties and normative data in a large non-clinical sample. Br J Clin Psychol 43(3):245–265
13.
Cronbach L (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297–334
14.
Devillers L, Vasilescu I (2004) Reliability of lexical and prosodic cues in two real-life spoken dialog corpora. In: Proceedings of LREC, pp 865–868
15.
Devillers L, Vidrascu L, Lamel L (2005) Challenges in real-life emotion annotation and machine learning based detection. Neural Netw 18(4):407–422CrossRef
16.
Douglas-Cowie E, Cowie R, Schröder M (2000) A new emotion database: considerations, sources and scope. In: Proceedings of ITRW on speech and, emotion, pp 39–44
17.
Douglas-Cowie E, Cowie R, Sneddon I, Cox C, Lowry O, McRorie M, Martin JC, Devillers L, Abrilian S, Batliner A, Amir N, Karpouzis K (2007) The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data. In: Proceedings of ACII. Berlin, Heidelberg, pp 488–500
18.
Douglas-Cowie E, Devillers L, Martin JC, Cowie R, Savvidou S, Abrilian S, Cox C (2005) Multimodal databases of everyday emotion: facing up to complexity. In: Proceedings of EUROSPEECH, pp 813–816
19.
Eggink J, Bland D (2012) A large scale experiment for mood-based classification of TV programmes. In: Proceedings of ICME, pp 140–145
20.
Ekman P (1992) Are there basic emotions? Psychol Rev 99(3):550–553
21.
El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit 44(3):572–587CrossRefMATH
22.
Engberg IS, Hansen AV (1996) Documentation of the danish emotional speech database (DES). Technical report, Center for Person, Kommunikation, Aalborg University, Denmark . Internal aau report
23.
Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 43(6):543–549CrossRef
24.
Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382
25.
Fleiss JL, Levin B, Paik MC (1991) Statistical methods for rates & proportions, 3rd edn. Wiley, Hoboken
26.
Fragopanagos N, Taylor J (2005) Emotion recognition in human-computer interaction. Neural Netw 18(4):389–405CrossRef
27.
Frommer J, Michaelis B, Rösner D, Wendemuth A, Friesen R, Haase M, Kunze M, Andrich R, Lange J, Panning A, Siegert I (2012) Towards emotion and affect detection in the multimodal LAST MINUTE corpus. In: Proceedings of LREC, pp 3064–3069
28.
Frommer J, Rösner D, Haase M, Lange J, Friesen R, Otto M (2012) Detection and avoidance of failures in dialogues-Wizard of Oz Experiment Operator’s Manual. Pabst Science Publishers
29.
Gehm T, Scherer K (1988) Factors determining the dimensions of subjective emotional space. In: Scherer K (ed) Facets of emotion: recent research. Erlbaum, Hillsdale, NJ, pp 99–114
30.
Gnjatović M, Rösner D (2008) The NIMITEK corpus of affected behavior in human-machine interaction. In: Proceedings of LREC, pp 5–8
31.
Grandjean D, Sander D, Scherer K (2008) Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Conscious Cogn 17(2):484–495CrossRef
32.
Grimm M, Kroschel K (2005) Evaluation of natural emotions using self assessment manikins. In: IEEE workshop on automatic speech recognition and understanding, pp 381–385
33.
Grimm M, Kroschel K, Narayanan S (2008) The vera am mittag german audio-visual emotional speech database. In: Proceedings of ICME, pp 865–868
34.
Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61(1):29–48CrossRefMathSciNet
35.
Gwet KL (2008) Intrarater reliability. In: D’Agostino RB, Sullivan L, Massaro J (eds) Wiley encyclopedia of clinical trials. Wiley, Hoboken, pp 473–485
36.
Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 24(1):77–89CrossRef
37.
Ibáñez J (2011) Showing emotions through movement and symmetry. Comput Hum Behav 27(1):561–567
38.
Izard CE, Libero DZ, Putnam P, Haynes OM (1993) Stability of emotion experiences and their relations to traits of personality. J Pers Soc Psychol 64(5):847–860
39.
Krippendorff K (2007) Computing Krippendorff’s alpha reliability. University of Pennsylvania, Annenberg School for Communication, Technical report
40.
Krippendorff K (2012) Content analysis: an introduction to its methodology, 3rd edn. SAGE Publications, Thousand Oaks
41.
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174
42.
Lang PJ (1980) Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski JB, Johnson JH, Williams TA (eds) Technology in mental health care delivery systems. Ablex Pub. Corp., pp 119–137
43.
Lee CM, Narayanan S (2005) Toward detecting emotions in spoken dialogs. IEEE Trans Speech Audio Process 13(2):293–303CrossRef
44.
McDougall W (1926) An introduction to social psychology, revised edn. John W. Luce & Co, Boston
45.
McKeown G, Valstar M, Cowie R, Pantic M (2010) The semaine corpus of emotionally coloured character interactions. In: Proceedings of ICME, pp 1079–1084
46.
McKeown G, Valstar M, Cowie R, Pantic M, Schroder M (2012) The semaine database: annotated multimodal records of emotionally coloured conversations between a person and a limited agent. IEEE Trans Affect Comput 3(1):5–17
47.
Mehrabian A (1970) A semantic space for nonverbal behavior. J Consult Clin Psychol 35(2):248–257
48.
Morris JD (1995) SAM: the self-assessment manikin an efficient cross-cultural measurement of emotional response. J Advert Res 35(6):63–68
49.
Morris JD, McMullen JS (1994) Measuring multiple emotional responses to a single television commercial. Adv Consum Res 21:175–180CrossRef
50.
Osgood CE, Miron MS, May WH (1975) Cross-cultural universals of affective meaning. University of Illinois Press, Urbana
51.
Plutchik R (1980) Emotion, a psychoevolutionary synthesis. Harper & Row, New York
52.
Pugmire D (1994) Real emotion. Philos Phenomen Res 54(1):105–122
53.
Rösner D, Friesen R, Otto M, Lange J, Haase M, Frommer J (2011) Intentionality in interacting with companion systems G an empirical approach. In: Human-Computer interaction. Towards mobile and intelligent interaction environments, LNCS, vol 6763. Springer, Berlin, Heidelberg, pp 593–602
54.
Russel J, Mehrabian A (1974) Distinguishing anger and anxiety in terms of emotional response factors. J Consult Clin Psychol 42:79–83CrossRef
55.
Russel JA (1980) Three dimensions of emotion. J Pers Soc Psychol 39(9):1161–1178CrossRef
56.
Sacharin V, Schlegel K, Scherer KR (2012) Geneva emotion wheel rating study. Center for Person, Kommunikation, Aalborg University, NCCR Affective Sciences, Technical report
57.
Scherer K (2005) What are emotions? and how can they be measured? Soc Sci Inform 44(4):695–729CrossRef
58.
Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking, vol 92. Oxford University Press, Oxford, pp. 92–120
59.
Schimmack U (1997) The Berlin everyday language mood inventory (BELMI): toward the content valid assessment of moods. Diagnostica 43(2):150–173
60.
Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4):350–353
61.
Schröder M, Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M (2000) Feeltrace: An instrument for recording perceived emotion in real time. In: Proceedings of ITRW on speech and, emotion, pp 19–24
62.
Sharp H, Rogers Y, Preece J (2007) Interaction design: beyond human-computer interaction, 2nd edn. Wiley, London
63.
Siegert I, Böck R, Wendemuth A (2013) The influence of context knowledge for multimodal affective annotation. In: Human-computer interaction, Part V, HCII 2013, LNCS, vol 8008. Springer, Berlin, pp 381–390
64.
Siegert I, Böck R, Philippou-Hübner D, Vlasenko B, Wendemuth A (2011) Appropriate emotional labeling of non-acted speech using basic emotions, Geneva emotion wheel and self assessment Manikins. In: Proceedings of ICME, s.p.
65.
Siegert I, Böck R, Wendemuth A (2012) The influence of context knowledge for multimodal annotation on natural material. In: Joint proceedings of the IVA 2012 workshops, pp 25–32
66.
67.
Sojka P, Horak A, Kopecek I, Pala K (eds) (2012) Aggression detection in speech using sensor and semantic information, vol 7499. Springer, Berlin
68.
Truong KP, van Leeuwen DA, de Jong FM (2012) Speech-based recognition of self-reported and observed emotion in a dimensional space. Speech Commun 54(9):1049–1063CrossRef
69.
Truong KP, Neerincx MA, van Leeuwen DA (2008) Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data. In: Proceedings of interspeech, pp 318–321
70.
Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54(6):1063–1070CrossRef
71.
Wendemuth A, Biundo S (2012) A companion technology for cognitive technical systems. In: Cognitive behavioural systems, Lecture Notes in Computer Science, vol 7403, Springer, Berlin, pp 89–103
72.
Wundt W (1922/1863) Vorlesungen über die Menschen- und Tierseele. L. Voss, Leipzig
73.
Yang YH, Lin YC, Su YF, Chen H (2007) Music emotion classification: a regression approach. In: Proceedings of ICME, pp 208–211
74.
Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1):39–58CrossRef
Metadaten
Titel
Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements
verfasst von
Ingo Siegert
Ronald Böck
Andreas Wendemuth
Publikationsdatum
01.03.2014
Verlag
Springer Berlin Heidelberg
Erschienen in
Journal on Multimodal User Interfaces / Ausgabe 1/2014
Print ISSN: 1783-7677
Elektronische ISSN: 1783-8738
DOI
https://doi.org/10.1007/s12193-013-0129-9

Weitere Artikel der Ausgabe 1/2014

Journal on Multimodal User Interfaces 1/2014 Zur Ausgabe

Original Paper

Analysis of significant dialog events in realistic human–computer interaction

Original Paper

Switching Wizard of Oz for the online evaluation of backchannel behavior

Original Paper

A model for incremental grounding in spoken dialogue systems

Original Paper

Facial expression-based affective speech translation

Original Paper

Using unlabeled data to improve classification of emotional states in human computer interaction

Original Paper

Recognizing signals of social attitude in interacting with Ambient Conversational Systems

Premium Partner

    Bildnachweise