Elsevier

Biological Psychology

Volume 86, Issue 3, March 2011, Pages 158-167
Biological Psychology

Event-related potential correlates of the expectancy violation effect during emotional prosody processing

https://doi.org/10.1016/j.biopsycho.2010.11.004Get rights and content

Abstract

The present study investigated the expectancy violation effects evoked by deviation in sentential emotional prosody (EP), and their association with the deviation patterns. Event-related potentials (ERPs) were recorded for mismatching EPs with different patterns of deviation and for matching control EPs while subjects performed emotional congruousness judgment in Experiment 1 and visual probe detection tasks in Experiment 2. In the control experiment, EPs and acoustically matched non-emotional materials were presented and ERPs were recorded while participants judged the sound intensity congruousness. It was found that an early negativity, whose peak latency varied with deviation pattern, was elicited by mismatching EPs relative to matching ones, irrespective of task-relevance. A late positivity was specifically induced by mismatching EPs, and was modulated by both deviation pattern and task-relevance. Moreover, these effects cannot be simply attributed to the change in non-emotional acoustic properties. These findings suggest that the brain detects the EP deviation rapidly, and then integrates it with context for comprehension, during which the emotionality plays a role of speeding up the perception and enhancing vigilance.

Research highlights

▶ We investigated the ERP correlates of expectancy violation effect during emotional prosody processing with three experiments. ▶ The brain is able to detect the deviation in sentential emotional prosody rapidly irrespective of attention allocation. ▶ Deviation with large emotional significance would increase the vigilance and assign significance during speech comprehension. ▶ A process of re-analyzing and integrating would take place if the deviation is task relevant. ▶ The emotionality seems to speed up the perception and step up more vigilance during this process.

Introduction

In addition to the verbal emotion conveyed by semantic content, a speaker expresses his or her emotion in prosody by manipulating acoustic parameters such as pitch, intensity, and speech rate, termed as emotional prosody (EP) or vocal emotion. Banse and Scherer (1996) found that each emotion has its own acoustic profile. For example, the vocalization of anger reveals a higher fundamental frequency (F0) and intensity than that of neutral utterance. Successful emotional communication is critical for social interaction. Particularly, it is adaptively important to detect EP deviation in time, since changes in EP are common in spoken interactions, and a sudden variation in prosody is a direct signal for the speakers’ emotion changes. Despite the fact that human beings are born with competence to process deviation in EP efficiently, the underlying neural mechanism remains unclear. Therefore, it is highly important to investigate the neural correlates of the perception of EP deviation and to further clarify the cognitive mechanism of vocal emotion perception.

The auditory perception is based on predictive representation of temporal regularities, which are continuously generating expectations of the future behavior of sound sources (Winkler, 2007, Winkler et al., 2009). In fact, ERP correlates underlying the expectancy violation in sequential auditory processing have been widely investigated. It was found that pitch deviation in melody and spoken language elicited an early negativity followed by a prominent P300 (Brattico et al., 2006, Magne et al., 2006, Schön et al., 2004). Moreover, syntactic deviants in auditory language elicited an Early Left-lateralized Anterior Negativity (ELAN) (Hahne and Friederici, 1999), while unexpected music chords elicited an Early Right-lateralized Anterior Negativity (ERAN) (Koelsch et al., 2000, Koelsch et al., 2007). In short, all these studies have suggested that the expectancy violations in sequential auditory signals can be detected rapidly, and manifested by an early negativity in brain potentials.

The decoding of EP, similar to that of other sequential auditory material like language and music pieces, is based on predictive representations of temporal regularities. Because of the pitch contour and temporal structure inherent for specific EP, specific events are expected at given time points as acoustic events unfolded. Therefore, deviation in EP is bound to bring about expectancy violation, that is, the expectation for the EP development was violated by deviant sounds (Kotz and Paulmann, 2007). In fact, several studies investigated neural correlates of the expectancy violations in EP. Prosodic expectancy violations were reported to elicit a positive deflection 350 ms post violation (Prosodic Expectancy Positivity, PEP) while the combined prosodic-semantic expectancy violations elicited a negative deflection 100 ms after the violation onset, regardless of task relevance, emotional category and speaker identity (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b).

It was reported by a handful of studies that the human brain can differentiate emotional from neutral prosody rapidly. For instance, it was found that ERPs induced by emotional prosodic materials differed from those by neutral materials in P200 component (Paulmann and Kotz, 2008a). Furthermore, studies using oddball paradigm found that emotional category changes demonstrated early negative responses around 200 ms (Goydke et al., 2004, Schirmer et al., 2005, Thönnessen et al., 2010), indicating that the brain is able to use complex acoustic differences of different emotional expression for rapid categorization. In addition, studies by Brosch et al., 2008, Brosch et al., 2009 indicated that response times for targets were faster when they appeared at location of emotional compared to neutral prosody. The explanation for this processing difference is that the evolutionary significance of emotion can lead to prioritized processing strategies which entail fast attentional orienting to emotional stimuli.

A comprehensive working model of EP perception assumed that vocal emotional comprehension is a multi-stage process with individual sub-processes: analyzing acoustic parameters in the time frame of approximately 100 ms, deriving and integrating emotional significance from acoustic cues at about 200 ms, and applying the emotional significance to higher cognitive processing in later time point (Schirmer and Kotz, 2006). According to this Multi-stage model, to comprehend deviation in EP, one has to analyze acoustic features of the deviant sound to extract emotional significance, subsequently, to integrate them with the prosodic context preceding the deviation. Moreover, given the evidence of rapid differentiation of vocal emotion (Goydke et al., 2004, Thönnessen et al., 2010), it is conceivable that the brain detects the deviation in EP during the first two stages and completes all these processes in a short time.

Despite extensive studies of expectancy violation in language and music, the expectancy deviation in EP remains undetermined. The PEP starting 350 ms after violation onset (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b) seems unlikely to be the earliest marker of the brain's detection of vocal emotion deviation, as it is incompatible with the long observed fact that the brain detects auditory deviance early before 200 ms and distinguishes emotional from neutral prosodies within 200 ms. Moreover, only the deviation with emotion transition “neutral-to-emotional” in these studies leaves the temporal features of the opposite transition direction “emotional-to-neutral”, which is lower in magnitude of deviance and emotional significance, unknown as yet.

Therefore, the current study addressed when the human brain detects EP deviation in sentential prosody in double directions and whether the brain's detection of EP deviation is independent of attention allocation. Specifically, we measured participants’ brain responses to prosodic changes in two experiments, one of which required subjects to judge the congruence of the prosodies via the emotion conveyed by the prosodies, while the other required them to detect visual probes while ignoring prosodies. Experimental materials comprised two types of mismatching EPs with bidirectional deviations: “neutral-to-angry” which shifts from a state of calmness to an intense state of anger while “angry-to-neutral” has the opposite shifting. Matching angry and neutral prosodies served as the control comparison respectively. The mismatching EPs were created through the method of cross-splicing auditory signals, which has proven an effective way to investigate the nature of prosody processing (Astesano et al., 2004, Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b, Steinhauer et al., 1999). Based on the oddball research on EP (Goydke et al., 2004, Thönnessen et al., 2010) and the research on auditory expectancy violation (Brattico et al., 2006, Schön et al., 2004) elaborated above, we hypothesized that the brain might detect the deviation in sentential EP rapidly, probably indexed by enhanced early negativities. Moreover, late positivities comparable to PEP (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b) elicited by mismatching EPs were also expected. Additionally, to exclude the possibility of brain detecting the deviation simply via low level acoustic features and to specify the role of emotion in EP deviation detection, a control experiment was carried out, in which participants were asked to detect the sound intensity change during listening to EPs and their non-emotional spectrally rotated counterparts (Blesser, 1972, Sauter and Eimer, 2010, Warren et al., 2006).

Section snippets

Experiment 1

Experiment 1 was conducted in order to investigate when the brain detects the deviation in EPs and how the deviation pattern influences deviation processing. For this purpose, two types of mismatching EPs with different patterns of deviation and their corresponding control EPs were presented while subjects were instructed to decide the congruence of the emotion conveyed by the prosodies.

Participants

Fifteen right-handed university students (eight women, aged 20–26, and mean 22.14) participated in the experiment for payment. None of them had participated in Experiment 1. All participants were right-handed native speakers of Mandarin Chinese with no history of affective or hearing disorder. One participant was excluded from the analysis because of too many artifacts during the EEG recording session.

Stimuli and procedure

Stimuli and procedure were identical to Experiment 1 except that the task for participants was

Participants

Sixteen right-handed native speakers of Mandarin Chinese (nine women, aged 22–25, mean 23.44), who did not participate in the former two experiments, were recruited to participate in the experiment. All participants reported normal auditory and normal or corrected-to-normal visual acuity and no neurological, psychiatric, or other medical problems. Participants gave informed consent and received monetary compensation.

Stimuli and procedure

Half of the stimuli were EPs (only “neutral-to-angry” prosodies and their

General discussion

Because of the predictive encoding of sequential auditory information and the fast differentiation of emotional and neutral prosodies, we hypothesized that the brain could detect the EP deviation quickly and then integrate it with the preceding context during the processing of sentential EP. Consistent with our prediction, the mismatching EPs elicited early negativities relative to the matching ones irrespective of attention access, and the peak latency was shorter when the deviation brought in

Conclusion

The present study demonstrated that the brain is able to rapidly detect the deviation in sentential EP irrespective of attention allocation. Moreover, the EP deviation with large emotional significance like “neutral-to-angry” would increase the vigilance and assign significance during speech comprehension. If the deviation is task relevant, the brain would re-analyze and integrate the deviation with context. During these processes, the emotionality of EP seems to speed up the perception and

Acknowledgements

This research was supported by the National Natural Science Foundation of China (31070989). We would like to thank Jiajin Yuan and Weijun Li for very helpful comments on an earlier version of the manuscript.

References (44)

  • E.D. Ross et al.

    The effect of affect on various acoustic measures of prosody in tone and non-tone languages: a comparison based on computer analysis of voice

    Journal of Phonetics

    (1986)
  • K.R. Scherer

    Vocal communication of emotion: a review of research paradigms

    Speech Communication

    (2003)
  • A. Schirmer et al.

    Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing

    Trends in Cognitive Sciences

    (2006)
  • H. Thönnessen et al.

    Early sensory encoding of affective prosody: neuromagnetic tomography of emotional category changes

    NeuroImage

    (2010)
  • P. Vuust et al.

    Predictive coding of music – brain responses to rhythmic incongruity

    Cortex

    (2009)
  • I. Winkler et al.

    Modeling the auditory scene: predictive regularity representations and perceptual objects

    Trends in Cognitive Sciences

    (2009)
  • J. Yuan et al.

    Are we sensitive to valence differences in emotionally negative stimuli? Electrophysiological evidence from an ERP study

    Neuropsychologia

    (2007)
  • R. Banse et al.

    Acoustic profiles in vocal emotion expression

    Journal of Personality and Social Psychology

    (1996)
  • M. Besson et al.

    Comparison between language and music

    Annals of the New York Academy of Sciences

    (2001)
  • B. Blesser

    Speech perception under conditions of spectral transformation. I. Phonetic characteristics

    Journal of Speech and Hearing Research

    (1972)
  • Boersma, P., Weenink, D., 2006. Praat: doing phonetics by computer [computer program] (Version 4.2). Retrieved July...
  • V. Bostanov et al.

    Recognition of affective prosody: continuous wavelet measures of event-related brain potentials to emotional exclamations

    Psychophysiology

    (2004)
  • Cited by (25)

    • Updating emotional information in daily language comprehension: The influence of topic shifts

      2019, Journal of Neurolinguistics
      Citation Excerpt :

      Overall ANOVAs were followed up by simple effect tests where there were interactions with the topic structure. For the critical words position, based on visual inspection and previous studies on emotional processing (Baetens et al., 2011; Chen, Zhao, Jiang, & Yang, 2011; Ding et al., 2015; Ding, Wang, & Yang, 2016; León et al., 2010; Leuthold et al., 2015), we selected the time windows of P200 (180–300 ms), N400 (300–500 ms), and LPC (500–1000 ms) for statistical analysis. We performed repeated measures ANOVAs with topic structure (topic-maintained/topic-shifted), emotional state (emotion-maintained/emotion-shifted), Anteriority (anterior, central, posterior), and Laterality (left, middle, right) as independent variables.

    • Now listen to this! Evidence from a cross-spliced experimental design contrasting pressuring and supportive communications

      2019, Neuropsychologia
      Citation Excerpt :

      That is, if the change required listeners’ immediate and strong reaction, as controlling motivational communications tend to do, resources were quickly allocated to attend to the situation. This speculation receives support from previous reports that fail to identify effects for transitions from angry to neutral prosody (Chen et al., 2011), possibly because a change to a neutral sounding voice does not require an immediate response from the listener as neutral is considered to be a state of relaxation when compared to anger. The findings then also substantiate our previous reports of the relevance of controlling prosody as behavior of immediate action (Zougkou et al., 2017).

    • Emotional prosody Stroop effect in Hindi: An event related potential study

      2019, Progress in Brain Research
      Citation Excerpt :

      Detailed integration of contextual meaning of word with relevant or irrelevant emotional prosody is thought to occur late, 300–400 ms after stimulus onset. A detailed processing of prosodic meaning takes place in the late components of P300, N300 and N400, when participants perceived the sentential prosody match-mismatch through cross-splicing of sentences from angry to neutral and neutral to angry (Chen et al., 2011). In addition to ERP studies, a few neuroimaging studies on emotional prosody at the level of words and sentences have found a greater activation at the right than left-hemispheric side of anterior-superior temporal gyrus (STG) and superior temporal sulcus (STS) for affective compared to neutral sentences (Beaucousin et al., 2007; Ethofer et al., 2009a,b).

    • Influence of attention on bimodal integration during emotional change decoding: ERP evidence

      2016, International Journal of Psychophysiology
      Citation Excerpt :

      Previous studies showed that inattentive detection of emotion change in the visual and auditory oddball paradigm was associated with mismatch negativity (Kimura et al., 2012; Stefanics et al., 2012; Thonnessen et al., 2010), while attentive processing of emotional changes was associated with a N2/P3 complex (Campanella et al., 2013; Campanella et al., 2002; Wambacq and Jerger, 2004). Furthermore, vocal emotional change in auditory sentence context was associated with a positive deflection 350 ms post change point (Chen et al., 2012; Chen et al., 2011; Kotz and Paulmann, 2007; Paulmann et al., 2012) as well as theta synchronization (Chen et al., 2015b; Chen et al., 2012) independent of attention direction, but beta desynchronization was observed just under explicit instead of implicit task condition (Chen et al., 2015a, 2015b). These findings suggest that emotional changes in single modality can be detected independent of attention assignment.

    View all citing articles on Scopus
    View full text