Event-related potential correlates of the expectancy violation effect during emotional prosody processing
Research highlights
▶ We investigated the ERP correlates of expectancy violation effect during emotional prosody processing with three experiments. ▶ The brain is able to detect the deviation in sentential emotional prosody rapidly irrespective of attention allocation. ▶ Deviation with large emotional significance would increase the vigilance and assign significance during speech comprehension. ▶ A process of re-analyzing and integrating would take place if the deviation is task relevant. ▶ The emotionality seems to speed up the perception and step up more vigilance during this process.
Introduction
In addition to the verbal emotion conveyed by semantic content, a speaker expresses his or her emotion in prosody by manipulating acoustic parameters such as pitch, intensity, and speech rate, termed as emotional prosody (EP) or vocal emotion. Banse and Scherer (1996) found that each emotion has its own acoustic profile. For example, the vocalization of anger reveals a higher fundamental frequency (F0) and intensity than that of neutral utterance. Successful emotional communication is critical for social interaction. Particularly, it is adaptively important to detect EP deviation in time, since changes in EP are common in spoken interactions, and a sudden variation in prosody is a direct signal for the speakers’ emotion changes. Despite the fact that human beings are born with competence to process deviation in EP efficiently, the underlying neural mechanism remains unclear. Therefore, it is highly important to investigate the neural correlates of the perception of EP deviation and to further clarify the cognitive mechanism of vocal emotion perception.
The auditory perception is based on predictive representation of temporal regularities, which are continuously generating expectations of the future behavior of sound sources (Winkler, 2007, Winkler et al., 2009). In fact, ERP correlates underlying the expectancy violation in sequential auditory processing have been widely investigated. It was found that pitch deviation in melody and spoken language elicited an early negativity followed by a prominent P300 (Brattico et al., 2006, Magne et al., 2006, Schön et al., 2004). Moreover, syntactic deviants in auditory language elicited an Early Left-lateralized Anterior Negativity (ELAN) (Hahne and Friederici, 1999), while unexpected music chords elicited an Early Right-lateralized Anterior Negativity (ERAN) (Koelsch et al., 2000, Koelsch et al., 2007). In short, all these studies have suggested that the expectancy violations in sequential auditory signals can be detected rapidly, and manifested by an early negativity in brain potentials.
The decoding of EP, similar to that of other sequential auditory material like language and music pieces, is based on predictive representations of temporal regularities. Because of the pitch contour and temporal structure inherent for specific EP, specific events are expected at given time points as acoustic events unfolded. Therefore, deviation in EP is bound to bring about expectancy violation, that is, the expectation for the EP development was violated by deviant sounds (Kotz and Paulmann, 2007). In fact, several studies investigated neural correlates of the expectancy violations in EP. Prosodic expectancy violations were reported to elicit a positive deflection 350 ms post violation (Prosodic Expectancy Positivity, PEP) while the combined prosodic-semantic expectancy violations elicited a negative deflection 100 ms after the violation onset, regardless of task relevance, emotional category and speaker identity (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b).
It was reported by a handful of studies that the human brain can differentiate emotional from neutral prosody rapidly. For instance, it was found that ERPs induced by emotional prosodic materials differed from those by neutral materials in P200 component (Paulmann and Kotz, 2008a). Furthermore, studies using oddball paradigm found that emotional category changes demonstrated early negative responses around 200 ms (Goydke et al., 2004, Schirmer et al., 2005, Thönnessen et al., 2010), indicating that the brain is able to use complex acoustic differences of different emotional expression for rapid categorization. In addition, studies by Brosch et al., 2008, Brosch et al., 2009 indicated that response times for targets were faster when they appeared at location of emotional compared to neutral prosody. The explanation for this processing difference is that the evolutionary significance of emotion can lead to prioritized processing strategies which entail fast attentional orienting to emotional stimuli.
A comprehensive working model of EP perception assumed that vocal emotional comprehension is a multi-stage process with individual sub-processes: analyzing acoustic parameters in the time frame of approximately 100 ms, deriving and integrating emotional significance from acoustic cues at about 200 ms, and applying the emotional significance to higher cognitive processing in later time point (Schirmer and Kotz, 2006). According to this Multi-stage model, to comprehend deviation in EP, one has to analyze acoustic features of the deviant sound to extract emotional significance, subsequently, to integrate them with the prosodic context preceding the deviation. Moreover, given the evidence of rapid differentiation of vocal emotion (Goydke et al., 2004, Thönnessen et al., 2010), it is conceivable that the brain detects the deviation in EP during the first two stages and completes all these processes in a short time.
Despite extensive studies of expectancy violation in language and music, the expectancy deviation in EP remains undetermined. The PEP starting 350 ms after violation onset (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b) seems unlikely to be the earliest marker of the brain's detection of vocal emotion deviation, as it is incompatible with the long observed fact that the brain detects auditory deviance early before 200 ms and distinguishes emotional from neutral prosodies within 200 ms. Moreover, only the deviation with emotion transition “neutral-to-emotional” in these studies leaves the temporal features of the opposite transition direction “emotional-to-neutral”, which is lower in magnitude of deviance and emotional significance, unknown as yet.
Therefore, the current study addressed when the human brain detects EP deviation in sentential prosody in double directions and whether the brain's detection of EP deviation is independent of attention allocation. Specifically, we measured participants’ brain responses to prosodic changes in two experiments, one of which required subjects to judge the congruence of the prosodies via the emotion conveyed by the prosodies, while the other required them to detect visual probes while ignoring prosodies. Experimental materials comprised two types of mismatching EPs with bidirectional deviations: “neutral-to-angry” which shifts from a state of calmness to an intense state of anger while “angry-to-neutral” has the opposite shifting. Matching angry and neutral prosodies served as the control comparison respectively. The mismatching EPs were created through the method of cross-splicing auditory signals, which has proven an effective way to investigate the nature of prosody processing (Astesano et al., 2004, Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b, Steinhauer et al., 1999). Based on the oddball research on EP (Goydke et al., 2004, Thönnessen et al., 2010) and the research on auditory expectancy violation (Brattico et al., 2006, Schön et al., 2004) elaborated above, we hypothesized that the brain might detect the deviation in sentential EP rapidly, probably indexed by enhanced early negativities. Moreover, late positivities comparable to PEP (Kotz and Paulmann, 2007, Paulmann and Kotz, 2008b) elicited by mismatching EPs were also expected. Additionally, to exclude the possibility of brain detecting the deviation simply via low level acoustic features and to specify the role of emotion in EP deviation detection, a control experiment was carried out, in which participants were asked to detect the sound intensity change during listening to EPs and their non-emotional spectrally rotated counterparts (Blesser, 1972, Sauter and Eimer, 2010, Warren et al., 2006).
Section snippets
Experiment 1
Experiment 1 was conducted in order to investigate when the brain detects the deviation in EPs and how the deviation pattern influences deviation processing. For this purpose, two types of mismatching EPs with different patterns of deviation and their corresponding control EPs were presented while subjects were instructed to decide the congruence of the emotion conveyed by the prosodies.
Participants
Fifteen right-handed university students (eight women, aged 20–26, and mean 22.14) participated in the experiment for payment. None of them had participated in Experiment 1. All participants were right-handed native speakers of Mandarin Chinese with no history of affective or hearing disorder. One participant was excluded from the analysis because of too many artifacts during the EEG recording session.
Stimuli and procedure
Stimuli and procedure were identical to Experiment 1 except that the task for participants was
Participants
Sixteen right-handed native speakers of Mandarin Chinese (nine women, aged 22–25, mean 23.44), who did not participate in the former two experiments, were recruited to participate in the experiment. All participants reported normal auditory and normal or corrected-to-normal visual acuity and no neurological, psychiatric, or other medical problems. Participants gave informed consent and received monetary compensation.
Stimuli and procedure
Half of the stimuli were EPs (only “neutral-to-angry” prosodies and their
General discussion
Because of the predictive encoding of sequential auditory information and the fast differentiation of emotional and neutral prosodies, we hypothesized that the brain could detect the EP deviation quickly and then integrate it with the preceding context during the processing of sentential EP. Consistent with our prediction, the mismatching EPs elicited early negativities relative to the matching ones irrespective of attention access, and the peak latency was shorter when the deviation brought in
Conclusion
The present study demonstrated that the brain is able to rapidly detect the deviation in sentential EP irrespective of attention allocation. Moreover, the EP deviation with large emotional significance like “neutral-to-angry” would increase the vigilance and assign significance during speech comprehension. If the deviation is task relevant, the brain would re-analyze and integrate the deviation with context. During these processes, the emotionality of EP seems to speed up the perception and
Acknowledgements
This research was supported by the National Natural Science Foundation of China (31070989). We would like to thank Jiajin Yuan and Weijun Li for very helpful comments on an earlier version of the manuscript.
References (44)
- et al.
Brain potentials during semantic and prosodic processing in French
Cognitive Brain Research
(2004) - et al.
The role of intonation in emotional expressions
Speech Communication
(2005) - et al.
Musical scale properties are automatically processed in the human auditory cortex
Brain Research
(2006) - et al.
Behold the voice of wrath: cross-modal modulation of visual attention by anger prosody
Cognition
(2008) - et al.
Event-related potentials recorded during the discrimination of improbable stimuli
Biological Psychology
(1983) - et al.
Changes in emotional tone and instrumental timbre are reflected by the mismatch negativity
Cognitive Brain Research
(2004) - et al.
When emotional prosody and semantics dance cheek to cheek: ERP evidence
Brain Research
(2007) - et al.
Evoked potentials to consonant-vowel syllables
Acta Psychologica
(1981) - et al.
The mismatch negativity (MMN) in basic research of central auditory processing: a review
Clinical Neurophysiology
(2007) - et al.
An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context
Brain and Language
(2008)
The effect of affect on various acoustic measures of prosody in tone and non-tone languages: a comparison based on computer analysis of voice
Journal of Phonetics
Vocal communication of emotion: a review of research paradigms
Speech Communication
Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing
Trends in Cognitive Sciences
Early sensory encoding of affective prosody: neuromagnetic tomography of emotional category changes
NeuroImage
Predictive coding of music – brain responses to rhythmic incongruity
Cortex
Modeling the auditory scene: predictive regularity representations and perceptual objects
Trends in Cognitive Sciences
Are we sensitive to valence differences in emotionally negative stimuli? Electrophysiological evidence from an ERP study
Neuropsychologia
Acoustic profiles in vocal emotion expression
Journal of Personality and Social Psychology
Comparison between language and music
Annals of the New York Academy of Sciences
Speech perception under conditions of spectral transformation. I. Phonetic characteristics
Journal of Speech and Hearing Research
Recognition of affective prosody: continuous wavelet measures of event-related brain potentials to emotional exclamations
Psychophysiology
Cited by (25)
Updating emotional information in daily language comprehension: The influence of topic shifts
2019, Journal of NeurolinguisticsCitation Excerpt :Overall ANOVAs were followed up by simple effect tests where there were interactions with the topic structure. For the critical words position, based on visual inspection and previous studies on emotional processing (Baetens et al., 2011; Chen, Zhao, Jiang, & Yang, 2011; Ding et al., 2015; Ding, Wang, & Yang, 2016; León et al., 2010; Leuthold et al., 2015), we selected the time windows of P200 (180–300 ms), N400 (300–500 ms), and LPC (500–1000 ms) for statistical analysis. We performed repeated measures ANOVAs with topic structure (topic-maintained/topic-shifted), emotional state (emotion-maintained/emotion-shifted), Anteriority (anterior, central, posterior), and Laterality (left, middle, right) as independent variables.
Now listen to this! Evidence from a cross-spliced experimental design contrasting pressuring and supportive communications
2019, NeuropsychologiaCitation Excerpt :That is, if the change required listeners’ immediate and strong reaction, as controlling motivational communications tend to do, resources were quickly allocated to attend to the situation. This speculation receives support from previous reports that fail to identify effects for transitions from angry to neutral prosody (Chen et al., 2011), possibly because a change to a neutral sounding voice does not require an immediate response from the listener as neutral is considered to be a state of relaxation when compared to anger. The findings then also substantiate our previous reports of the relevance of controlling prosody as behavior of immediate action (Zougkou et al., 2017).
Emotional prosody Stroop effect in Hindi: An event related potential study
2019, Progress in Brain ResearchCitation Excerpt :Detailed integration of contextual meaning of word with relevant or irrelevant emotional prosody is thought to occur late, 300–400 ms after stimulus onset. A detailed processing of prosodic meaning takes place in the late components of P300, N300 and N400, when participants perceived the sentential prosody match-mismatch through cross-splicing of sentences from angry to neutral and neutral to angry (Chen et al., 2011). In addition to ERP studies, a few neuroimaging studies on emotional prosody at the level of words and sentences have found a greater activation at the right than left-hemispheric side of anterior-superior temporal gyrus (STG) and superior temporal sulcus (STS) for affective compared to neutral sentences (Beaucousin et al., 2007; Ethofer et al., 2009a,b).
That note sounds wrong! Age-related effects in processing of musical expectation
2017, Brain and CognitionMusical training shapes neural responses to melodic and prosodic expectation
2016, Brain ResearchInfluence of attention on bimodal integration during emotional change decoding: ERP evidence
2016, International Journal of PsychophysiologyCitation Excerpt :Previous studies showed that inattentive detection of emotion change in the visual and auditory oddball paradigm was associated with mismatch negativity (Kimura et al., 2012; Stefanics et al., 2012; Thonnessen et al., 2010), while attentive processing of emotional changes was associated with a N2/P3 complex (Campanella et al., 2013; Campanella et al., 2002; Wambacq and Jerger, 2004). Furthermore, vocal emotional change in auditory sentence context was associated with a positive deflection 350 ms post change point (Chen et al., 2012; Chen et al., 2011; Kotz and Paulmann, 2007; Paulmann et al., 2012) as well as theta synchronization (Chen et al., 2015b; Chen et al., 2012) independent of attention direction, but beta desynchronization was observed just under explicit instead of implicit task condition (Chen et al., 2015a, 2015b). These findings suggest that emotional changes in single modality can be detected independent of attention assignment.