Principles of extinction learning of nonaversive experience

Extinction and the role of context

Our environment is usually quite predictable: it does not rain when there is a cloudless sky; tasting your morning coffee is preceded by visual and olfactory perceptions of the beverage. Thus, certain events are related and often occur in a particular order. Humans and other animals are able to learn about event relationships, which allows us to predict future events based on the presence of preceding stimuli or actions (Lachnit et al., 2004; Melchers et al., 2005). This ability for associative learning is a considerable advantage for adaption and survival.

Classical conditioning and instrumental conditioning are two basic forms of associative learning. In classical conditioning (Pavlov, 1927), a neutral stimulus is repeatedly presented before a motivationally relevant outcome. As a result of these pairings, the neutral stimulus comes to elicit a response that indicates anticipation of the outcome. Consider Pavlov’s dog who salivated when hearing a bell that had been repeatedly presented before feeding. Instrumental conditioning (Skinner, 1938) reflects our ability to learn about the consequences of our actions. Reward or punishment that follows a behavior increases or decreases the probability with which that behavior will occur in the future.

Classical conditioning and instrumental conditioning are crucial for successful interactions with our environment. However, they are also involved in the development of maladaptive behavior. Both forms of associative learning play key roles for a variety of psychopathological disorders, including phobias, eating disorders, and drug abuse. Many therapeutic treatments aimed at overcoming maladaptive behavior are based on the principle of extinction learning (Craske et al., 2014; Podlesnik et al., 2017). When a stimulus or an action is no longer followed by the expected outcome, we will cease the acquired behavior: Pavlov’s dog will eventually stop salivating in response to the bell when subsequent feeding is repeatedly omitted; a patient’s fear of spiders will decrease significantly when repeatedly exposed to spiders in the absence of actual danger.

However, extinction of acquired behavior does not always endure. Rather, acquired responses have been observed to reappear after extinction under various conditions (Bouton, 1993; Bouton et al., 2012). An intriguing example is the renewal effect, which refers to the finding that changing the context in which a behavior was extinguished can restore (renew) the original response. In a typical renewal experiment, the conditioned response is first established in a particular context. Then, the acquired behavior is extinguished in a different context. During a final test, it has been observed that the original response reoccurs either when the individual is shifted to the context of initial conditioning or when the individual is exposed to a third, novel context (Bouton and Bolles, 1979). Renewal has also been observed when behavioral acquisition and extinction take place in the same context, but testing occurs in a different context (Bouton and Ricker, 1994). Analogous results have been reported for human associative learning with motivationally insignificant stimuli (Rosas and Callejas-Aguilera, 2006; Üngör and Lachnit, 2006, 2008). Thus, the absence of the context of extinction learning appears to be sufficient to induce a recovery of acquired behavior.

The renewal effect has rather challenging implications for therapeutic treatments involving extinction learning. It suggests that full expression of therapeutic success may be limited to the therapeutic environment: the likelihood of relapse increases outside the therapeutic setting.

Basic research has revealed several experimental strategies that reduce or even prevent the renewal effect. These findings may provide important insights for improving the long-term success of therapeutic interventions. One experimental strategy that has received considerable attention comprises extinction learning in multiple contexts (Craske et al., 2014; Laborda et al., 2011). However, experiments involving human associative learning (Bustamante et al., 2016b) and instrumental conditioning in rats (Bernal-Gamboa et al., 2017) have indicated that the impact of this strategy may depend on the type of renewal procedure: extinction in multiple contexts resulted in weaker response recovery than extinction in a single context, when testing for renewal occurred in a novel context. However, when the test took place in the context in which the response had been originally acquired, extinction in multiple contexts exerted no attenuating effect on renewal (Bernal-Gamboa et al., 2017; Bustamante et al., 2016b).

Another experimental strategy aimed to counter the renewal effect is the application of so-called reminder cues, which refer to discrete stimuli that are repeatedly presented during the extinction of a response. Using visual reminder cues in human associative learning (Bustamante et al., 2016a) and auditory reminder cues in instrumental conditioning with rats (Nieto et al., 2020), experiments have shown that the application of reminder cues during renewal testing in a novel context completely prevented the recovery of acquired responding. Although this level of effectiveness is not reached when testing occurs in the context of initial acquisition, reminder cues weaken the degree of response recovery in this test situation (Nieto et al., 2017).

The renewal effect is also influenced by experimental manipulations that target the informational value of contexts. For many cases, contexts have low informational value, in the sense that they are irrelevant for the relationship between events – the delicious taste after biting into an apple occurs regardless of whether you are at home or in your workplace. However, in other cases, the relationship between events varies across contexts – having a lively conversation is welcomed at a party, but the same behavior is considered inappropriate in a library. Thus, contexts can carry relevant information about the current relationship between events. Studies of human associative learning have revealed that response recovery after extinction is weaker when initial acquisition (Lucke et al., 2013) or extinction (Lucke et al., 2014) was conducted in a context that had been trained as being irrelevant for other stimulus-outcome relationships, compared with a context trained as being relevant. Measures of eye-gaze behavior (Lucke et al., 2013) and other experimental approaches (Uengoer et al., 2018) suggest that the impact of context information on context-dependent learning is based on processes of selective attention.

Brain regions involved in extinction and renewal of nonaversive experience

Extinction learning can comprise aversive/maladaptive (fear, phobias, addiction) or benign/appetitive elements. Extinction of aversive and maladaptive behavior has received the greatest degree of scrutiny to date, and it has become apparent that structures such as the amygdala, prefrontal cortex, and hippocampus play important roles in the processing of context in human subjects and in rodents during extinction of fear responses (Kalisch, 2006; Lang et al., 2009; Lingawi et al., 2019; Marek et al., 2019; Milad et al., 2007) and in fear renewal (Hermann et al., 2016). Extinction of appetitive, or nonaversive, learning in humans (Lissek et al., 2013) and rodents (Mendez-Couz et al., 2019) also involves the hippocampus.

Imaging studies investigating extinction related to nonaversive learning in humans (Figure 1) have demonstrated that the hippocampus and ventromedial prefrontal cortex (vmPFC) mediate renewal of acquired behavior (Lissek et al., 2013). Both regions showed higher activation in participants who exhibited renewal than in those who did not: the hippocampus encoded context information during extinction, displaying even higher activation in response to a stimulus presented in a novel context, while the vmPFC retrieved this information during renewal testing to decide upon response recovery. Recent studies on rats have demonstrated that information processing in discrete hippocampal subfields contribute to specific elements of context-dependent acquisition, extinction, and renewal in an appetitive spatial learning task (Mendez-Couz et al., 2019; see Figure 1), indicating that the hippocampus may be intrinsically involved in determining the specificity of the learned response.

Figure 1:

Paradigms for the study of extinction learning in humans or in rodents. A. In this paradigm, human subjects are presented with a sequence of trials each showing a compound of a food item (cue) and the name of a restaurant (context; e.g. “Zum Krug”). Each compound is associated with a specific outcome. Following an intertrial interval of 5–9 s, one cue/context compound is presented for 3 s. Then, a question appears asking the participant to predict whether consumption of the food in the restaurant will cause stomachache in a hypothetical patient, followed by a response period of maximally 4 s. Feedback, providing the correct answer, is then shown for 2 s (Golisch et al., 2017). B. The task comprises three phases: acquisition, extinction, and test. In the AAA condition, all phases occur in the same context, while in the ABA condition, the extinction context differs. During the test in both conditions, cues are presented in the same context as during acquisition (Golisch et al., 2017). C. Examples of food images used in the task (Golisch et al., 2017). D/E. In rodents, nonaversive extinction learning can be studied by examining associative spatial learning and memory. Over a period of days, rodents learn that a food reward can be found (with low probability) at a specific end of a T-Maze arm. The T-Maze has a specific floor pattern, and a mild odor is present at the end of both T-Maze arms and visuospatial cues are placed outside of the T-maze, in visible range. The food reward is hidden in an indentation in the floor near the end of the target arm. One day after the animals have reached at least 80% arm-choice accuracy, extinction learning is examined either in the presence (D) or absence (E) of a context change. Here, the floor pattern, odor cues, and distal visuospatial cues are changed. During extinction learning trials, no food reward is present. Renewal is assessed in the ABA paradigm (D) by returning the animals to the original context. In the AAA paradigm (E), animals are simply reexposed to the same context (André et al., 2015b; Mendez-Couz et al., 2019; Wiescholleck et al. 2014).

In line with this, individuals with, and without, a propensity for renewal differ in context-related hippocampal activation not only during extinction but also during initial acquisition, where context is irrelevant (Lissek et al., 2016). All individuals – regardless of their propensity for renewal – showed increased activation of the posterior hippocampus in a novelty response to the presentation of only the context. However, only those participants with a propensity for renewal maintained this hippocampal activation when a cue was added to the context, indicating processing of the context/cue compound.

While the amygdala is consistently active during extinction of fear responses (Hermann et al., 2020; Merz et al., 2013), it is also active in extinction related to nonaversive experience (Lissek et al., 2013). The finding supports a proposed broader role of the amygdala in aversive and appetitive learning (Everitt et al., 2003; Knapska et al., 2006). Other regions previously shown to be involved in fear extinction (Sehlmeyer et al., 2009) that are regularly found to be active during nonaversive extinction learning comprise the anterior cingulate cortex (ACC) and insula, which exhibited higher activity in participants with a propensity for renewal (Lissek et al., 2013). This increased activity indicates that attentional processing mediated by the ACC and processing of salient events by the insula (Menon and Uddin, 2010) are more pronounced in these participants.

Neurotransmitter systems involved in extinction and renewal of nonaversive experience

The creation of associative memories depends on cortical and hippocampal plasticity processes that in turn critically depend on the activation and regulation of neurotransmitter receptor systems including glutamatergic N-methyl-D-aspartate (NMDA) receptors (Hansen et al., 2017), gamma aminobutyric acid (GABA) receptors (Swanson and Maffei, 2019), and catecholaminergic receptors (Hagena et al., 2016; Hansen and Manahan-Vaughan, 2014). Although studies of nonaversive extinction learning are less numerous than the wealth of data available with regard to extinction of aversive learning, it is apparent that neurotransmitter receptors that are essential for cortical and synaptic plasticity serve to modulate the efficacy of extinction of nonaversive learning (Table 1).

Pharmacological manipulation of NMDA receptors modulated extinction related to nonaversive learning in human subjects when conducted within the same context as for initial acquisition: strikingly both the NMDA receptor agonist, d-cycloserine, DCS, (Klass et al., 2017) and the NMDA receptor antagonist, memantine (Golisch et al., 2017), enhanced extinction learning. This latter finding, which was associated with dose-related effects of memantine modulated by body mass index, suggests that fine-tuning of the degree of activation of NMDA receptors is a key facet of effective extinction learning. This may relate to a possible differential regulation, by the ligands used in these studies, of GluN2A- or GluN2B-containing NMDA receptors, which determine, in turn, the amplitude and persistency of synaptic plasticity (Ballesteros et al., 2016).

Research on extinction and renewal related to nonaversive learning in humans demonstrated a specific role for dopamine (DA) receptors for extinction learning in a novel context, whereas the DA antagonist, tiapride, when administered as a single dose before the extinction phase, impaired performance (Lissek et al., 2015b), and the DA agonist, bromocriptine, enhanced extinction learning, particularly in those individuals with a propensity for renewal (Lissek et al., 2018). The role of specific DA receptors was scrutinized in rodent experiments: Studies of extinction learning using a spatial appetitive task in rats demonstrated that dopamine acting on the D1/D5 receptor modulates both the acquisition and the consolidation of extinction learning. D2 receptors modulated context-independent aspects of extinction learning (André and Manahan-Vaughan, 2016).

The noradrenergic system also contributes to extinction learning. Administration of the noradrenaline reuptake inhibitor, atomoxetine, to human subjects (Lissek et al., 2015a) or to rats (Janak and Corbit, 2011) enhanced extinction in nonaversive or appetitive tasks. In rats, extinction learning within a spatial appetitive task was unaffected by antagonism of beta-adrenergic receptors (André et al., 2015), however, suggesting that either this process is supported by alpha-adrenergic receptors or attentional demand is a determinant of the involvement of the noradrenergic system in extinction learning. Consistent with the latter possibility, activation of beta-adrenergic receptors is required for extinction learning in the absence of a context change (André et al., 2015a). This latter process is also supported by metabotropic glutamate receptors (mGluR; André et al., 2015b).

Extinction related to nonaversive learning in human subjects was impaired by pharmacological activation of GABA receptors with the agonist lorazepam, irrespective of the context in which extinction occurred (Lissek et al., 2015a, 2017). These results correspond to animal studies reporting impairments of extinction learning by local hippocampal GABA receptor agonism (Corcoran, 2005; Corcoran and Maren, 2001).

Consistent with the likelihood that extinction learning involves de novo encoding of associative experience (Mendez-Couz et al., 2019), enhanced hippocampal activation during extinction learning and renewal testing was observed after stimulation of noradrenergic, dopaminergic, or glutamatergic NMDA receptors in human subjects before extinction training. In contrast, hippocampal activity was reduced by dopaminergic antagonism and GABA agonism (Lissek et al., 2015a, 2015b, 2017). Activation of the vmPFC was enhanced by noradrenergic stimulation during extinction learning and by GABA agonism during renewal testing and reduced by DA antagonism during extinction in the acquisition context, but not in a novel one. NMDA or noradrenergic receptor activation increased activation of the dorsolateral prefrontal cortex and inferior frontal gyrus, whereas the DA receptor antagonism, GABA receptor activation, and NMDA receptor antagonism reduced activation. In addition, both noradrenergic and NMDA receptor stimulation increased ACC and insula activation in extinction and renewal testing, while GABA receptor agonism and the DA receptor antagonism reduced activation in these regions (Lissek et al., 2015a, 2015b; Klass et al., 2017).

Taken together, results obtained in pharmacological studies on humans and rodents indicate that during extinction learning, dopamine, acting in the prefrontal cortex and hippocampus, is involved in readjusting the cue-outcome relationship in the presence of a novel context. Hippocampal dopamine is important for the encoding and provision of context information and is, thus, essentially involved in the renewal effect. In contrast, prefrontal and hippocampal NMDA receptors appear to be specifically involved in the modification of established stimulus-outcome associations in the context of initial acquisition. Moreover, the noradrenergic system is involved in the modification of established associations during extinction learning, regardless of context, underlining the supposed importance of attentional processes in extinction learning.

Catecholaminergic, GABAergic, and glutamatergic regulation of extinction learning is not restricted to nonaversive experience. Noradrenaline acting on beta-adrenergic receptors in the amygdala impairs extinction of fear, whereas noradrenaline acting on alpha-adrenergic receptors in the prefrontal cortex enhances it (Likhtik and Johansen, 2019). Furthermore, the robustness of fear memory and consequently the effectiveness of extinction learning is regulated by dopamine release from the central tegmental area acting on key brain circuitry such as the hippocampus, prefrontal cortex, and amygdala (Likhtik and Johansen, 2019). GABAergic transmission and mGluR and NMDA receptor activity in these structures also modulate fear memory and fear extinction (Courtin et al., 2014; Kaplan and Moore, 2011; Myers et al., 2001; Walker and Davis, 2002).

In conclusion, despite their clear differences in terms of behavior and cognition, extinction learning of aversive and nonaversive experience shares many functional similarities in terms of the brain regions that are engaged by these processes and the neurotransmitter receptors that mediate the behavioral outcome. This suggests that knowledge gained through studies of processes that optimize extinction learning in an experimental setting harbors significant potential in translation into therapeutic strategies for maladaptive behavior.