Construction of a Hierarchical Organization in Semantic Memory: A Model Based on Neural Masses and Gamma-Band Synchronization

The capacity to recognize objects and recover their contents is an essential aspect of our cognitive system, involving the so-called “semantic memory.” The common idea is that the representation of objects in semantic memory consists of a list of features that describe an object’s fundamental aspects in a context-independent and impersonal manner [1‐3]. These features are acquired with time, reflecting a long experience in the course of life.

Given the importance of semantics in our cognition and the enormous consequences of its damage in daily life, it becomes crucial to understand the neural mechanisms involved and to formulate mechanistic explanations. Neurocomputational models, inspired by brain functioning, can play a relevant role in this domain, proposing possible solutions for neurological problems, emphasizing putative mechanisms and circuits, and suggesting testable predictions to validate or reject hypotheses. Furthermore, neural networks inspired by biology can represent innovative tools in artificial intelligence and machine learning, devising new technological solutions to old problems.

Indeed, the development of neurocomputational models of semantic memory has a long tradition, dating back to the early nineties.

Of particular relevance, Rumelhart et al. [4] and Rogers and McClelland [5, 6] used a feedforward schema trained with backpropagation, with the primary objective of investigating which representation can develop in the hidden units of a multilayer network. This approach was further extended by Rogers et al., as described below.

Conversely, other models, starting from the early nineties, were based on attractor dynamics, in which the object representation is stored as an equilibrium point of a recurrent network. These provided essential indications of semantic aphasia and associative priming [7‐10]. Specifically, Cree et al. [9] found that distinctive features play a more significant role in activating a concept than shared features. O’Connor, Cree, and McRae [11] demonstrated that a single layer of feature nodes can represent both subordinate and superordinate concepts without needing any a priori hierarchical organization. However, these models trained synapses using the recurrent backpropagation-through-time algorithm.

Other authors, conversely, used auto-associative networks trained with the Hebb rule, as in the classic well-known Hopfield model [12]. McRae et al. [13] used an attractor network trained with the Hebb rule to investigate the role of feature correlation in the organization of semantic memory and explained several semantic priming effects. Kawamoto [14] demonstrated that attractor basins are crucial to understanding priming, deep dyslexia, and ambiguity resolution. Miikkulainen and Silberman et al. [15, 16] developed a model consisting of two self-organizing maps. Siekmeier and Hoffman [17] used the Hebb rule in a Hopfield network to compare semantic priming in normal subjects and schizophrenic patients. However, classic Hebb rules produce symmetrical synapses and have limitations in discovering a hierarchical organization of concepts.

More recent advanced models of semantic memory introduced a multilayer topology inspired by cortical structure and function. One of the purposes of these models is to reconcile divergent hypotheses in the literature, such as the presence of category-specific vs. category-general semantic areas (i.e., a sensory-functional vs. a distributed representation). In particular, Rogers et al. developed a series of multilayer models, implementing a distributed-plus-hub theory of semantic memory, assuming that concepts reflect both hub and spoke representations and their interactions. This neurocomputational model explains several neuroimaging and patient pieces of evidence [18] and indicates how the semantic representation changes dynamically with stimulus processing [19]. It is worth noting that these networks too are trained with a variant of backpropagation adapted to recurrent networks to minimize squared errors.

However, a second multilayer semantic model, which exploits the Hebb rule, was developed by Garagnani et al. in a series of papers during the last decade [20‐22]. Based on neuroanatomical studies, the network includes twelve cortical areas and their within-area and between-area connectivity. In particular, the model mimics the function of primary and secondary sensorimotor regions of the frontal, temporal, and occipital cortex, along with a connector hub. Hebbian mechanisms for synaptic modification are used to study the emergence of associations among word forms (phonological/lexical) and the object’s semantic content. The results show that cell assemblies are formed after learning, reflecting the semantic content, and explain the presence of both category-specific and more general semantic processes. A version of the model with spiking neurons also analyzes high-frequency synchronous oscillations during word processing [23, 24]. Interestingly, a recent version [25] also analyzes differences between concrete and abstract concepts, ascribing the formation of concrete concepts to the presence of shared features. Conversely, abstract concepts are explained by family resemblance among instances.

Despite the significant value of the last models, a few relevant aspects of semantic memory still need to be dealt with. First, concepts in nature exhibit a strong correlation among their features, which can lead to a hierarchical organization among concepts. Nevertheless, Hebbian mechanisms have difficulty finding these organizations (see Mac Rae et al. [13] for an excellent critical analysis). Furthermore, several authors have argued that not all features are equally important in representing the concepts [9]. In particular, feature listing tasks [10, 26, 27] show that not all features within an object play the same role. While some are salient and quickly come to mind when thinking of an object, others are marginal and rarely evoked. The Hebbian learning procedures used in most previous models neglect these two essential aspects of the semantics, using orthogonal object representations and assuming that the total object content (i.e., all features) is experienced at any step, neglecting the probability aspects of our experience and differences in feature saliency.

A few more recent studies investigated how the Hebb rule could be modified in attractor networks to deal with correlated patterns and, more generally, to improve storage capacity. To overcome the limitations of attractor networks in the presence of correlated patterns, Blumenfeld et al. [28] introduced a learning rule in which synaptic weight changes are facilitated by the novelty (i.e., the difference between the present input and stored memories) and demonstrated that this rule allows memory representations of correlated stimuli depending on the learning order. Tang et al. [29] extended the previous work using a more plausible network and examining the role of saliency weights (i.e., the patterns are stored with a variable saliency factor). Results show that saliency weights can vividly affect the memory representations, allowing flexibility of the resulting attractors. Kropff and Treves [30] introduced a Hebbian rule in which the presynaptic threshold reflects the neuron popularity and showed that this rule can store and retrieve sets of correlated patterns. It is worth noting that this rule requires the extraction of statistical properties; hence, it is inappropriate for one-shot learning. Boboeva et al. [31] studied the capacity of a Potts network (i.e., a variant of the Hopfield network in which neurons have multiple possible discrete states) in the presence of correlated data. They showed that correlation reduces the storage capacity. However, when the storage capacity is overcome, the network can maintain the gross core semantic components, and only fine details are compromised. Finally, Pereira and Brunel [32] used a variant of the Hebb rule in which the presynaptic and postsynaptic terms are described through nonlinear functions of neuron activity, and these functions are empirically derived from data. Although not directly mentioning correlated patterns, the authors showed that this rule, with sigmoidal dependence on pre- and postsynaptic firing rates, almost maximizes the number of stored patterns.

Extending previous studies, the present work investigates the role of Hebbian learning in forming a semantic memory. Still, it introduces new aspects: a further analysis of correlation among patterns, according to a hierarchical category structure, and a probabilistic experience so that features in an object are perceived with a given (higher or smaller) probability at each presentation. To deal with these aspects, we propose a new version of the Hebb rule, able to produce a not-symmetrical pattern of synapses, and we show that this rule automatically generates a distinction between marginal and relevant features and category representations based on shared and distinctive features.

To test this rule, we use a simplified auto-associative network with only one layer of units, as in older models of semantic memory. We know that this is a substantial simplification of reality and that a multilayer network is necessary to simulate the real neural processing circuits in the brain. The simplification is justified by the possibility of presenting results and synapse changes in a much more straightforward way, putting in evidence the virtues and limitations of the proposed Hebb rule within a simple auto-associative net. In the last section, we discuss how the present model can be extended in future work to fit novel, more complete models (e.g., by Garagnani et al.) or, more generally, with multilayer deep neural networks.

Finally, brain rhythms are known to play a significant role in cognition. Slow gamma oscillations of neural activity (in the 30–40 Hz range) have been frequently observed during various memory tasks involving working, episodic, and semantic aspects [33‐39].

Despite the presence of some controversies in the recent literature (see [40, 41] for cons), an influential theory suggests a role for the gamma rhythm in binding conceptual information [38, 42, 43]. Notably, Tallon-Baudry et al. [33, 34] suggest that neural discharges in the gamma band play a fundamental role in binding activities in areas involved in an object representation, merging bottom-up (i.e., sensory) and top-down (i.e., memory and past experience) information in a coherent entity. This idea is supported by the observation that stimuli for which subjects have a long-term memory representation lead to significantly larger gamma responses than unknown stimuli [44]. Indeed, gamma-band activity accompanies many cognitive functions, like attention [45‐47], arousal [48], object recognition [33, 34], and language perception [49]. Hence, Herrmann et al. suggested that in these tasks, the gamma band response realizes a match of sensory information with memory contents, a mechanism named by the authors “match and utilization model” [44, 50]. Gamma rhythms are observed in the hippocampus during episodic memory retrieval (often linked with a slower theta oscillation) [51‐53] and in the prefrontal cortex during working memory tasks [35, 54], all conditions where a relationship with semantics can be postulated [55].

Furthermore, a compromised link between semantic organization and brain oscillations has been observed in several neurological conditions. Some of them, like Alzheimer’s disease or semantic dementia, are characterized by a progressive loss of object recognition, possibly involving the theta-gamma code [27, 56, 57]. Others, like schizophrenia, are characterized by an illogical use of concepts and a possible involvement of alterations in the gamma rhythm [58‐60].

Since oscillatory aspects are of value, activity in each computational unit in our model is simulated through a neural mass model developed by the authors in recent years [61], in which brain rhythms arise from feedback interactions among local excitatory and inhibitory populations. Specifically, parameters are assigned so that each unit oscillates in the gamma band. Neural masses are a valuable alternative way to simulate oscillations compared with spiking neurons, assuming that a single unit describes the behavior of entire populations of neurons, coding for the same aspect. This formalism is more oriented to analyzing local field potentials or cortical activity reconstructed in regions of interest from high-density scalp EEG [62]. A taxonomy of various animals (mammals and birds), including several subcategories and salient-plus-marginal features, shows the main model virtues and limitations in representing semantic objects. A sensitivity analysis on some parameters involving the Hebb rule or the gamma rhythm generation is finally performed to test the robustness of the network, suggest some testable predictions, and unmask conditions leading to pathological behavior.

The present work proposes a model of semantic organization based on a feature representation of objects, attractor dynamics, and gamma-band oscillations. Compared with the recent literature, the fundamental new aspect consists of using an original asymmetric Hebb rule to deal with correlated patterns and distinguish between superordinate and subordinate concepts. Moreover, training is performed in a probabilistic environment, where not all features are simultaneously presented, but some can be lacking at any iteration. As discussed below, these aspects are original and can lead to an enrichment of existing models.

This study presents a neurocomputational model of semantic memory based on a feature representation of objects, Hebbian learning, and gamma-band synchronization. Compared with previous models, we suggest using a new version of the Hebb rule, joined with probabilistic learning, to deal with correlated patterns, hierarchical representations, and different feature saliency. Through simulations, we show that the network can distinguish between superior and inferior categories without errors and provides a different role for salient and marginal features based on their frequency in previous experience. Furthermore, simulations indicate that rapid synchronization among neural populations can be realized in the gamma band through trained excitatory and inhibitory synapses, provided the delay is 20–25 ms. Additionally, alterations in some parameters concerning the learning rule or the GABAergic interneurons provide interesting, although preliminary, indications about the possible causes of semantic disorders, characterized by confusion among categories or the recovery of a distorted reality. The model’s topological structure is relatively more straightforward than in recent sophisticated models implementing different cortical regions and their neuroanatomical connections. Still, in perspective, our results can be incorporated into multilayer networks, improving their capacity to realize a more consistent semantic description based on a hierarchical and probabilistic representation of reality.