Top

Neural Processing Letters

Published in:

28-05-2019

Increasing Capacity of Association Memory by Means of Synaptic Clustering

Authors: Viacheslav Osaulenko, Bernard Girau, Oleksandr Makarenko, Patrick Henaff

Published in: Neural Processing Letters | Issue 3/2019

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Making association is an essential property of human cognition. Systems that try to mimic this process and to make a coherent model of the world should have robust and high capacity association memory. Findings of nonlinear properties of dendritic tree suggest an alternative way how neurons can store associations. In this paper, we present a minimalistic neuron model with clustered synapses and show that it provides much higher association memory capacity compared to traditional models. Due to properties of sparse activation and tracking higher-order correlations in the input pattern an individual neuron can recognize thousands of patterns. Theoretical examination shows that this high capacity is reached because learning exact combinations of active neurons extends the dimension of an input space and thus increases pattern separability. We argue that such beneficial computational properties is realized in biological neural networks through synaptic clustering and sustaining sparse activity in memory-related areas.

previous article The Parameter Identification of PMSM Based on Improved Cuckoo Algorithm

next article Sample Based Fast Adversarial Attack Method

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci 79(8):2554. https://doi.org/10.1073/pnas.79.8.2554 MathSciNetCrossRefMATH

Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 222(5197):960. https://doi.org/10.1038/222960a0 CrossRef

Sommer FT, Wennekers T (2001) Associative memory in networks of spiking neurons. Neural Netw 14(6–7):825. https://doi.org/10.1016/S0893-6080(01)00064-8 CrossRef

Zenke F, Agnes EJ, Gerstner W (2015) Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nature Commun 6:6922. https://doi.org/10.1038/ncomms7922 CrossRef

Hunter R, Cutsuridis V, Cobb S, Graham B (2011) Improving associative memory in a model network of two- compartment spiking neurons, thesis submitted

Hebb (1949) DO the organization of behavior. https://doi.org/10.2307/1418888. http://www.amazon.com/dp/0805843000 CrossRef

London M, Häusser M (2005) Dendritic Comput. Ann Rev Neurosci 28(1):503. https://doi.org/10.1146/annurev.neuro.28.061604.135703 CrossRef

Stuart GJ, Spruston N (2015) Dendritic integration: 60 years of progress. Nature Neurosci 18(12):1713. https://doi.org/10.1038/nn.4157 CrossRef

Du K, Wu YW, Lindroos R, Liu Y, Rózsa B, Katona G, Ding JB, Kotaleski JH (2017) Cell-typespecific inhibition of the dendritic plateau potential in striatal spiny projection neurons. Proc Natl Acad Sci 114(36):E7612. https://doi.org/10.1073/pnas.1704893114 CrossRef

10.

Mel BW, Schiller J, Poirazi P (2017) Synaptic plasticity in dendrites: complications and coping strategies. Curr Opin Neurobiol 43:177. https://doi.org/10.1016/j.conb.2017.03.012 CrossRef

11.

Branco T, Häusser M (2010) The single dendritic branch as a fundamental functional unit in the nervous system. Curr Opin Neurobiol 20(4):494. https://doi.org/10.1016/j.conb.2010.07.009 CrossRef

12.

Sjöström PJ, Rancz A, Roth A, Häusser M, Sjostrom PJ, Rancz EA, Roth A, Hausser M, PJ S, EA R, RA, HM, Sjostrom PJ, Rancz EA, Roth A, Hausser M, (2008) Dendritic excitability and synaptic plasticity. Physiol Rev 88(2):769. https://doi.org/10.1152/physrev.00016.2007 CrossRef

13.

Kastellakis G, Cai DJ, Mednick SC, Silva AJ, Poirazi P (2015) Synaptic clustering within dendrites: an emerging theory of memory formation. Prog Neurobiol 126:19. https://doi.org/10.1016/j.pneurobio.2014.12.002 CrossRef

14.

Bhalla UUS (2017) Dendrites, deep learning, and sequences in the hippocampus. Hippocampus 2014(6). https://doi.org/10.1002/hipo.22806 CrossRef

15.

Poirazi P, Mel BW (2001) Impact of active dendrites and structural plasticity on the memory capacity of neural tissue. Neuron 29(3):779. https://doi.org/10.1016/S0896-6273(01)00252-5 CrossRef

16.

Frank AC, Huang S, Zhou M, Gdalyahu A, Kastellakis G, Silva TK, Lu E, Wen X, Poirazi P, Trachtenberg JT, Silva AJ (2018) Hotspots of dendritic spine turnover facilitate clustered spine addition and learning and memory. Nat Commun 9(1):422. https://doi.org/10.1038/s41467-017-02751-2 CrossRef

17.

Bengio S, Bengio Y, Cloutier J (1995) On the search for new learning rules for ANNs. Neural Process Lett 2(4):26. https://doi.org/10.1007/BF02279935 CrossRef

18.

Brea J, Gerstner W (2016) Does computational neuroscience need new synaptic learning paradigms? Current opinion in behavioral sciences, pp 1–6. https://doi.org/10.1016/j.cobeha.2016.05.012. http://linkinghub.elsevier.com/retrieve/pii/S2352154616301048 CrossRef

19.

Tran-Van-Minh A, Cazé RD, Abrahamsson T, Cathala L, Gutkin BS, DiGregorio DA (2015) Contribution of sublinear and supralinear dendritic integration to neuronal computations. Front Cell Neurosci 9(March):1. https://doi.org/10.3389/fncel.2015.00067 CrossRef

20.

Suvrathan A (2018) Beyond STDP towards diverse and functionally relevant plasticity rules. Curr Opin Neurobiol 54:12. https://doi.org/10.1016/j.conb.2018.06.011 CrossRef

21.

Mel BW, Koch C (1990) Sigma-pi learning: on radial basis functions and cortical associative learning. In: Advances in neural information processing systems. pp 474–481

22.

Mel BW (1991) The clusteron: toward a simple abstraction for a complex neuron. Nips, pp 35–42

23.

Plate TA (2000) Randomly connected sigma-pi neurons can form associator networks. Netw Comput Neural Syst 11:321–332. https://doi.org/10.1088/0954-898X_11_4_305 CrossRefMATH

24.

Baldi P (1987) Number of stable points for spin-glasses and neural networks of higher orders. Phys Rev Lett 58(9):913MathSciNetCrossRef

25.

Giles CL, Maxwell T (1987) Learning, invariance, and generalization in high-order neural networks. Appl Opt 26:4972. https://doi.org/10.1364/AO.26.004972 CrossRef

26.

Krotov D, Hopfield JJ (2016) Dense associative memory for pattern recognition (Nips). arXiv:1606.01164

27.

Guerguiev J, Lillicrap TP, Richards BA (2017) Towards deep learning with segregated dendrites. eLife 6:1. https://doi.org/10.7554/eLife.22901 CrossRef

28.

Wu X, Liu X, Li W, Wu Q (2018) Improved expressivity through dendritic neural networks. In: Advances in neural information processing systems. pp 8057–8068

29.

Demircigil M, Heusel J, Löwe M, Upgang S, Vermet F (2017) On a model of associative memory with huge storage capacity. J Stat Phys 168(2):288. https://doi.org/10.1007/s10955-017-1806-y MathSciNetCrossRefMATH

30.

Poirazi P, Mel B (2000) Memory capacity of linear vs . nonlinear models of dendritic integration. Advances in neural information processing systems (November), p 157

31.

Sjöström PJ, Rancz a, Roth a, Häusser M (2010) Dendritic excitability and synaptic plasticity dendritic excitability and synaptic plasticity. Physiol Rev (November 2009) 1. https://doi.org/10.1152/physrev.00016.2007 CrossRef

32.

Poirazi P, Brannon T, Mel BW (2003) Pyramidal neuron as two-layered neural network. Neuron 37:989CrossRef

33.

Hawkins J, Ahmad S (2016) Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Front Neural Circuits 10(March):1. https://doi.org/10.3389/fncir.2016.00023 CrossRef

34.

van Bommel B, Mikhaylova M (2016) Talking to the neighbours: the molecular and physiological mechanisms of clustered synaptic plasticity. Neurosci Biobeheav Rev 71:352. https://doi.org/10.1016/j.neubiorev.2016.09.016 CrossRef

35.

Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14(4):481. https://doi.org/10.1016/j.conb.2004.07.007 CrossRef

36.

Quiroga RQ, Kreiman G, Koch C, Fried I (2008) Sparse but not ’Grandmother-cell’ coding in the medial temporal lobe. Trends Cognit Sci 12(3):87. https://doi.org/10.1016/j.tics.2007.12.003 CrossRef

37.

Luczak A, McNaughton BL, Harris KD (2015) Packet-based communication in the cortex. Nat Rev Neurosci 16(12):745. https://doi.org/10.1038/nrn4026 CrossRef

38.

Gritsenko VI, Rachkovskij DA, Frolov AA, Gayler R, Kleyko D, Osipov E (2017) Neural distributed autoassociative memories: a survey. Cybern Comput Eng 2(188):5

39.

Palm G (2013) Neural associative memories and sparse coding. Neural Netw 37:165. https://doi.org/10.1016/j.neunet.2012.08.013 CrossRef

40.

Fauth M, Tetzlaff C (2016) Opposing effects of neuronal activity on structural plasticity. Front Neuroanat 10(June):1. https://doi.org/10.3389/fnana.2016.00075 CrossRef

41.

Knoblauch A (2005) Neural associative memory for brain modeling and information retrieval. Inf Process Lett 95(6 SPEC. ISS.):537. https://doi.org/10.1016/j.ipl.2005.05.021 MathSciNetCrossRefMATH

42.

Knoblauch A, Palm G, Sommer FT (2010) Memory capacities for synaptic and structural plasticity. Neural Comput 22(2):289. https://doi.org/10.1162/neco.2009.08-07-588 MathSciNetCrossRefMATH

43.

Graham B, Willshaw D (1997) Capacity and information efficiency of the associative net. Netw Comput Neural Syst 8(1):35. https://doi.org/10.1088/0954-898X/8/1/005 CrossRefMATH

44.

Markram H, Muller E et al (2015) Reconstruction and simulation of neocortical microcircuitry. Cell 163(2):456. https://doi.org/10.1016/j.cell.2015.09.029 CrossRef

45.

Lane SH, Handelman DA, Gelfand JJ (1992) Theory and development of higher-order CMAC neural networks. IEEE Control Syst 12(2):23. https://doi.org/10.1109/37.126849 CrossRef

46.

Taylor JG, Coombes S (1993) Learning higher order correlations. Neural Netw 6:423CrossRef

47.

Chen D, Li S, Wu Q (2018) Rejecting chaotic disturbances using a super-exponential-zeroing neurodynamic approach for synchronization of chaotic sensor systems. Sensors 19(1):74. https://doi.org/10.3390/s19010074 CrossRef

48.

Ahmad S, Hawkins J (2015) Properties of sparse distributed representations and their application to hierarchical temporal memory. https://doi.org/10.1063/1.4918346, arXiv:1503.07469 CrossRef

49.

Ahmad S, Hawkins J (2016) How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites. arXiv:1601.00720 [q–bio.NC]

50.

Kanerva P (1988) Sparse distributed memory. MIT Press, BostonMATH

51.

Kanerva P (2009) Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cognit Comput 1(2):139. https://doi.org/10.1007/s12559-009-9009-8 CrossRef

52.

Plate TA (2012) Distributed representations. Encycl Cognit Sci 59:77

53.

Rachkovskij DA, Kussul EM, Baidyk TN (2013) Building a world model with structure-sensitive sparse binary distributed representations. Biol Inspired Cognit Archit 3:64. https://doi.org/10.1016/j.bica.2012.09.004 CrossRef

54.

Knoblauch A, Sommer FT (2016) Structural plasticity, effectual connectivity, and memory in cortex. Front Neuroanat 10(June):1. https://doi.org/10.3389/fnana.2016.00063 CrossRef

Title: Increasing Capacity of Association Memory by Means of Synaptic Clustering
Authors: Viacheslav Osaulenko
Bernard Girau
Oleksandr Makarenko
Patrick Henaff
Publication date: 28-05-2019
Publisher: Springer US
Published in: Neural Processing Letters / Issue 3/2019
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI: https://doi.org/10.1007/s11063-019-10051-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2019

Synchronizing Chaotic Systems with Uncertain Model and Unknown Interference Using Sliding Mode Control and Wavelet Neural Networks

Online Hard Region Mining for Semantic Segmentation

Exponential Stabilization for Hybrid Recurrent Neural Networks by Delayed Noises Rooted in Discrete Observations of State and Mode

A Novel Noise-Enhanced Back-Propagation Technique for Weak Signal Detection in Neyman–Pearson Framework

Application of Mind Evolutionary Algorithm and Artificial Neural Networks for Prediction of Profile and Flatness in Hot Strip Rolling Process

Surface Electromyography-Based Daily Activity Recognition Using Wavelet Coherence Coefficient and Support Vector Machine