Top

Cognitive Neurodynamics

Hint

Swipe to navigate through the articles of this issue

Published in:

20-04-2022 | Research Article

Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks

Author: Cecilia Jarne

Published in: Cognitive Neurodynamics | Issue 1/2023

Abstract

Different brain areas, such as the cortex and, more specifically, the prefrontal cortex, show great recurrence in their connections, even in early sensory areas. Several approaches and methods based on trained networks have been proposed to model and describe these regions. It is essential to understand the dynamics behind the models because they are used to build different hypotheses about the functioning of brain areas and to explain experimental results. The main contribution here is the description of the dynamics through the classification and interpretation carried out with a set of numerical simulations. This study sheds light on the multiplicity of solutions obtained for the same tasks and shows the link between the spectra of linearized trained networks and the dynamics of the counterparts. The patterns in the distribution of the eigenvalues of the recurrent weight matrix were studied and properly related to the dynamics in each task.

previous article Effect of disturbances in neuronal calcium and IP3 dynamics on β-amyloid production and degradation

next article Obstacle induced spiral waves in a multilayered Huber-Braun (HB) neuron model

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt 90 Tage mit der neuen Mini-Lizenz testen!

inform now

Available only for authorised users

Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G.S, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, .alwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. Software available from https://www.tensorflow.org/

Ahmadian Y, Fumarola F, Miller KD (2015) Properties of networks with partially structured and partially random connectivity. Phys Rev E 91:012820. https://doi.org/10.1103/PhysRevE.91.012820 CrossRef

Arjovsky M, Shah A, Bengio Y (2016) PMLR, New York, New York, USA. In: Proceedings of Machine Learning Research, vol 48, pp 1120–1128. http://proceedings.mlr.press/v48/arjovsky16.html

Asllani M, Lambiotte R, Carletti T (2018) Structure and dynamical behavior of non-normal networks. Sci Adv 4(12). https://doi.org/10.1126/sciadv.aau9403

Bagur S, Averseng M, Elgueda D, David S, Fritz J, Yin P, Shamma S, Boubenec Y, Ostojic S (2018) Go/No-Go task engagement enhances population representation of target stimuli in primary auditory cortex. Nat Commun 9(1):2529

Barak O (2017) Recurrent neural networks as versatile tools of neuroscience research. Curr Opin Neurobiol 46:1. https://doi.org/10.1016/j.conb.2017.06.003 CrossRef

Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157 CrossRef

Bi Z, Zhou C (2020) Understanding the computation of time using neural network models. Proc Natl Acad Sci 117(19):10530. https://doi.org/10.1073/pnas.1921609117.

Bondanelli G, Ostojic S (2020) Coding with transient trajectories in recurrent neural networks. PLoS Comput Biol 16(2):1. https://doi.org/10.1371/journal.pcbi.1007655 CrossRef

Chollet F, et al (2015) Keras. \(https://keras.io\)

Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning, Dec 2014

Cunningham JP, Yu BM (2014) Dimensionality reduction for large-scale neural recordingsDimensionality reduction for large-scale neural recordings. Nat Neurosci 17(11):1500. https://doi.org/10.1038/nn.3776 CrossRef

Dayan P, Abbott LF (2005) Theoretical neuroscience: computational and mathematical modeling of neural systems. The MIT Press

DePasquale B, Cueva CJ, Rajan K, Escola GS, Abbott LF (2018) full-FORCE: a target-based method for training recurrent networks. PLoS ONE 13(2):1. https://doi.org/10.1371/journal.pone.0191527 CrossRef

Durstewitz D (2017) A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLoS Comput Biol 13(6):1. https://doi.org/10.1371/journal.pcbi.1005542 CrossRef

Floyd T (2003) Digital fundamentals. Prentice Hall. https://books.google.com.ar/books?id=gjAfAQAAIAAJ

García del Molino LC, Pakdaman K, Touboul J, Wainrib G (2013) Synchronization in random balanced networks. Phys Rev E 88:042824. https://doi.org/10.1103/PhysRevE.88.042824

Girko V (1985) Circular law. Theory Probab Appl 29(4):694. https://doi.org/10.1137/1129095 CrossRef

Gisiger T, Boukadoum M (2011) Mechanisms gating the flow of information in the cortex: what they might look like and what their uses may be. Front Comput Neurosci 5:1. https://doi.org/10.3389/fncom.2011.00001 CrossRef

Goldman MS (2009) Memory without feedback in a neural network. Neuron 61(4):621. https://doi.org/10.1016/j.neuron.2008.12.012 CrossRef

Gudowska-Nowak E, Nowak MA, Chialvo DR, Ochab JK, Tarnowski W (2020) From synaptic interactions to collective dynamics in random neuronal networks models: critical role of eigenvectors and transient behavior. Neural Comput 32(2):395. https://doi.org/10.1162/neco_a_01253 CrossRef

Hassabis D, Kumaran D, Summerfield C, Botvinick M (2017) Neuroscience-inspired artificial intelligence. Neuron 95(2):245. https://doi.org/10.1016/j.neuron.2017.06.011 CrossRef

Heeger DJ, Mackey WE (2019) Oscillatory recurrent gated neural integrator circuits (ORGaNICs), a unifying theoretical framework for neural dynamics. Proc Natl Acad Sci 116(45):22783. https://doi.org/10.1073/pnas.1911633116 CrossRef

Hopfield JJ (1984) Neurons with graded response have collective computational properties like those of two-state neurons. Proc Natl Acad Sci 81(10):3088. https://doi.org/10.1073/pnas.81.10.3088 CrossRef

Ingrosso A, Abbott LF (2019) Training dynamically balanced excitatory-inhibitory networks. PLoS ONE 14(8):1. https://doi.org/10.1371/journal.pone.0220547 CrossRef

Jarne C (2021a) Multitasking in RNN: an analysis exploring the combination of simple tasks. J Phys 2(1):015009. https://doi.org/10.1088/2632-072x/abdee3 CrossRef

Jarne C (2021b) What you need to know to train recurrent neural networks to make flip flops memories and more. https://arxiv.org/abs/2010.07858

Jarne C, Laje R (2020) Graceful degradation of recurrent neural networks as a function of network size, memory length, and connectivity damage. https://arxiv.org/abs/1906.01094

Kao JC (2019) Considerations in using recurrent neural networks to probe neural dynamics. J Neurophysiol 122(6):2504. https://doi.org/10.1152/jn.00467.2018

Kao TC, Hennequin G (2019) Neuroscience out of control: control-theoretic perspectives on neural circuit dynamics. Curr Opin Neurobiol 58:122. https://doi.org/10.1016/j.conb.2019.09.001 CrossRef

Kim R, Li Y, Sejnowski TJ (2019) Simple framework for constructing functional spiking recurrent neural network. Proc Natl Acad Sci 116(45):22811. https://doi.org/10.1073/pnas.1905926116 CrossRef

Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. http://arxiv.org/abs/1412.6980

Kuroki S, Isomura T (2018) Task-related synaptic changes localized to small neuronal population in recurrent neural network cortical models. Front Comput Neurosci 12:83. https://doi.org/10.3389/fncom.2018.00083.

Laje R, Buonomano DV (2013) Robust timing and motor patterns by taming chaos in recurrent neural networks. Nat Neurosci 16(7):925. https://doi.org/10.1038/nn.3405 CrossRef

Landau ID, Sompolinsky H (2018) Coherent chaos in a recurrent neural network with structured connectivity. PLoS Comput Biol 14(12):1. https://doi.org/10.1371/journal.pcbi.1006309 CrossRef

Lillicrap TP, Santoro A, Marris L, Akerman CJ, Hinton G (2020) Backpropagation and the brain. Nat Rev Neurosc. https://doi.org/10.1038/s41583-020-0277-3

Maheswaranathan N, Williams A, Golub M, Ganguli S, Sussillo D (2019a) Universality and individuality in neural dynamics across large populations of recurrent networks. In: Wallach H, Larochelle H, Beygelzimer A, Fox E, Garnett R (eds) Advances in neural information processing systems 32. Curran Associates Inc, Red Hook, pp 15629–15641

Maheswaranathan N, Williams A, Golub M, Ganguli S, Sussillo D (2019b) Universality and individuality in neural dynamics across large populations of recurrent networks. In: Advances in neural information processing systems 32. Curran Associates, Inc., Red Hooks. pp 15,629–15,641. http://papers.nips.cc/paper/9694-universality-and-individuality-in-neural-dynamics-across-large-populations-of-recurrent-networks.pdf

Mante V, Sussillo D, Shenoy KV, Newsome WT (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503(7474):78. https://doi.org/10.1038/nature12742 CrossRef

Martí D, Brunel N, Ostojic S (2018) Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks. Phys Rev E 97:062314. https://doi.org/10.1103/PhysRevE.97.062314 CrossRef

Murphy BK, Miller KD (2009) Balanced amplification: a new mechanism of selective amplification of neural activity patterns. Neuron 61(4):635. https://doi.org/10.1016/j.neuron.2009.02.005 CrossRef

Nambu A, Tsuda I, Mima T (2020) Oscillology: nonlinear neural oscillations. Neurosci Res 156:1 https://doi.org/10.1016/j.neures.2020.05.007.

Pandarinath C, O’Shea DJ, Collins J, Jozefowicz R, Stavisky SD, Kao JC, Trautmann EM, Kaufman MT, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV, Abbott LF, Sussillo D (2018) Inferring single-trial neural population dynamics using sequential auto-encoders. Nat Methods 15(10):805. https://doi.org/10.1038/s41592-018-0109-9 CrossRef

Pascanu R, Mikolov T, Bengio Y(2013) On the difficulty of training Recurrent Neural Networks. In: ICML’13: JMLR: W&CP vol 28

Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825

Pehlevan C, Ali F, Ölveczky B.P(2018) Flexibility in motor timing constrains the topology and dynamics of pattern generator circuits. Nat Commun. https://doi.org/10.1038/s41467-018-03261-5

Pollock E, Jazayeri M (2019), Engineering recurrent neural networks from task-relevant manifolds and dynamics. bioRxiv. https://doi.org/10.1101/2019.12.19.883207

Rajan K, Abbott LF (2006) Eigenvalue spectra of random matrices for neural networks. Phys Rev Lett 97:188104. https://doi.org/10.1103/PhysRevLett.97.188104

Remington ED, Narain D, Hosseini EA, Jazayeri M (2018) Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98(5):1005. https://doi.org/10.1016/j.neuron.2018.05.020 CrossRef

Rivkind A, Barak O (2017) Local dynamics in trained recurrent neural networks. Phys Rev Lett 118:258101. https://doi.org/10.1103/PhysRevLett.118.258101 CrossRef

Russo AA, Khajeh R, Bittner SR, Perkins SM, Cunningham JP, Abbott L, Churchland MM (2020) Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron 107(4):745. https://doi.org/10.1016/j.neuron.2020.05.020.

Salatiello A, Giese MA (2020) Recurrent neural network learning of performance and intrinsic population dynamics from sparse neural data. In: Farkaš I, Masulli P, Wermter S (eds) Artificial neural networks and machine learning - ICANN 2020. Springer International Publishing, Cham, pp 874–886 CrossRef

Sengupta B, Friston K.J(2018) How Robust are Deep Neural Networks?. How robust are deep neural networks?

Shi X, Chen Z, Wang H, Yeung DY, Wong WK, WOO W.C, (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems 28. Curran Associates Inc, Red Hook, pp 802–810

Siegel M, Buschman TJ, Miller EK (2015) Cortical information flow during flexible sensorimotor decisions. Nat Rev Neurosci. https://doi.org/10.1126/science.aab0551

Strogatz SHSHa (2015) Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering, 2 ed. Westview Press, a member of the Perseus Books Group, Boulder. https://search.library.wisc.edu/catalog/9910223127702121

Sussillo D (2014) Neural circuits as computational dynamical systems. Curr Opin Neurobiol 25:156. https://doi.org/10.1016/j.conb.2014.01.008 CrossRef

Sussillo D, Barak O (2013) Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput 25(3):626. https://doi.org/10.1162/NECO_a_00409 CrossRef

Vibert JF, Pakdaman K, Azmy N (1994) Interneural delay modification synchronizes biologically plausible neural networks. Neural Netw 7(4):589. https://doi.org/10.1016/0893-6080(94)90039-6 CrossRef

Vyas S, Golub MD, Sussillo D, Shenoy KV (2020) Computation through neural population dynamics. Annu Rev Neurosci 43(1):249. https://doi.org/10.1146/annurev-neuro-092619-094115 CrossRef

Williams AH, Kim TH, Wang F, Vyas S, Ryu SI, Shenoy KV, Schnitzer M, Kolda TG, Ganguli S (2018) Unsupervised discovery of demixed, low-dimensional neural dynamics across multiple timescales through tensor component analysis. Neuron 98(6):1099. https://doi.org/10.1016/j.neuron.2018.05.015.

Zhang GH, Nelson DR (2019) Eigenvalue repulsion and eigenvector localization in sparse non-Hermitian random matrices. Phys Rev E 100:052315. https://doi.org/10.1103/PhysRevE.100.052315

Zhou Q, Jin T, Zhao H (2009) Correlation between eigenvalue spectra and dynamics of neural networks. Neural Comput 21(10):2931. https://doi.org/10.1162/neco.2009.12-07-671

Title: Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks
Author: Cecilia Jarne
Publication date: 20-04-2022
Publisher: Springer Netherlands
Published in: Cognitive Neurodynamics / Issue 1/2023
Print ISSN: 1871-4080
Electronic ISSN: 1871-4099
DOI: https://doi.org/10.1007/s11571-022-09802-5

Springer Professional

Hint

Swipe to navigate through the articles of this issue

Abstract

Please log in to get access to this content

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2023

Generalised exponential-Gaussian distribution: a method for neural reaction time analysis

Culture and gender modulate dlPFC integration in the emotional brain: evidence from dynamic causal modeling

Human factors engineering of BCI: an evaluation for satisfaction of BCI based on motor imagery

Generalizable epileptic seizures prediction based on deep transfer learning

Formal perspectives on shared interbrain activity in social communication

Brain potentials related to violent video clips