Local Dynamics in Trained Recurrent Neural Networks

Alexander Rivkind and Omri Barak
Phys. Rev. Lett. 118, 258101 – Published 23 June 2017
PDFHTMLExport Citation

Abstract

Learning a task induces connectivity changes in neural circuits, thereby changing their dynamics. To elucidate task-related neural dynamics, we study trained recurrent neural networks. We develop a mean field theory for reservoir computing networks trained to have multiple fixed point attractors. Our main result is that the dynamics of the network’s output in the vicinity of attractors is governed by a low-order linear ordinary differential equation. The stability of the resulting equation can be assessed, predicting training success or failure. As a consequence, networks of rectified linear units and of sigmoidal nonlinearities are shown to have diametrically different properties when it comes to learning attractors. Furthermore, a characteristic time constant, which remains finite at the edge of chaos, offers an explanation of the network’s output robustness in the presence of variability of the internal neural dynamics. Finally, the proposed theory predicts state-dependent frequency selectivity in the network response.

  • Figure
  • Figure
  • Received 18 December 2015

DOI:https://doi.org/10.1103/PhysRevLett.118.258101

© 2017 American Physical Society

Physics Subject Headings (PhySH)

Networks

Authors & Affiliations

Alexander Rivkind1,2,* and Omri Barak1,2,†

  • 1Faculty of Medicine, Technion–Israel Institute of Technology, Haifa 32000, Israel
  • 2Network Biology Research Laboratories, Technion–Israel Institute of Technology, Haifa 32000, Israel

  • *arivkind@tx.technion.ac.il
  • omri.barak@gmail.com

Article Text (Subscription Required)

Click to Expand

Supplemental Material (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 118, Iss. 25 — 23 June 2017

Reuse & Permissions
Access Options
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review Letters

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×