Abstract
A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Change history
25 April 2018
In the version of this article initially published, equation (7) read It should have read The error has been corrected in the HTML and PDF versions of the article.
References
Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).
Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189–208 (1948).
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993).
O'Keefe, J. & Nadel, L. The Hippocampus as a Cognitive Map (Clarendon Press, 1978).
Muller, R.U., Stead, M. & Pach, J. The hippocampus as a cognitive graph. J. Gen. Physiol. 107, 663–694 (1996).
Penny, W.D., Zeidman, P. & Burgess, N. Forward and backward inference in spatial cognition. PLOS Comput. Biol. 9, e1003383 (2013).
Foster, D.J., Morris, R.G. & Dayan, P. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10, 1–16 (2000).
Gustafson, N.J. & Daw, N.D. Grid cells, place cells, and geodesic generalization for spatial reinforcement learning. PLOS Comput. Biol. 7, e1002235 (2011).
Hafting, T., Fyhn, M., Molden, S., Moser, M.B. & Moser, E.I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005).
Gershman, S.J., Moore, C.D., Todd, M.T., Norman, K.A. & Sederberg, P.B. The successor representation and temporal context. Neural Comput. 24, 1553–1568 (2012).
Russek, E.M., Momennejad, I., Botvinick, M.M., Gershman, S.J. & Daw, N.D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. Preprint at https://doi.org/10.1101/083857 (2017).
Schapiro, A.C., Turk-Browne, N.B., Norman, K.A. & Botvinick, M.M. Statistical learning of temporal community structure in the hippocampus. Hippocampus 26, 3–8 (2016).
Dordek, Y., Meir, R. & Derdikman, D. Extracting grid characteristics from spatially distributed place cell inputs using non-negative PCA. Preprint at https://arxiv.org/abs/1505.03711 (2015).
Mehta, M.R., Quirk, M.C. & Wilson, M.A. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25, 707–715 (2000).
Muller, R.U. & Kubie, J.L. The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J. Neurosci. 7, 1951–1968 (1987).
Skaggs, W.E. & McNaughton, B.L. Spatial firing properties of hippocampal CA1 populations in an environment containing two visually identical regions. J. Neurosci. 18, 8455–8466 (1998).
Alvernhe, A., Save, E. & Poucet, B. Local remapping of place cell firing in the Tolman detour task. Eur. J. Neurosci. 33, 1696–1705 (2011).
Hollup, S.A., Molden, S., Donnett, J.G., Moser, M.B. & Moser, E.I. Accumulation of hippocampal place fields at the goal location in an annular water maze task. J. Neurosci. 21, 1635–1644 (2001).
Strange, B.A., Witter, M.P., Lein, E.S. & Moser, E.I. Functional organization of the hippocampal longitudinal axis. Nat. Rev. Neurosci. 15, 655–669 (2014).
Garvert, M.M., Dolan, R.J. & Behrens, T.E. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex. eLife 6, 17086 (2017).
Deuker, L., Bellmund, J.L., Navarro Schröder, T. & Doeller, C.F. An event map of memory space in the hippocampus. eLife 5, e16534 (2016).
Krupic, J., Bauza, M., Burton, S., Barry, C. & O'Keefe, J. Grid cell symmetry is shaped by environmental geometry. Nature 518, 232–235 (2015).
Derdikman, D. et al. Fragmentation of grid cell maps in a multicompartment environment. Nat. Neurosci. 12, 1325–1332 (2009).
Carpenter, F., Manson, D., Jeffery, K., Burgess, N. & Barry, C. Grid cells form a global representation of connected environments. Curr. Biol. 25, 1176–1182 (2015).
Mazumder, R., Hastie, T. & Tibshirani, R. Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11, 2287–2322 (2010).
Mahadevan, S. & Maggioni, M. Proto-value functions: a Laplacian framework for learning representation and control in markov decision processes. J. Mach. Learn. Res. 8, 2169–2231 (2007).
Bonnevie, T. et al. Grid cells require excitatory drive from the hippocampus. Nat. Neurosci. 16, 309–317 (2013).
Hales, J.B. et al. Medial entorhinal cortex lesions only partially disrupt hippocampal place cells and hippocampus-dependent place memory. Cell Rep. 9, 893–901 (2014).
Solway, A. et al. Optimal behavioral hierarchy. PLoS Comput. Biol. 10, e1003779 (2014).
Ribas-Fernandes, J.J. et al. A neural signature of hierarchical reinforcement learning. Neuron 71, 370–379 (2011).
Schlesiger, M.I. et al. The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity. Nat. Neurosci. 18, 1123–1132 (2015).
Blum, K.I. & Abbott, L.F. A model of spatial map formation in the hippocampus of the rat. Neural Comput. 8, 85–93 (1996).
Levy, W.B., Hocking, A.B. & Wu, X. Interpreting hippocampal function as recoding and forecasting. Neural Netw. 18, 1242–1264 (2005).
Hassabis, D. & Maguire, E.A. The construction system of the brain. Phil. Trans. R. Soc. Lond. B 364, 1263–1271 (2009).
Buckner, R.L. The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27–48, C1–C8 (2010).
Barry, C., Hayman, R., Burgess, N. & Jeffery, K.J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682–684 (2007).
Fanselow, M.S. From contextual fear to a dynamic view of memory systems. Trends Cogn. Sci. 14, 7–15 (2010).
Wiltgen, B.J., Sanders, M.J., Anagnostaras, S.G., Sage, J.R. & Fanselow, M.S. Context fear learning in the absence of the hippocampus. J. Neurosci. 26, 5484–5491 (2006).
Maurer, A.P. & McNaughton, B.L. Network and intrinsic cellular mechanisms underlying theta phase precession of hippocampal neurons. Trends Neurosci. 30, 325–333 (2007).
Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).
Pezzulo, G., van der Meer, M.A., Lansink, C.S. & Pennartz, C.M. Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn. Sci. 18, 647–657 (2014).
Hasselmo, M.E. & Stern, C.E. Theta rhythm and the encoding and retrieval of space and time. Neuroimage 85, 656–666 (2014).
Ekstrom, A.D., Meltzer, J., McNaughton, B.L. & Barnes, C.A. NMDA receptor antagonism blocks experience-dependent expansion of hippocampal “place fields”. Neuron 31, 631–638 (2001).
Hafting, T., Fyhn, M., Bonnevie, T., Moser, M.-B. & Moser, E.I. Hippocampus-independent phase precession in entorhinal grid cells. Nature 453, 1248–1252 (2008).
Sutton, R.S. DYNA, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin 2, 160–163 (1991).
Zhang, J., Springenberg, J.T., Boedecker, J. & Burgard, W. Deep reinforcement learning with successor features for navigation across similar environments. IEEE/RSJ International Conference on Intelligent Robots and Systems (2017).
Momennejad, I. et al. The successor representation in human reinforcement learning. Preprint at https://doi.org/10.1101/083824 (2017).
Howard, M.W., Fotedar, M.S., Datey, A.V. & Hasselmo, M.E. The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychol. Rev. 112, 75–116 (2005).
Krupic, J., Burgess, N. & O'Keefe, J. Neural representations of location composed of spatially periodic bands. Science 337, 853–857 (2012).
Sutton, R. & Barto, A. Reinforcement Learning: an Introduction (MIT Press, 1998).
Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).
Bialek, W. Biophysics: Searching for Principles (Princeton University Press, 2012).
Weng, J., Zhang, Y. & Hwang, W. Candid covariance-free incremental principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1034–1040 (2003).
Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 888–905 (2000).
Acknowledgements
We are grateful to T. Behrens, I. Mommenejad, and K. Miller for helpful discussions, and to A. Mathis, H. Sanders, M. Chadwick, and D. Kumaran for comments on an earlier draft of the paper. This research was supported by the NSF Collaborative Research in Computational Neuroscience (CRCNS) Program Grant IIS-120 7833 and The John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the funding agencies.
Author information
Authors and Affiliations
Contributions
All authors conceived the model and wrote the manuscript. Simulations were carried out by K.S.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Note and Supplementary Figures 1–19 (PDF 13297 kb)
Rights and permissions
About this article
Cite this article
Stachenfeld, K., Botvinick, M. & Gershman, S. The hippocampus as a predictive map. Nat Neurosci 20, 1643–1653 (2017). https://doi.org/10.1038/nn.4650
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.4650
This article is cited by
-
Conjunctive encoding of exploratory intentions and spatial information in the hippocampus
Nature Communications (2024)
-
Prediction error processing and sharpening of expected information across the face-processing hierarchy
Nature Communications (2024)
-
Transcranial direct current stimulation suggests a causal role of the medial prefrontal cortex in learning social hierarchy
Communications Biology (2024)
-
Local prediction-learning in high-dimensional spaces enables neural networks to plan
Nature Communications (2024)
-
A generative model of memory construction and consolidation
Nature Human Behaviour (2024)