Skip to main content
SpringerLink
Log in

Sensory Fusion

  • Chapter
  • First Online:
Perception-Action Cycle

Part of the book series: Springer Series in Cognitive and Neural Systems ((SSCNS))

Abstract

Multisensory integration is known to occur in many regions of the brain, and involves several aspects of our daily life; however, the underlying neural mechanisms are still insufficiently understood. This chapter presents two mathematical models of multisensory integration, inspired by real neurophysiological systems. The first considers the integration of visual and auditory stimuli, as it occurs in the superior colliculus (a subcortical region involved in orienting eyes and head toward external events). The second model considers the integration of tactile stimuli and visual stimuli close to the body to form the perception of the peripersonal space (the space immediately around our body, within which we can interact with the external world). Although devoted to two specific problems, the mechanisms delineated in the models (lateral inhibition and excitation, nonlinear neuron characteristics, recurrent connections, and competition) may govern more generally the fusion of senses in the brain. The models, besides improving our comprehension of brain function, may drive future neurophysiological experiments and provide valuable ideas to build artificial systems devoted to sensory fusion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Anastasio TJ, Patton PE (2003) A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J Neurosci 23:6713–6727

    CAS  PubMed  Google Scholar 

  2. Anastasio TJ, Patton PE, Belkacem-Boussaid K (2000) Using bayes rule to model multisensory enhancement in the superior colliculus. Neural Comput 12:1165–1187

    Article  CAS  PubMed  Google Scholar 

  3. Avillac M, Denève S, Olivier E et al (2005) Reference frames for representing visual and tactile locations in parietal cortex. Nat Neurosci 8:941–949

    CAS  PubMed  Google Scholar 

  4. Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 43:487–493

    Google Scholar 

  5. Bolognini N, Rasi F, Coccia M et al (2005) Visual search improvement in hemianopic patients after audio-visual stimulation. Brain 128:2830–2842

    Article  PubMed  Google Scholar 

  6. Bonifazi S, Farnè A, Rinaldesi L et al (2007) Dynamic size-change of peri-hand space through tool-use: spatial extension or shift of the multi-sensory area. J Neuropsychol 1:101–114

    Article  CAS  PubMed  Google Scholar 

  7. Bremmer F, Schlack A, Shah NJ et al (2001) Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys. Neuron 29:287–296

    Article  CAS  PubMed  Google Scholar 

  8. Calvert GA, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT, Cambridge

    Google Scholar 

  9. Colonius H, Diederich A (2004) Why aren’t all deep superior colliculus neurons multisensory? A Bayes’ ratio analysis. Cogn Affect Behav Neurosci 4:344–353

    Article  PubMed  Google Scholar 

  10. Cooke DF, Graziano MS (2004) Sensorimotor integration in the precentral gyrus: polysensory neurons and defensive movements. J Neurophysiol 91:1648–1660

    Article  PubMed  Google Scholar 

  11. Cuppini C, Ursino M, Magosso E et al (2008) A neural network model of multisensory maturation in superior colliculus neurons. Program No. 457.5. 2008 Neuroscience Meeting Planner. Washington, DC: Society for Neuroscience, 2008. Online

    Google Scholar 

  12. Dalton P, Doolittle N, Nagata H et al (2000) The merging of the senses: integration of subthreshold taste and smell. Nat Neurosci 3:431–432

    Article  CAS  PubMed  Google Scholar 

  13. Denève S, Latham PE, Pouget A (2001) Efficient computation and cue integration with noisy population codes. Nat Neurosci 4:826–831

    Article  PubMed  Google Scholar 

  14. di Pellegrino G, Làdavas E, Farnè A (1997) Seeing where your hands are. Nature 388:730

    Google Scholar 

  15. Driver J, Spence C (2000) Multisensory perception: beyond modularity and convergence. Curr Biol 10:R731-R735

    Article  CAS  PubMed  Google Scholar 

  16. Duhamel JR, Colby CL, Goldberg ME (1998) Ventral intraparietal area of the macaque: congruent visual and somatic response properties. J Neurophysiol 79:126–136

    CAS  PubMed  Google Scholar 

  17. Edwards SB, Ginsburgh CL, Henkel CK et al (1979) Sources of subcortical projections to the superior colliculus in the cat. J Comp Neurol 184:309–330

    Article  CAS  PubMed  Google Scholar 

  18. Farnè A, Làdavas E (2000) Dynamic size-change of hand peripersonal space following tool use. Neuroreport 11:1645–1649

    Article  PubMed  Google Scholar 

  19. Farnè A, Iriki A, Làdavas E (2005) Shaping multisensory action-space with tools: evidence from patients with cross-modal extinction. Neuropsychologia 43:238–248

    Article  PubMed  Google Scholar 

  20. Farnè A, Serino A, Làdavas E (2007) Dynamic size-change of peri-hand space following tool-use: determinants and spatial characteristics revealed through cross-modal extinction. Cortex 43:436–443

    Article  PubMed  Google Scholar 

  21. Fogassi L, Gallese V, Fadiga L et al (1996) Coding of peripersonal space in inferior premotor cortex (area F4). J Neurophysiol 76:141–157

    CAS  PubMed  Google Scholar 

  22. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10:278–285

    Article  PubMed  Google Scholar 

  23. Graziano MS, Cooke DF (2006) Parieto-frontal interactions, personal space, and defensive behavior. Neuropsychologia 44:2621–2635

    Article  PubMed  Google Scholar 

  24. Graziano MS, Yap GS, Gross CG (1994) Coding of visual space by premotor neurons. Science 266:1054–1057

    Article  CAS  PubMed  Google Scholar 

  25. Graziano MS, Hu XT, Gross CG (1997) Visuospatial properties of ventral premotor cortex. J Neurophysiol 77:2268–2292

    CAS  PubMed  Google Scholar 

  26. Hihara S, Notoya T, Tanaka M et al (2006) Extension of corticocortical afferents into the anterior bank of the intraparietal sulcus by tool-use training in adult monkeys. Neuropsychologia 44:2636–2646

    Article  PubMed  Google Scholar 

  27. Hillis AE, Chang S, Heidler-Gary J et al (2006) Neural correlates of modality-specific spatial extinction. J Cogn Neurosci 18:1889–1898

    Article  PubMed  Google Scholar 

  28. Huerta MF, Harting JK (1984) The mammalian superior colliculus: studies of its morphology and connections. In: Vanegas H (ed) Comparative neurology of the optic tectum. Plenum, New York

    Google Scholar 

  29. Iriki A, Tanaka M, Iwamura Y (1996) Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7:2325–2330

    Article  CAS  PubMed  Google Scholar 

  30. Ishibashi H, Hihara S, Takahashi M et al (2002) Tool-use learning selectively induces expression of brain-derived neurotrophic factor, its receptor trkB, and neurotrophin 3 in the intraparietal mutlisensorycortex of monkeys. Brain Res Cogn Brain Res 14:3–9

    Article  CAS  PubMed  Google Scholar 

  31. Jiang W, Stein BE (2003) Cortex controls multisensory depression in superior colliculus. J Neurophysiol 90:2123–2135

    Article  PubMed  Google Scholar 

  32. Jiang W, Wallace MT, Jiang H et al (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522

    CAS  PubMed  Google Scholar 

  33. Kadunce DC, Vaughan JW, Wallace MT et al (1997) Mechanisms of within- and cross-modality suppression in the superior colliculus. J Neurophysiol 78:2834–2847

    CAS  PubMed  Google Scholar 

  34. Kadunce DC, Vaughan JW, Wallace MT et al (2001) The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp Brain Res 139:303–310

    Article  CAS  PubMed  Google Scholar 

  35. Kennett S, Taylor-Clarke M, Haggard P (2001) Noninformative vision improves the spatial resolution of touch in humans. Curr Biol 11:1188–1191

    Article  CAS  PubMed  Google Scholar 

  36. Làdavas E (2002) Functional and dynamic properties of visual peripersonal space. Trends Cogn Sci 6:17–22

    Article  Google Scholar 

  37. Làdavas E (2008) Multisensory-based approach to the recovery of unisensory deficit. Ann N Y Acad Sci 1124:98–110

    Article  PubMed  Google Scholar 

  38. Làdavas E, Farnè A (2004) Visuo-tactile representation of near-the-body space. J Physiol Paris 98:161–170

    Article  PubMed  Google Scholar 

  39. Làdavas E, di Pellegrino G, Farnè A et al (1998) Neuropsychological evidence of an integrated visuotactile representation of peripersonal space in humans. J Cogn Neurosci 10:581–589

    Article  PubMed  Google Scholar 

  40. Làdavas E, Farnè A, Zeloni G et al (2000) Seeing or not seeing where your hands are. Exp Brain Res 131:458–467

    Article  PubMed  Google Scholar 

  41. Legrand D, Brozzoli C, Rossetti Y et al (2007) Close to me: multisensory space representations for action and pre-reflexive consciousness of oneself-in-the-world. Conscious Cogn 16:687–699

    Article  PubMed  Google Scholar 

  42. Macaluso E, Frith CD, Driver J (2000) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208

    Article  CAS  PubMed  Google Scholar 

  43. Magosso E, Cuppini C, Serino A et al (2008) A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw 21:817–829

    Article  PubMed  Google Scholar 

  44. Magosso E, Ursino M, di Pellegrino G et al (2010a) Neural bases of peri-hand space plasticity through tool-use: insights from a combined computational-experimental approach. Neuropsychologia 48:812–830

    Article  PubMed  Google Scholar 

  45. Magosso E, Zavaglia M, Serino A et al (2010b) Visuotactile representation of peripersonal space: a neural network study. Neural Comput 22:190–243

    Article  PubMed  Google Scholar 

  46. Makin TR, Holmes NP, Zohary E (2007) Is that near my hand? Multisensory representation of peripersonal space in human intraparietal sulcus. J Neurosci 27:731–740

    Article  CAS  PubMed  Google Scholar 

  47. Maravita A, Husain M, Clarke K et al (2001) Reaching with a tool extends visual-tactile interactions into far space: evidence from cross-modal extinction. Neuropsychologia 39:580–585

    Article  CAS  PubMed  Google Scholar 

  48. Mattingley JB, Driver J, Beschin N et al (1997) Attentional competition between modalities: extinction between touch and vision after right hemisphere damage. Neuropsychologia 35: 867–880

    Article  CAS  PubMed  Google Scholar 

  49. Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857

    CAS  PubMed  Google Scholar 

  50. Patton PE, Anastasio TJ (2003) Modelling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput 15:783–810

    Article  PubMed  Google Scholar 

  51. Patton PE, Belkacem-Boussaid K, Anastasio TJ (2002) Multimodality in the superior colliculus: an information theoretic analysis. Brain Res Cogn Brain Res 14:10–19

    Article  PubMed  Google Scholar 

  52. Perrault TJ Jr, Vaughan JW, Stein BE et al (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586

    Article  PubMed  Google Scholar 

  53. Pouget A, Sejnowski JT (1997) Spatial transformations in the parietal cortex using basis functions. J Cogn Neurosci 9:222–237

    Article  Google Scholar 

  54. Pouget A, Deneve S, Duhamel JR (2002) A computational perspective on the neural basis of multisensory spatial representations. Nat Rev Neurosci 3:741–747

    Article  CAS  PubMed  Google Scholar 

  55. Press C, Taylor-Clarke M, Kennett S et al (2004) Visual enhancement of touch in spatial body representation. Exp Brain Res 154:238–245

    Article  PubMed  Google Scholar 

  56. Rizzolatti G, Scandolara C, Matelli M et al (1981) Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses. Behav Brain Res 2:147–163

    CAS  Google Scholar 

  57. Rizzolatti G, Fadiga L, Fogassi L et al (1997) The space around us. Science 277:190–191

    Article  CAS  PubMed  Google Scholar 

  58. Rizzolatti G, Luppino G, Matelli M (1998) The organization of the cortical motor system: new concepts. Electroencephalogr Clin Neurophysiol 106:283–296

    Article  CAS  PubMed  Google Scholar 

  59. Salinas E, Abbott LF (1995) Transfer of coded information from sensory to motor networks. J Neurosci 15:6461–6474

    CAS  PubMed  Google Scholar 

  60. Sarri M, Blankenburg F, Driver J (2006) Neural correlates of crossmodal visual-tactile extinction and of tactile awareness revealed by fMRI in a right-hemisphere stroke patient. Neuropsychologia 44:2398–2410

    Article  PubMed  Google Scholar 

  61. Schroeder CE, Foxe J (2005) Multisensory contributions to low-level, ‘unisensory’ processing. Curr Opin Neurobiol 15:454–458

    Article  CAS  PubMed  Google Scholar 

  62. Sereno MI, Huang RS (2006) A human parietal face area contains aligned head-centered visual and tactile maps. Nat Neurosci 9:1337–1343

    Article  CAS  PubMed  Google Scholar 

  63. Stein BE (2005) The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp Brain Res 166:305–315

    Article  PubMed  Google Scholar 

  64. Stein BE, Meredith MA (1993) The merging of the senses. MIT, Cambridge

    Google Scholar 

  65. Taylor-Clarke M, Kennett S, Haggard P (2002) Vision modulates somatosensory cortical processing. Curr Biol 12:233–236

    Article  CAS  PubMed  Google Scholar 

  66. Tipper SP, Lloyd D, Shorland B et al (1998) Vision influences tactile perception without proprioceptive orienting. Neuroreport 9:1741–1744

    Article  CAS  PubMed  Google Scholar 

  67. Ursino M, Cuppini C, Magosso E et al (2009) Multisensory integration in the superior colliculus: a neural network model. J Comput Neurosci 26:55–73

    Article  PubMed  Google Scholar 

  68. Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol 69: 1797–1809

    CAS  PubMed  Google Scholar 

  69. Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the Superior Colliculus of the alert cat. J Neurophysiol 80:1006–1010

    CAS  PubMed  Google Scholar 

  70. Woods TM, Recanzone GH (2004) Visually induced plasticity of auditory spatial perception in macaques. Curr Biol 14:1559–1564

    Article  CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics, Computer Science and Systems, University of Bologna, Viale Risorgimento 2, I-40136, Bologna, Italy

    Mauro Ursino

Authors
  1. Mauro Ursino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elisa Magosso
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cristiano Cuppini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mauro Ursino .

Editor information

Editors and Affiliations

  1. Department of Psychology, Boston University, Boston, 02215, USA

    Vassilis Cutsuridis

  2. Dept. Computing Science, University of Stirling, Stirling, FK9 4LA, United Kingdom

    Amir Hussain

  3. King's College London, Dept. Mathematics, University of London, London, WC2R 2LS, United Kingdom

    John G. Taylor

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Ursino, M., Magosso, E., Cuppini, C. (2011). Sensory Fusion. In: Cutsuridis, V., Hussain, A., Taylor, J. (eds) Perception-Action Cycle. Springer Series in Cognitive and Neural Systems. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-1452-1_2

Download citation

Publish with us

Policies and ethics

Search

Navigation