Skip to main content
Log in

Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Concurrent augmented feedback has been shown to be less effective for learning simple motor tasks than for complex tasks. However, as mostly artificial tasks have been investigated, transfer of results to tasks in sports and rehabilitation remains unknown. Therefore, in this study, the effect of different concurrent feedback was evaluated in trunk-arm rowing. It was then investigated whether multimodal audiovisual and visuohaptic feedback are more effective for learning than visual feedback only. Naïve subjects (N = 24) trained in three groups on a highly realistic virtual reality-based rowing simulator. In the visual feedback group, the subject’s oar was superimposed to the target oar, which continuously became more transparent when the deviation between the oars decreased. Moreover, a trace of the subject’s trajectory emerged if deviations exceeded a threshold. The audiovisual feedback group trained with oar movement sonification in addition to visual feedback to facilitate learning of the velocity profile. In the visuohaptic group, the oar movement was inhibited by path deviation-dependent braking forces to enhance learning of spatial aspects. All groups significantly decreased the spatial error (tendency in visual group) and velocity error from baseline to the retention tests. Audiovisual feedback fostered learning of the velocity profile significantly more than visuohaptic feedback. The study revealed that well-designed concurrent feedback fosters complex task learning, especially if the advantages of different modalities are exploited. Further studies should analyze the impact of within-feedback design parameters and the transferability of the results to other tasks in sports and rehabilitation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262

    Article  CAS  PubMed  Google Scholar 

  • Blandin Y, Toussaint L, Shea CH (2008) Specificity of practice: interaction between concurrent sensory information and terminal feedback. J Exp Psychol Learn Mem Cogn 34(4):994–1000

    Article  PubMed  Google Scholar 

  • Braun DA, Mehring C, Wolpert DM (2010) Structure learning in action. Behav Brain Res 206(2):157–165

    Article  PubMed Central  PubMed  Google Scholar 

  • Brown R, Palmer C (2012) Auditory–motor learning influences auditory memory for music. Mem Cognit 40(4):567–578

    Article  PubMed  Google Scholar 

  • Buchanan J, Wang C (2012) Overcoming the guidance effect in motor skill learning: feedback all the time can be beneficial. Exp Brain Res 219(2):305–320

    Article  PubMed  Google Scholar 

  • Burke JL, Prewett MS, Gray AA, Yang L, Stilson FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In: Proceedings of the 8th international conference on multimodal interfaces, New York, NY, USA, pp 108–117

  • Carson RG, Kelso JAS (2004) Governing coordination: behavioural principles and neural correlates. Exp Brain Res 154(3):267–274

    Article  CAS  PubMed  Google Scholar 

  • Cesqui B, Aliboni S, Mazzoleni S, Carrozza M, Posteraro F, Micera S (2008) On the use of divergent force fields in robot-mediated neurorehabilitation. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 854–861

  • Chen X, Agrawal S (2013) Assisting versus repelling force-feedback for learning of a line following task in a wheelchair. IEEE Trans Neural Syst Rehabil Eng 21(6):959–968

    Article  PubMed  Google Scholar 

  • Chen JL, Penhune VB, Zatorre RJ (2008) Listening to musical rhythms recruits motor regions of the brain. Cereb Cortex 18(12):2844–2854

    Article  PubMed  Google Scholar 

  • Chiviacowsky S, Wulf G (2007) Feedback after good trials enhances learning. Res Q Exerc Sport 78:40–47

    Article  PubMed  Google Scholar 

  • Chollet D, Madani M, Micallef JP (1992) Effects of two types of biomechanical bio-feedback on crawl performance. In: MacLaren D, Reilly T, Lees A (eds) Biomechanics and medicine in swimming. E & FN Spon, London, pp 57–62

    Google Scholar 

  • Effenberg AO (2005) Movement sonification: effects on perception and action. IEEE Multimedia 12(2):53–59

    Article  Google Scholar 

  • Emken J, Reinkensmeyer DJ (2005) Robot-enhanced motor learning: accelerating internal model formation during locomotion by transient dynamic amplification. IEEE Trans Neural Syst Rehabil Eng 13(1):33–39

    Article  PubMed  Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433

    Article  CAS  PubMed  Google Scholar 

  • Freides D (1974) Human information processing and sensory modality: cross-modal functions information complexity memory and deficit. Psychol Bull 81(5):284–310

    Article  CAS  PubMed  Google Scholar 

  • Giese MA, Poggio T (2000) Morphable models for the analysis and synthesis of complex motion patterns. Int J Comput Vis 38(1):59–73

    Article  Google Scholar 

  • Guadagnoli M, Kohl R (2001) Knowledge of results for motor learning: relationship between error estimation and knowledge of results frequency. J Mot Behav 33(2):217–224

    Article  CAS  PubMed  Google Scholar 

  • Guadagnoli MA, Lee TD (2004) Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J Mot Behav 36(2):212–224

    Article  PubMed  Google Scholar 

  • Hale K, Stanney K (2004) Deriving haptic design guidelines from human physiological psychophysical and neurological foundations. IEEE Comput Graph Appl 24(2):33–39

    Article  PubMed  Google Scholar 

  • Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8(3):187–211

    Article  PubMed  Google Scholar 

  • Huang F, Patton J (2013) Augmented dynamics and motor exploration as training for stroke. IEEE Trans Biomed Eng 60(3):838–844

    Article  PubMed  Google Scholar 

  • Huang H, Ingalls T, Olson L, Ganley K, Rikakis T, He J (2005) Interactive multimodal biofeedback for task-oriented neural rehabilitation. In: 27th annual international conference of the engineering in medicine and biology society 2005 IEEE-EMBS 2005, Shanghai, pp 2547–2550

  • Hubbard T (2013) Auditory imagery contains more than audition. In: Lacey S, Lawson R (eds) Multisensory imagery. Springer, New York, pp 221–247

    Chapter  Google Scholar 

  • Huegel J, O’Malley MK (2010) Progressive haptic and visual guidance for training in a virtual dynamic task. In: haptics symposium 2010 IEEE, pp 343–350

  • Israel J, Campbell D, Kahn J, Hornby T (2006) Metabolic costs and muscle activity patterns during robotic-and therapist-assisted treadmill walking in individuals with incomplete spinal cord injury. Phys Ther 86(11):1466–1478

    Article  PubMed  Google Scholar 

  • Kapur A, Tzanetakis G, Virji-Babul N, Wang G, Cook PR (2005) A framework for sonification of vicon motion capture data. In: Proceedings of the 8th conference on digital audio effects, Madrid, Spain

  • Kennedy D, Boyle J, Shea C (2013) The role of auditory and visual models in the production of bimanual tapping patterns. Exp Brain Res 224(4):507–518

    Article  PubMed  Google Scholar 

  • Keysers C, Gazzola V (2010) Social neuroscience: mirror neurons recorded in humans. Curr Biol 20(8):R353–R354

    Article  CAS  PubMed  Google Scholar 

  • Kim RS, Seitz AR, Shams L (2008) Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1):e1532

    Article  PubMed Central  PubMed  Google Scholar 

  • Kovacs AJ, Shea CH (2011) The learning of 90° continuous relative phase with and without lissajous feedback: external and internally generated bimanual coordination. Acta Psychol 136(3):311–320

    Article  Google Scholar 

  • Krakauer J, Mazzoni P (2011) Human sensorimotor learning: adaptation skill and beyond. Curr Opin Neurobiol 21(4):636–644

    Article  CAS  PubMed  Google Scholar 

  • Kramer G (1994) Auditory display: sonification audification and auditory interfaces. Addison-Wesley, Reading MA

    Google Scholar 

  • Krebs HI, Palazzolo JJ, Dipietro L, Ferraro M, Krol J, Rannekleiv K, Volpe BT, Hogan N (2003) Rehabilitation robotics: performance-based progressive robot-assisted therapy. Auton Robots 15(1):7–20

    Article  Google Scholar 

  • Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27(2):308–314

    Article  CAS  PubMed  Google Scholar 

  • Liebermann DG, Katz L, Hughes MD, Bartlett RM, McClements J, Franks IM (2002) Advances in the application of information technology to sport performance. J Sports Sci 20(10):755–769

    Article  PubMed  Google Scholar 

  • Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27(35):9354–9368

    Article  CAS  PubMed  Google Scholar 

  • Liu J, Wrisberg CA (1997) The effect of knowledge of results delay and the subjective estimation of movement form on the acquisition and retention of a motor skill. Res Q Exerc Sport 68(2):145–151

    Article  CAS  PubMed  Google Scholar 

  • Lüttgen J, Heuer H (2012a) The influence of haptic guidance on the production of spatio-temporal patterns. Hum Mov Sci 31(3):519–528

    Article  PubMed  Google Scholar 

  • Lüttgen J, Heuer H (2012b) Robotic guidance benefits the learning of dynamic but not of spatial movement characteristics. Exp Brain Res 222(1–2):1–9

    Article  PubMed  Google Scholar 

  • Lüttgen J, Heuer H (2013) The influence of robotic guidance on different types of motor timing. J Mot Behav 45(3):249–258

    Article  PubMed  Google Scholar 

  • Marchal-Crespo L, Reinkensmeyer DJ (2008a) Effect of robotic guidance on motor learning of a timing task. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 199–204

  • Marchal-Crespo L, Reinkensmeyer DJ (2008b) Haptic guidance can enhance motor learning of a steering task. J Mot Behav 40(6):545–557

    Article  PubMed  Google Scholar 

  • Marchal-Crespo L, Reinkensmeyer DJ (2009) Review of control strategies for robotic movement training after neurologic injury. J Neuroeng Rehabil 6(1):20

    Article  PubMed Central  PubMed  Google Scholar 

  • Marchal-Crespo L, Furumasu J, Reinkensmeyer DJ (2010) A robotic wheelchair trainer: design overview and a feasibility study. J Neuroeng Rehabil 7(1):40–51

    Article  PubMed Central  PubMed  Google Scholar 

  • Marchal-Crespo L, Raai M, Rauter G, Wolf P, Riener R (2013) The effect of haptic guidance and visual feedback on learning a complex tennis task. Exp Brain Res 231(3):277–291

    Article  PubMed  Google Scholar 

  • Marschall F, Bund A, Wiemeyer J (2007) Does frequent feedback really degrade learning? A meta analysis. E-Journal Bewegung und Training 1:75–86

    Google Scholar 

  • Milot MH, Marchal-Crespo L, Green CS, Cramer SC, Reinkensmeyer DJ (2010) Comparison of error-amplification and haptic-guidance training techniques for learning of a timing-based motor task by healthy individuals. Exp Brain Res 201(2):119–131

    Article  PubMed  Google Scholar 

  • Minogue J, Jones MG (2006) Haptics in Education: exploring an untapped sensory modality. Rev Educ Res 76(3):3–17

    Article  Google Scholar 

  • Patton JL, Stoykov M, Kovic M, Mussa-Ivaldi F (2006) Evaluation of robotic training forces that either enhance or reduce error in chronic hemiparetic stroke survivors. Exp Brain Res 168(3):368–383

    Article  PubMed  Google Scholar 

  • Patton JL, Wei YJ, Bajaj P, Scheidt RA (2013) Visuomotor learning enhanced by augmenting instantaneous trajectory error feedback during reaching. PLoS One 8(1):e46466

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Proteau L (1992) On the specificity of learning and the role of visual information for movement control. In: Proteau L, Elliott D (eds) Vision and motor control, vol 85. North-Holland, Amsterdam, pp 67–103

    Chapter  Google Scholar 

  • Rauter G, von Zitzewitz J, Duschau-Wicke A, Vallery H, Riener R (2010) A tendon based parallel robot applied to motor learning in sports. In: 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob) 2010, Tokyo, Japan, pp 82–87

  • Rauter G, Sigrist R, Marchal-Crespo L, Vallery H, Riener R, Wolf P (2011) Assistance or challenge? Filling a gap in user-cooperative control. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3068–3073

  • Rauter G, Sigrist R, Koch C, Crivelli F, van Raai M, Riener R, Wolf P (2013) Transfer of complex skill learning from virtual to real rowing. PLoS One 8(12):1–18

    Article  Google Scholar 

  • Reinkensmeyer DJ, Akoner O, Ferris D, Gordon K (2009) Slacking by the human motor system: computational models and implications for robotic orthoses. In: Engineering in medicine and biology society 2009 EMBC 2009 annual international conference of the IEEE, pp 2129–2132

  • Ribeiro DC, Sole G, Abbott JH, Milosavljevic S (2011) Extrinsic feedback and management of low back pain: a critical review of the literature. Man Ther 16(3):231–239

    Article  PubMed  Google Scholar 

  • Robin C, Toussaint L, Blandin Y, Proteau L (2005) Specificity of learning in a video-aiming task: modifying the salience of dynamic visual cues. J Mot Behav 37(5):367–376

    Article  PubMed  Google Scholar 

  • Ronsse R, Puttemans V, Coxon JP, Goble DJ, Wagemans J, Wenderoth N, Swinnen SP (2011) Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cereb Cortex 21(6):1283–1294

    Article  PubMed  Google Scholar 

  • Salmoni S (1984) Knowledge of results and motor learning A review and critical reappraisal. Psychol Bull 95(3):355–386

    Article  CAS  PubMed  Google Scholar 

  • Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6(2):392–405

    Article  Google Scholar 

  • Schmidt RA (1991) Frequent augmented feedback can degrade learning: evidence and interpretations. Tutor Motor Neurosci 62:59–75

    Article  Google Scholar 

  • Schmidt RA, Wrisberg C (2008) Motor learning and performance: a situation-based learning approach. Human Kinetics, Champaign, IL

    Google Scholar 

  • Schmidt RA, Wulf G (1997) Continuous concurrent feedback degrades skill learning: implications for training and simulation. Hum Factors 39(4):509–525

    Article  CAS  PubMed  Google Scholar 

  • Schmidt RA, Young DE, Swinnen S, Shapiro DC (1989) Summary knowledge of results for skill acquisition: support for the guidance hypothesis. J Exp Psychol Learn Mem Cogn 15(2):352–359

    Article  CAS  PubMed  Google Scholar 

  • Schmitz G, Mohammadi B, Hammer A, Heldmann M, Samii A, Munte T, Effenberg A (2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci 14(1):1–11

    Article  Google Scholar 

  • Secoli R, Milot M, Rosati G, Reinkensmeyer DJ (2011) Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke. J Neuroeng Rehabil 8(1):1–10

    Article  Google Scholar 

  • Seitz AR, Dinse HR (2007) A common framework for perceptual learning. Curr Opin Neurobiol 17(2):148–153

    Article  CAS  PubMed  Google Scholar 

  • Seitz AR, Kim R, Shams R (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427

    Article  CAS  PubMed  Google Scholar 

  • Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417

    Article  PubMed  Google Scholar 

  • Sigrist R, Schellenberg J, Rauter G, Broggi S, Riener R, Wolf P (2011) Visual and auditory augmented concurrent feedback in a complex motor task. Presence Teleop Virt 20(1):15–32

    Article  Google Scholar 

  • Sigrist R, Rauter G, Riener R, Wolf P (2013a) Augmented visual auditory haptic and multimodal feedback in motor learning: a review. Psychon Bull Rev 20:21–53

    Article  PubMed  Google Scholar 

  • Sigrist R, Rauter G, Riener R, Wolf P (2013b) Terminal feedback outperforms concurrent visual auditory and haptic feedback in learning a complex rowing-type task. J Mot Behav 45(6):455–472

    Article  PubMed  Google Scholar 

  • Snodgrass SJ, Rivett DA, Robertson VJ, Stojanovski E (2010) Real-time feedback improves accuracy of manually applied forces during cervical spine mobilization. Man Ther 15:19–25

    Article  PubMed  Google Scholar 

  • Swinnen SP, Schmidt RA, Nicholson DE, Shapiro DC (1990) Information feedback for skill acquisition: instantaneous knowledge of results degrades learning. J Exp Psychol Learn Mem Cogn 16(4):706–716

    Article  Google Scholar 

  • Swinnen SP, Lee TD, Verschueren S, Serrien DJ, Bogaerds H (1997) Interlimb coordination: learning and transfer under different feedback conditions. Hum Mov Sci 16(6):749–785

    Article  Google Scholar 

  • Thoroughman K, Shadmehr R (2000) Learning of action through adaptive combination of motor primitives. Nature 407(6805):742

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Timmermans AAA, Seelen HAM, Willmann RD, Kingma H (2009) Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroeng Rehabil 6:1

    Article  PubMed Central  PubMed  Google Scholar 

  • Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7:907–915

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5:1226–1235

    Article  CAS  PubMed  Google Scholar 

  • Todorov E, Shadmehr R, Bizzi E (1997) Augmented feedback presented in a virtual environment accelerates learning of a difficult motor task. J Mot Behav 29(2):147–158

    Article  PubMed  Google Scholar 

  • van Beers RJ (2009) Motor learning is optimally tuned to the properties of motor noise. Neuron 63(3):406–417

    Article  PubMed  Google Scholar 

  • van Beers RJ, Sittig AC, Gon JJ (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81(3):1355

    PubMed  Google Scholar 

  • van der Linden DW, Cauraugh JH, Greene TA (1993) The effect of frequency of kinetic feedback on learning an isometric force production task in nondisabled subjects. Phys Ther 73(2):79–87

    Google Scholar 

  • Vlachos M, Hadjieleftheriou M, Gunopulos D, Keogh E (2003) Indexing multi-dimensional time-series with support for multiple distance measures. In: proceedings of the ninth acm sigkdd international conference on knowledge discovery and data mining KDD’03, New York, NY, USA, pp 216–225

  • von Zitzewitz J, Wolf P, Novakovic V, Wellner M, Rauter G, Brunschweiler A, Riener R (2008) Real-time rowing simulator with multimodal feedback. Sports Technol 1(6):257–266

    Article  Google Scholar 

  • Wei K, Körding K (2009) Relevance of error: what drives motor adaptation? J Neurophysiol 101(2):655–664

    Article  PubMed Central  PubMed  Google Scholar 

  • Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667

    Article  CAS  PubMed  Google Scholar 

  • Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2):159–177

    Article  Google Scholar 

  • Winstein CJ (1991) Knowledge of results and motor learning—implications for physical therapy. Phys Ther 71(2):140–149

    CAS  PubMed  Google Scholar 

  • Winstein CJ, Pohl PS, Cardinale C, Green A, Scholtz L, Waters CS (1996) Learning a partial-weight-bearing skill: effectiveness of two forms of feedback. Phys Ther 76(9):985–993

    CAS  PubMed  Google Scholar 

  • Wishart LR, Lee TD, Cunningham SJ, Murdoch JE (2002) Age-related differences and the role of augmented visual feedback in learning a bimanual coordination pattern. Acta Psychol (Amst) 110(2–3):247–263

    Article  Google Scholar 

  • Wolpert D, Flanagan J (2010) Motor learning. Curr Biol 20(11):R467–R472

    Article  CAS  PubMed  Google Scholar 

  • Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739–749

    CAS  PubMed  Google Scholar 

  • Wulf G (2007) Self-controlled practice enhances motor learning: implications for physiotherapy. Physiotherapy 93(2):96–101

    Article  Google Scholar 

  • Wulf G, Shea CH (2002) Principles derived from the study of simple skills do not generalize to complex skill learning. Psychon Bull Rev 9(2):185–211

    Article  PubMed  Google Scholar 

  • Wulf G, Shea CH, Matschiner S (1998) Frequent feedback enhances complex motor skill learning. J Mot Behav 30(2):180–192

    Article  CAS  PubMed  Google Scholar 

  • Wulf G, Hörger M, Shea CH (1999) Benefits of blocked over serial feedback on complex motor skill learning. J Mot Behav 31(1):95–103

    Article  PubMed  Google Scholar 

  • Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory–motor interactions in music perception and production. Nat Rev Neurosci 8(7):547–558

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

We want to thank Prof. Nicole Wenderoth for her enriching comments. We thank Alessandro Rotta, Pascal Wespe, and Marco Bader for their indispensable technical contributions to the design and setup of the simulator. Moreover, thanks go to Michael Herold-Nadig for his support on technical and safety issues and Mark van Raai for designing the visual feedback and rowing scenario. We want to thank Stefan Seiterle, Luca Lörtscher, Florian Kübler, and Nicolas Gerig for their help in conducting the measurements, and the subjects for participating. We thank Andrew Pennycott for proofreading the manuscript. This work was supported by ETH Zurich, the SNF-Grant “Impact of Different Feedback Modalities on Complex Skill Learning,” CR22I2 135101/1, and the SNF-Grant “Acceleration of complex motor learning by skill level-dependent feedback design and automatic selection”, CR23I2_152817.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Roland Sigrist.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sigrist, R., Rauter, G., Marchal-Crespo, L. et al. Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233, 909–925 (2015). https://doi.org/10.1007/s00221-014-4167-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-014-4167-7

Keywords

Navigation