Skip to main content
SpringerLink
Log in

The role of mismatches in the sensory feedback provided to indicate selection within a virtual environment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

It is generally understood that virtual reality simulations have a high computational cost. Hence, they rarely can reduce completely all the incoherence within the cross-modal sensory outputs provided. The main research approaches to date have consisted in technically reducing possible mismatches, however minimal research has been conducted so as to analyse their influence on human capabilities. Thus, the objective of this study is to provide further insights to the designers of virtual reality about the negative influence of simulation lags and interesting design implications. To clearly show this, we have investigated the importance of coherent sensory feedback by incorporating time delays and spatial misalignments in the feedback provided by the simulation as a response to participant´s actions to mimic computationally expensive environments. We have also evaluated these misalignments considering two typical interaction setups. In particular, the sensory mismatches influence has been assessed in human factors, such as the sense of presence, task performance and delay perception. Our experimental results indicate that the closer the interaction conditions are to real configurations the higher the sensory requirements are regarding accuracy. The implications of this study offer the designer guidelines to prioritise the reduction of those mismatches in the sensory cues provided depending on the simulations goals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Note that average time elapsed between buttons pressings without any mismatch was around 800 ms and the average displacement over the Simon surface was around 10 cm. Thus, assuming a constant speed of 12.5 cm/s a spatial lag of 0.25 cm and 0.75 cm correspond with a time delay of around 20 ms and 60 ms respectively.

References

  1. Allison RS, Harris LR, Jenkin M, Jasiobedzka U, Zacher JE (2001) Tolerance of temporal delay in virtual environments. In Proc. of IEEE Virtual Reality, pp 247–254. doi:http://doi.ieeecomputersociety.org/10.1109/VR.2001.913793

  2. Ardito C, Costabile MF, Angeli A, Pittarello F (2007) Navigation help in 3D worlds: some empirical evidences on use of sound. Multimed Tools Appl 33:201–216. doi:10.1007/s11042-006-0060-0

    Article  Google Scholar 

  3. Arsenault R, Ware C (2000) Eye-hand co-ordination with force feedback. In Proc. of the SIGCHI conf. on Human factors in computing systems, pp 408–414

  4. Bailenson JN, Yee N (2008) Virtual interpersonal touch: haptic interaction and copresence in collaborative virtual environments. Multimed Tools Appl 37:5–14. doi:10.1007/s11042-007-0171-2

    Article  Google Scholar 

  5. Begault DR (2000) 3-D sound for virtual reality and multimedia. Facsimile reprint of 1994 edition, Preface to the NASA Technical Memorandum version, Ames Research Center, Moffett Field, CA, USA

  6. Boring EG (1929) A history of experimental psychology. Appleton Century Crofts, New York

    Google Scholar 

  7. Boukerche A, Shirmohammadi S, Hossain A (2006) Moderating simulation lag in haptic virtual environments. In Proc. of the 39th Annual Symposium on Simulation, pp 269–277, IEEE Computer Society, Washington, USA. doi:10.1109/ANSS.2006.31

  8. Burke JL, Prewett MS, Gray AA, Yang L, Stilson, FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In Proc. Multimodal interfaces, pp 108–117. ACM, NY, USA

  9. Burns E, Razzaque S, Whitton M, McCallus M, Panter A, Brooks FP (2005) The hand is slower than the eye: a quantitative exploration of visual dominance over proprioception. In Proc. of IEEE Virtual Reality, pp 3–10

  10. Buttazzo G, Lipari G, Abeni L, Caccamo M (2005) Soft real-time systems: predictability vs. efficiency (Series in Computer Science). Springer, Berlin

    MATH  Google Scholar 

  11. Congedo M, Lécuyer A, Gentaz E (2006) The influence of spatial delocation on perceptual integration of vision and touch. Presence-Teleoper Virtual Env 15:353–357

    Article  Google Scholar 

  12. De Boeck J, Raymaekers C, Coninx K (2005) Are existing metaphors in virtual environments suitable for haptic interaction. In Proc. of Virtual Reality, pp 261–268. Laval, France

  13. Durlach PJ, Fowlkes J, Metevier CJ (2005) Effect of variations in sensory feedback on performance in a virtual reaching task. Presence-Teleoper Virtual Env 14:450–462

    Article  Google Scholar 

  14. Ehmann S, Lin MC (2001) Accurate and fast proximity queries between polyhedral using convex surface decomposition. In Proc. of Eurographics, 20, pp 500–510

  15. Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7:773–778

    Article  Google Scholar 

  16. Harris CS (1965) Perceptual adaptation to inverted, reversed, and displaced vision. Psychol Rev 72:419–444

    Article  Google Scholar 

  17. Hirsh IJ, Sherrick CE (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432

    Article  Google Scholar 

  18. Ijsselsteijn WA, deKort YAW, Haans A (2006) Is this my hand I see before me? The rubber hand illusion in reality, virtual reality, and mixed reality. Presence-Teleoper Virtual Env 15:455–464. doi:10.1162/pres.15.4.455

    Article  Google Scholar 

  19. Jay C, Hubbold R (2005) Delayed visual and haptic feedback in a reciprocal tapping task. In Proc. of World Haptics, pp 655–656. doi:http://doi.ieeecomputersociety.org/10.1109/WHC.2005.29

  20. Jay C, Hubbold R (2006) Quantifying the effects of latency on sensory feedback in distributed virtual environments. In Proc. of Virtual Images Seminar, pp 9–16. Paris, France

  21. Keetels M, Vroomen J (2007) No effect of auditory-visual spatial disparity on temporal recalibration. Exp Brain Res 182:559–565. doi:10.1007/s00221-007-1012-2

    Article  Google Scholar 

  22. Larsson P, Västfjäll D, Kleiner, M (2001) Ecological acoustics and the multi-modal perception of rooms: real and unreal experiences of auditory-visual virtual environments. In Proc. of the Conf. on Auditory Display, pp 245–249, Espoo, Finland

  23. Lécuyer A, Burkhardt JM, Coquillart S, Coiffet P (2001) “Boundary of illusion”: an experiment of sensory integration with a pseudo-haptic system. In Proc. of IEEE Virtual Reality, pp 115–122

  24. Lee I, Choi S (2007) Discrimination of virtual environments under visual and haptic rendering delays, frontiers in the convergence of bioscience and information technologies, pp 554–562. doi:http://doi.ieeecomputersociety.org/10.1109/FBIT.2007.124

  25. Levitin DJ, MacLean K, Mathews M, Chu L, Jensen E (2000) The perception of cross-modal simultaneity. In AIP Conf. Proc. 517, pp 323, doi:10.1063/1.1291270

  26. Loomis JM (1992) Distal attribution and presence. Presence-Teleoper Virtual Env 1:113–119

    Google Scholar 

  27. MacKenzie IS, Ware C (1993) Lag as a determinant of human performance in interactive systems. In Proc. of the INTERACT ’93 and CHI ’93, pp 488–493, NY, USA. doi:http://doi.acm.org/10.1145/169059.169431

  28. Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17:154–163. doi:10.1016/S0926-6410(03)00089-2

    Article  Google Scholar 

  29. Munhall KG, Gribble P, Sacco L, Ward M (1996) Temporal constraints on the McGurk effect. Percept Psychophys 58:351–362

    Article  Google Scholar 

  30. Navarra J, Soto-Faraco S, Spence C (2007) Adaptation to audiotactile asynchrony. Neurosci Lett 413:72–76. doi:10.1016/j.neulet.2006.11.027

    Article  Google Scholar 

  31. Pellegrini RS (2001) Quality assessment of auditory virtual environments. In Proceedings of ICAD, Espoo, Finland

  32. Radeau M, Bertelson P (1977) Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Percept Psychophys 22:137–146

    Article  Google Scholar 

  33. Rossetti Y, Koga K, Mano T (1993) Prismatic displacement of vision induces transient changes in the timing of eye-hand coordination. Percept Psychophys 54:355–364

    Article  Google Scholar 

  34. Rothrock L, Barron K, Simpson T, Frecker M, Ligetti C, Barton R (2006) Applying the proximity compatibility and the control-display compatibility principles to engineering design interfaces. Hum Factors Ergon Manuf 16:61–81. doi:10.1002/hfm.v16:1

    Article  Google Scholar 

  35. Ryu S, Kim H, Park J, Kwon Y, Jeong C (2007) Collaborative object-oriented visualization environment. Multimed Tools Appl 3:209–234. doi:10.1007/s11042-006-0066-7

    Article  Google Scholar 

  36. Simpson TW, Barron K, Rothrock L, Frecker M, Barton RR, Ligetti C (2007) Impact of response delay and training on user performance with text-based and graphical user interfaces for engineering design. Res Eng Des 18:49–65. doi:10.1007/s00163-007-0033-y

    Article  Google Scholar 

  37. Slater M, Usoh M, Steed A (1994) Depth of presence in virtual environments. Presence-Teleoper Virtual Env 3:113–119

    Google Scholar 

  38. Sousa Santos B, Dias P, Pimentel A, Baggerman J, Ferreira C, Silva S, Madeira J (2009) Head-mounted display versus desktop for 3D navigation in virtual reality: a user study. Multimed Tools Appl 4:161–181. doi:10.1007/s11042-008-0223-2

    Article  Google Scholar 

  39. Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol Gen 130:799–832

    Article  Google Scholar 

  40. Swapp D, Pawar V, Loscos C (2006) Interaction with co-located haptic feedback in virtual reality. Virtual Real 10:24–30. doi:10.1007/s10055-006-0027-5

    Article  Google Scholar 

  41. Szameitat AJ, Rummel J, Szameitat DP, Sterr A (2009) Behavioral and emotional consequences of brief delays in human–computer interaction. Int J Hum Comput Stud 67:561–570. doi:10.1016/j.ijhcs.2009.02.004

    Article  Google Scholar 

  42. Theile G (1993) Trends and activities in the development of multichannel sound systems. In Proc. of AES International Conf., pp 180–187

  43. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111:134–142. doi:10.1016/j.brainres.2006.05.078

    Article  Google Scholar 

  44. Viciana-Abad R, Reyes-Lecuona A (2008) Effects of Co-location and crossmodal interaction between haptic, auditory and visual cues in presence. Haptics: Perception, Devices and Scenarios, LNCS 5024: 832–837, Berlin: Springer. doi:10.1007/978-3-540-69057-3

  45. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 2:32–35. doi:10.1016/j.cogbrainres.2004.07.003

    Article  Google Scholar 

  46. Watson B, Walker N, Ribarsky B, Spaulding V (1999) Managing temporal detail in virtual environments: relating system responsiveness to feedback. In Proc. of CHI ’99, pp 280–281, NY, USA. doi:http://doi.acm.org/10.1145/632716.632888

  47. Watson B, Walker N, Woytiuk P, Ribarsky W (2003) Maintaining usability during 3D placement despite delay. In Proc. of IEEE Virtual Reality Conf., pp 133–140. doi:http://doi.ieeecomputersociety.org/10.1109/VR.2003.1191131

  48. Witmer BG, Jerome CJ, Singer MJ (2005) The factor structure of the presence questionnaire. Presence-Teleoper Virtual Env 14:298–331. doi:10.1162/105474605323384654

    Article  Google Scholar 

  49. Woszczyk W (1993) Quality assessment of multichannel sound recordings. In Proc. of AES Int. Conf., pp 197–218

  50. Yang U, Jounghyun KG (2004) Increasing the effective egocentric field of view with proprioceptive and tactile feedback. In Proc. of IEEE Virtual Reality, pp 27–34

Download references

Acknowledgements

The authors wish to thank the participants in the study for their collaboration and comments. This work has been partially supported by DIANA group (University of Málaga) and by the University of Jaén through project UJA2009/12/12.

Author information

Authors and Affiliations

  1. Department of Telecommunication Engineering, University of Jaén, Jaén, Spain

    Raquel Viciana-Abad & José Escolano

  2. Department of Electronic Technology, University of Málaga, Málaga, Spain

    Arcadio Reyes-Lecuona & Matthieu Poyade

Authors
  1. Raquel Viciana-Abad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arcadio Reyes-Lecuona
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthieu Poyade
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José Escolano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raquel Viciana-Abad.

Appendix I. Presence Items

Appendix I. Presence Items

figure e

Rights and permissions

Reprints and permissions

About this article

Cite this article

Viciana-Abad, R., Reyes-Lecuona, A., Poyade, M. et al. The role of mismatches in the sensory feedback provided to indicate selection within a virtual environment. Multimed Tools Appl 55, 353–378 (2011). https://doi.org/10.1007/s11042-010-0551-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0551-x

Keywords

Search

Navigation