Skip to main content
Top
Published in:

2024 | OriginalPaper | Chapter

Virtual Reality Hand Tracking for Immersive Telepresence in Rehabilitative Serious Gaming

Authors : Noaman Mazhar, Aya Gaballa, Amit Kumar Pandey, John-John Cabibihan

Published in: Social Robotics

Publisher: Springer Nature Singapore

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Intelligent systems face an increasingly complex array of challenges in the rapidly evolving landscape of human-device interactions. To address these challenges, there arises a pressing need for the development of refined methods of information transfer, capable of capturing subtle nuances such as body language and tonal subtleties. The system presented in this paper is designed to highlight a novel method of refined human-device interactions and its potential impact on the medical domain. Leveraging the Oculus SDK in the Unity Game Engine alongside the Oculus VR headset, our system allows users to interact with virtual objects naturally, mirroring real-world hand movements while simultaneously promoting upper limb rehabilitation through engaging gameplay scenarios. The integration of haptic feedback enriches the immersive experience, enabling users to not only visualize but also feel their virtual interactions. The system accommodates varying levels of mobility, adapting its complexity to individual progress. Furthermore, our VR rehabilitation system has garnered positive outcomes from user assessments, demonstrating the system’s effectiveness and user satisfaction. While limited in addressing severe upper limb impairments, the system’s flexibility allows for modular improvements and broader clinical integration.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Ernst, E., Merola, R., Samaan, D.: Economics of artificial intelligence: implications for the future of work. IZA J. Labor Policy 9(1) (2019) Ernst, E., Merola, R., Samaan, D.: Economics of artificial intelligence: implications for the future of work. IZA J. Labor Policy 9(1) (2019)
2.
go back to reference Damiano, L., Dumouchel, P., Lehmann, H.: Towards human-robot affective co-evolution overcoming oppositions in constructing emotions and empathy. Int. J. Soc. Robot. 7, 7–18 (2015)CrossRef Damiano, L., Dumouchel, P., Lehmann, H.: Towards human-robot affective co-evolution overcoming oppositions in constructing emotions and empathy. Int. J. Soc. Robot. 7, 7–18 (2015)CrossRef
3.
go back to reference Amin, M.M., Cambria, E., Schuller, B.W.: Will affective computing emerge from foundation models and general AI? a first evaluation on ChatGPT. arXiv preprint arXiv:2303.03186 (2023) Amin, M.M., Cambria, E., Schuller, B.W.: Will affective computing emerge from foundation models and general AI? a first evaluation on ChatGPT. arXiv preprint arXiv:​2303.​03186 (2023)
4.
go back to reference Chenna, S.: Augmented reality and AI: enhancing human-computer interaction in the metaverse. Available at SSRN 4324629 (2023) Chenna, S.: Augmented reality and AI: enhancing human-computer interaction in the metaverse. Available at SSRN 4324629 (2023)
5.
go back to reference Bonarini, A.: Communication in human-robot interaction. Curr. Robot. Rep. 1, 279–285 (2020)CrossRef Bonarini, A.: Communication in human-robot interaction. Curr. Robot. Rep. 1, 279–285 (2020)CrossRef
6.
go back to reference Xu, W., Dainoff, M.J., Ge, L., Gao, Z.: Transitioning to human interaction with AI systems: new challenges and opportunities for HCI professionals to enable human-centered AI. Int. J. Hum.-Comput. Interact. 39(3), 494–518 (2023)CrossRef Xu, W., Dainoff, M.J., Ge, L., Gao, Z.: Transitioning to human interaction with AI systems: new challenges and opportunities for HCI professionals to enable human-centered AI. Int. J. Hum.-Comput. Interact. 39(3), 494–518 (2023)CrossRef
7.
go back to reference Urakami, J., Seaborn, K.: Nonverbal cues in human-robot interaction: a communication studies perspective. ACM Trans. Hum.-Robot Interact. 12(2), 1–21 (2023)CrossRef Urakami, J., Seaborn, K.: Nonverbal cues in human-robot interaction: a communication studies perspective. ACM Trans. Hum.-Robot Interact. 12(2), 1–21 (2023)CrossRef
8.
go back to reference Ham, J., Bokhorst, R., Cuijpers, R., van der Pol, D., Cabibihan, J.-J.: Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power. In: Mutlu, B., Bartneck, C., Ham, J., Evers, V., Kanda, T. (eds.) ICSR 2011. LNCS (LNAI), vol. 7072, pp. 71–83. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25504-5_8CrossRef Ham, J., Bokhorst, R., Cuijpers, R., van der Pol, D., Cabibihan, J.-J.: Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power. In: Mutlu, B., Bartneck, C., Ham, J., Evers, V., Kanda, T. (eds.) ICSR 2011. LNCS (LNAI), vol. 7072, pp. 71–83. Springer, Heidelberg (2011). https://​doi.​org/​10.​1007/​978-3-642-25504-5_​8CrossRef
9.
go back to reference He, J., Baxter, S.L., Xu, J., Xu, J., Zhou, X., Zhang, K.: The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25(1), 30–36 (2019)CrossRef He, J., Baxter, S.L., Xu, J., Xu, J., Zhou, X., Zhang, K.: The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25(1), 30–36 (2019)CrossRef
10.
go back to reference Sahija, D.: Critical review of mixed reality integration with medical devices for patientcare. 2022, 10 (2022) Sahija, D.: Critical review of mixed reality integration with medical devices for patientcare. 2022, 10 (2022)
11.
go back to reference Snoswell, A.J., Snoswell, C.L.: Immersive virtual reality in health care: systematic review of technology and disease states. JMIR Biomed. Eng. 4(1), e15025 (2019)CrossRef Snoswell, A.J., Snoswell, C.L.: Immersive virtual reality in health care: systematic review of technology and disease states. JMIR Biomed. Eng. 4(1), e15025 (2019)CrossRef
12.
go back to reference Davenport, T., Kalakota, R.: The potential for artificial intelligence in healthcare. Future Healthc. J. 6(2), 94 (2019)CrossRef Davenport, T., Kalakota, R.: The potential for artificial intelligence in healthcare. Future Healthc. J. 6(2), 94 (2019)CrossRef
13.
go back to reference Kyrarini, M., et al.: A survey of robots in healthcare. Technologies 9(1), 8 (2021)CrossRef Kyrarini, M., et al.: A survey of robots in healthcare. Technologies 9(1), 8 (2021)CrossRef
14.
go back to reference Cabibihan, J.J., Javed, H., Ang, M., Aljunied, S.M.: Why robots? a survey on the roles and benefits of social robots in the therapy of children with autism. Int. J. Soc. Robot. 5, 593–618 (2013)CrossRef Cabibihan, J.J., Javed, H., Ang, M., Aljunied, S.M.: Why robots? a survey on the roles and benefits of social robots in the therapy of children with autism. Int. J. Soc. Robot. 5, 593–618 (2013)CrossRef
15.
go back to reference Mantovani, F., Castelnuovo, G., Gaggioli, A., Riva, G.: Virtual reality training for health-care professionals. Cyberpsychol. Behav. 6(4), 389–395 (2003)CrossRef Mantovani, F., Castelnuovo, G., Gaggioli, A., Riva, G.: Virtual reality training for health-care professionals. Cyberpsychol. Behav. 6(4), 389–395 (2003)CrossRef
16.
go back to reference Wang, M., Pan, C., Ray, P.K.: Technology entrepreneurship in developing countries: role of telepresence robots in healthcare. IEEE Eng. Manage. Rev. 49(1), 20–26 (2021)CrossRef Wang, M., Pan, C., Ray, P.K.: Technology entrepreneurship in developing countries: role of telepresence robots in healthcare. IEEE Eng. Manage. Rev. 49(1), 20–26 (2021)CrossRef
17.
go back to reference Dang, B.K., O’Leary-Kelley, C., Palicte, J.S., Badheka, S., Vuppalapati, C.: Comparing virtual reality telepresence and traditional simulation methods: a pilot study. Nurs. Educ. Perspect. 41(2), 119–121 (2020)CrossRef Dang, B.K., O’Leary-Kelley, C., Palicte, J.S., Badheka, S., Vuppalapati, C.: Comparing virtual reality telepresence and traditional simulation methods: a pilot study. Nurs. Educ. Perspect. 41(2), 119–121 (2020)CrossRef
18.
go back to reference Juan, M.C., Elexpuru, J., Dias, P., Santos, B.S., Amorim, P.: Immersive virtual reality for upper limb rehabilitation: comparing hand and controller interaction. Virtual Reality, 1–15 (2022) Juan, M.C., Elexpuru, J., Dias, P., Santos, B.S., Amorim, P.: Immersive virtual reality for upper limb rehabilitation: comparing hand and controller interaction. Virtual Reality, 1–15 (2022)
19.
go back to reference McGloin, R., Farrar, K., Krcmar, M.: Video games, immersion, and cognitive aggression: does the controller matter? Media Psychol. 16(1), 65–87 (2013)CrossRef McGloin, R., Farrar, K., Krcmar, M.: Video games, immersion, and cognitive aggression: does the controller matter? Media Psychol. 16(1), 65–87 (2013)CrossRef
20.
go back to reference Xie, B., et al.: A review on virtual reality skill training applications. Front. Virtual Reality 2, 645153 (2021)CrossRef Xie, B., et al.: A review on virtual reality skill training applications. Front. Virtual Reality 2, 645153 (2021)CrossRef
21.
go back to reference Cabibihan, J.J., et al.: Suitability of the openly accessible 3D printed prosthetic hands for war-wounded children. Front. Robot. AI 7, 594196 (2021)CrossRef Cabibihan, J.J., et al.: Suitability of the openly accessible 3D printed prosthetic hands for war-wounded children. Front. Robot. AI 7, 594196 (2021)CrossRef
22.
go back to reference Zhang, X.Y., Nakamura, Y., Goda, K., Yoshimoto, K.: Robustness of power grasp. In: Proceedings of the 1994 IEEE International Conference on Robotics and Automation, vol. 4, pp. 2828–2835 (1994) Zhang, X.Y., Nakamura, Y., Goda, K., Yoshimoto, K.: Robustness of power grasp. In: Proceedings of the 1994 IEEE International Conference on Robotics and Automation, vol. 4, pp. 2828–2835 (1994)
23.
go back to reference Bullock, I.M., Zheng, J.Z., De La Rosa, S., Guertler, C., Dollar, A.M.: Grasp frequency and usage in daily household and machine shop tasks. IEEE Trans. Haptics 6(3), 296–308 (2013)CrossRef Bullock, I.M., Zheng, J.Z., De La Rosa, S., Guertler, C., Dollar, A.M.: Grasp frequency and usage in daily household and machine shop tasks. IEEE Trans. Haptics 6(3), 296–308 (2013)CrossRef
24.
go back to reference Demartino, A.M., Rodrigues, L.C., Gomes, R.P., Michaelsen, S.M.: Hand function and type of grasp used by chronic stroke individuals in actual environment. Top. Stroke Rehabil. 26(4), 247–254 (2019)CrossRef Demartino, A.M., Rodrigues, L.C., Gomes, R.P., Michaelsen, S.M.: Hand function and type of grasp used by chronic stroke individuals in actual environment. Top. Stroke Rehabil. 26(4), 247–254 (2019)CrossRef
25.
go back to reference Goršič, M., Cikajlo, I., Goljar, N., Novak, D.: A multisession evaluation of an adaptive competitive arm rehabilitation game. J. Neuroeng. Rehabil. 14(1), 1–15 (2017)CrossRef Goršič, M., Cikajlo, I., Goljar, N., Novak, D.: A multisession evaluation of an adaptive competitive arm rehabilitation game. J. Neuroeng. Rehabil. 14(1), 1–15 (2017)CrossRef
26.
go back to reference Brooke, J.: SUS-a quick and dirty usability scale. Usability Evaluation in Industry, 189 (1996) Brooke, J.: SUS-a quick and dirty usability scale. Usability Evaluation in Industry, 189 (1996)
27.
go back to reference Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)CrossRef Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)CrossRef
32.
go back to reference Gentry, S.V., et al.: Serious gaming and gamification education in health professions: systematic review. J. Med. Internet Res. 21(3), e12994 (2019)CrossRef Gentry, S.V., et al.: Serious gaming and gamification education in health professions: systematic review. J. Med. Internet Res. 21(3), e12994 (2019)CrossRef
33.
go back to reference Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum.-Comput. Interact. 2013, 1–1 (2013)CrossRef Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum.-Comput. Interact. 2013, 1–1 (2013)CrossRef
Metadata
Title
Virtual Reality Hand Tracking for Immersive Telepresence in Rehabilitative Serious Gaming
Authors
Noaman Mazhar
Aya Gaballa
Amit Kumar Pandey
John-John Cabibihan
Copyright Year
2024
Publisher
Springer Nature Singapore
DOI
https://doi.org/10.1007/978-981-99-8715-3_1

Premium Partner