Skip to main content
main-content

Tipp

Weitere Artikel dieser Ausgabe durch Wischen aufrufen

Erschienen in: Mobile Networks and Applications 2/2020

29.03.2019

An Augmented Reality-Based Method for Remote Collaborative Real-Time Assistance: from a System Perspective

verfasst von: Dikai Fang, Huahu Xu, Xiaoxian Yang, Minjie Bian

Erschienen in: Mobile Networks and Applications | Ausgabe 2/2020

Einloggen, um Zugang zu erhalten
share
TEILEN

Abstract

To provide remote assistance to people more efficiently, an augmented reality (AR)-based method for remote real-time assistance for collaboration is proposed. This paper aims to reduce communication barriers and enhance the three-dimensional (3D) feel of immersive interactions. First, a multiplayer real-time video communication framework with WebRTC is built, which enables remote experts to observe a first-hand view of an operator’s site. Second, a shared cross-platform virtual whiteboard based on Canvas, WebSocket and Node.js is developed that enables remote experts to provide visual assistance, such as drawings or text, and adjust the position of the whiteboard for seamless integration with video. Last, the virtual assistance information provided by the remote experts is displayed on the screen of AR holographic glasses to enhance the assistance capability of the platform and enable an expert to explain to an operator how to correctly perform tasks. A hybrid tracking and registration technique based on natural features and gyroscopes is adopted to estimate the operator’s posture in real time to enable the virtual assistance information to be perfectly integrated with the real world at all times. An experimental analysis shows that this system is both practicable and stable and has broad application prospects in many fields.

Sie möchten Zugang zu diesem Inhalt erhalten? Dann informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 69.000 Bücher
  • über 500 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Testen Sie jetzt 15 Tage kostenlos.

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 50.000 Bücher
  • über 380 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




Testen Sie jetzt 15 Tage kostenlos.

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 58.000 Bücher
  • über 300 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Testen Sie jetzt 15 Tage kostenlos.

Literatur
1.
Zurück zum Zitat Bajura M, Neumann U (1995) Dynamic registration correction in augmented-reality systems[C]. In: Virtual reality international symposium, 1995. Proceedings. IEEE, pp 189–196 Bajura M, Neumann U (1995) Dynamic registration correction in augmented-reality systems[C]. In: Virtual reality international symposium, 1995. Proceedings. IEEE, pp 189–196
2.
Zurück zum Zitat Hanna MG, Ahmed I, Nine J et al (2018) Augmented reality technology using Microsoft HoloLens in anatomic pathology [J]. Arch Pathol Lab Med 142(5):638–644 CrossRef Hanna MG, Ahmed I, Nine J et al (2018) Augmented reality technology using Microsoft HoloLens in anatomic pathology [J]. Arch Pathol Lab Med 142(5):638–644 CrossRef
3.
Zurück zum Zitat Shenai MB, Dillavou M, Shum C et al (2011) Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance.[J]. Neurosurgery 68(2):200–207 Shenai MB, Dillavou M, Shum C et al (2011) Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance.[J]. Neurosurgery 68(2):200–207
4.
Zurück zum Zitat Choi J, Yoon B, Jung C et al (2017) ARClassNote: augmented reality based remote education solution with tag recognition and shared hand-written note[C]. In: IEEE international symposium on mixed and augmented reality. IEEE, pp 303–309 Choi J, Yoon B, Jung C et al (2017) ARClassNote: augmented reality based remote education solution with tag recognition and shared hand-written note[C]. In: IEEE international symposium on mixed and augmented reality. IEEE, pp 303–309
6.
Zurück zum Zitat Fette I, Melnikov A (2011) The WebSocket protocol. IETF internet draft, work in progress Fette I, Melnikov A (2011) The WebSocket protocol. IETF internet draft, work in progress
10.
Zurück zum Zitat Jang-Jaccard J, Nepal S, Celler B et al (2016) WebRTC-based video conferencing service for telehealth [J]. Computing 98(1-2):169–193 MathSciNetCrossRef Jang-Jaccard J, Nepal S, Celler B et al (2016) WebRTC-based video conferencing service for telehealth [J]. Computing 98(1-2):169–193 MathSciNetCrossRef
11.
Zurück zum Zitat Ma LV, Kim J, Park S et al (2016) An efficient Session_Weight load balancing and scheduling methodology for high-quality telehealth care service based on WebRTC [J]. J Supercomput 72(10):3909–3926 CrossRef Ma LV, Kim J, Park S et al (2016) An efficient Session_Weight load balancing and scheduling methodology for high-quality telehealth care service based on WebRTC [J]. J Supercomput 72(10):3909–3926 CrossRef
12.
Zurück zum Zitat Santosgonzález I, Riverogarcía A, Molinagil J et al (2017) Implementation and analysis of real-time streaming protocols [J]. Sensors 17(4):846 CrossRef Santosgonzález I, Riverogarcía A, Molinagil J et al (2017) Implementation and analysis of real-time streaming protocols [J]. Sensors 17(4):846 CrossRef
13.
Zurück zum Zitat Johnston AB, Burnett DC (2012) WebRTC: APIs and RTCWEB protocols of the HTML5 real-time web [M]. Digital codex LLC Johnston AB, Burnett DC (2012) WebRTC: APIs and RTCWEB protocols of the HTML5 real-time web [M]. Digital codex LLC
14.
Zurück zum Zitat Shen-hui C (2013) The study and implementation of NAT traversal technology. Nanjing University of Posts and Telecommunications Shen-hui C (2013) The study and implementation of NAT traversal technology. Nanjing University of Posts and Telecommunications
15.
Zurück zum Zitat Ringe S, Kedia R, Poddar A et al (2015) HTML5 based virtual whiteboard for real time interaction ☆[J]. Procedia Comput Sci 49(1):170–177 CrossRef Ringe S, Kedia R, Poddar A et al (2015) HTML5 based virtual whiteboard for real time interaction ☆[J]. Procedia Comput Sci 49(1):170–177 CrossRef
16.
Zurück zum Zitat Antunes M, Silva C, Barranca JA (2016) A Telemedicine application using WebRTC ☆ [J]. Procedia Comput Sci 100:414–420 CrossRef Antunes M, Silva C, Barranca JA (2016) A Telemedicine application using WebRTC ☆ [J]. Procedia Comput Sci 100:414–420 CrossRef
17.
Zurück zum Zitat Pimentel V, Nickerson BG (2012) Communicating and displaying real-time data with WebSocket [J]. IEEE Internet Comput 16(4):45–53 CrossRef Pimentel V, Nickerson BG (2012) Communicating and displaying real-time data with WebSocket [J]. IEEE Internet Comput 16(4):45–53 CrossRef
18.
Zurück zum Zitat Tilkov S, Vinoski S (2010) Node.js: using JavaScript to build high-performance network programs [J]. IEEE Internet Comput 14(6):80–83 CrossRef Tilkov S, Vinoski S (2010) Node.js: using JavaScript to build high-performance network programs [J]. IEEE Internet Comput 14(6):80–83 CrossRef
19.
Zurück zum Zitat Gabor D (1948) A new microscopic principle. [J]. Nature 161(4098):777 CrossRef Gabor D (1948) A new microscopic principle. [J]. Nature 161(4098):777 CrossRef
20.
Zurück zum Zitat Zeng F, Zhang X, Zhang J-p et al (2013) Holographic waveguide head-mounted display system design based on prisms-grating structure [J]. Acta Opt Sin 33(9):114–119 Zeng F, Zhang X, Zhang J-p et al (2013) Holographic waveguide head-mounted display system design based on prisms-grating structure [J]. Acta Opt Sin 33(9):114–119
21.
Zurück zum Zitat Anousaki GC, Kyriakopoulos KJ (1999) Simultaneous localization and map building for mobile robot navigation [J]. IEEE Robot Autom Mag 6(3):42–53 CrossRef Anousaki GC, Kyriakopoulos KJ (1999) Simultaneous localization and map building for mobile robot navigation [J]. IEEE Robot Autom Mag 6(3):42–53 CrossRef
22.
Zurück zum Zitat Davison AJ, Reid ID, Molton ND et al (2007) MonoSLAM: real-time single camera SLAM.[J]. IEEE Trans Pattern Anal Mach Intell 29(6):1052–1067 CrossRef Davison AJ, Reid ID, Molton ND et al (2007) MonoSLAM: real-time single camera SLAM.[J]. IEEE Trans Pattern Anal Mach Intell 29(6):1052–1067 CrossRef
23.
Zurück zum Zitat Mur-Artal R, Montiel JMM, Tardós JD (2017) ORB-SLAM: a versatile and accurate monocular SLAM system [J]. IEEE Trans Robot 31(5):1147–1163 CrossRef Mur-Artal R, Montiel JMM, Tardós JD (2017) ORB-SLAM: a versatile and accurate monocular SLAM system [J]. IEEE Trans Robot 31(5):1147–1163 CrossRef
24.
Zurück zum Zitat Mirzaei FM, Roumeliotis SI (2007) A Kalman filter-based algorithm for IMU-camera calibration[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 2427–2434 Mirzaei FM, Roumeliotis SI (2007) A Kalman filter-based algorithm for IMU-camera calibration[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 2427–2434
25.
Zurück zum Zitat Zhou S-l, Wu X-z, Liu G et al (2016) Integrated navigation method of monocular ORB-SLAM/INS [J]. J Chin Iner Techno 24(5):633–637 Zhou S-l, Wu X-z, Liu G et al (2016) Integrated navigation method of monocular ORB-SLAM/INS [J]. J Chin Iner Techno 24(5):633–637
26.
Zurück zum Zitat Sun L, Du J, Qin W (2015) Research on combination positioning based on natural features and gyroscopes for AR on Mobile phones[C]. In: International conference on virtual reality and visualization. IEEE, pp 301–307 Sun L, Du J, Qin W (2015) Research on combination positioning based on natural features and gyroscopes for AR on Mobile phones[C]. In: International conference on virtual reality and visualization. IEEE, pp 301–307
27.
Zurück zum Zitat Bailey T, Nieto J, Guivant J et al (2006) Consistency of the EKF-SLAM algorithm[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 3562–3568 Bailey T, Nieto J, Guivant J et al (2006) Consistency of the EKF-SLAM algorithm[C]. In: Ieee/rsj international conference on intelligent robots and systems. IEEE, pp 3562–3568
28.
Zurück zum Zitat Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone[C]. In: IEEE international symposium on mixed and augmented reality. IEEE Computer Society, pp 83–86 Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone[C]. In: IEEE international symposium on mixed and augmented reality. IEEE Computer Society, pp 83–86
29.
Zurück zum Zitat Davis BS (1998) Using low-cost MEMS accelerometers and gyroscopes as strapdown IMUs on rolling projectiles[C]. In: Position location and navigation symposium, IEEE. IEEE, pp 594–601 Davis BS (1998) Using low-cost MEMS accelerometers and gyroscopes as strapdown IMUs on rolling projectiles[C]. In: Position location and navigation symposium, IEEE. IEEE, pp 594–601
30.
Zurück zum Zitat Anton D, Kurillo G, Yang AY et al (2017) Augmented telemedicine platform for real-time remote medical consultation[M]. In: MultiMedia modeling. Springer International Publishing, pp 77–89 Anton D, Kurillo G, Yang AY et al (2017) Augmented telemedicine platform for real-time remote medical consultation[M]. In: MultiMedia modeling. Springer International Publishing, pp 77–89
31.
Zurück zum Zitat Hou L, Wang X, Truijens M (2012) Using augmented reality to facilitate piping assembly: an experiment-based evaluation[J]. J Comput Civ Eng 29(1):05014007 CrossRef Hou L, Wang X, Truijens M (2012) Using augmented reality to facilitate piping assembly: an experiment-based evaluation[J]. J Comput Civ Eng 29(1):05014007 CrossRef
32.
Zurück zum Zitat Oyekan J, Prabhu V, Tiwari A et al (2017) Remote real-time collaboration through synchronous exchange of digitised human-workpiece interactions[J]. Futur Gener Comput Syst 67:83–93 CrossRef Oyekan J, Prabhu V, Tiwari A et al (2017) Remote real-time collaboration through synchronous exchange of digitised human-workpiece interactions[J]. Futur Gener Comput Syst 67:83–93 CrossRef
33.
Zurück zum Zitat Mimaroğlu O (2014) Collaborative augmented reality[J]. Commun ACM 45(7):64–70 Mimaroğlu O (2014) Collaborative augmented reality[J]. Commun ACM 45(7):64–70
34.
Zurück zum Zitat Kim Y, Hong S, Kim GJ (2017) Augmented reality-based remote coaching for fast-paced physical task[J]. Virtual Reality 6:1–12 Kim Y, Hong S, Kim GJ (2017) Augmented reality-based remote coaching for fast-paced physical task[J]. Virtual Reality 6:1–12
35.
Zurück zum Zitat Cidota M, Lukosch S, Datcu D et al (2016) Workspace awareness in collaborative AR using HMDS: a user study comparing audio and visual notifications. In: Proceedings of the 7th augmented human international conference. ACM Press, New York Cidota M, Lukosch S, Datcu D et al (2016) Workspace awareness in collaborative AR using HMDS: a user study comparing audio and visual notifications. In: Proceedings of the 7th augmented human international conference. ACM Press, New York
36.
Zurück zum Zitat Gauglitz S, Nuernberger B, Turk M (2014) In touch with the remote world:remote collaboration with augmented reality drawings and virtual navigation[C]. In: Proceedings of the 20th ACM symposium on virtual reality software and technology. ACM Press, New York, pp 197–205 Gauglitz S, Nuernberger B, Turk M (2014) In touch with the remote world:remote collaboration with augmented reality drawings and virtual navigation[C]. In: Proceedings of the 20th ACM symposium on virtual reality software and technology. ACM Press, New York, pp 197–205
Metadaten
Titel
An Augmented Reality-Based Method for Remote Collaborative Real-Time Assistance: from a System Perspective
verfasst von
Dikai Fang
Huahu Xu
Xiaoxian Yang
Minjie Bian
Publikationsdatum
29.03.2019
Verlag
Springer US
Erschienen in
Mobile Networks and Applications / Ausgabe 2/2020
Print ISSN: 1383-469X
Elektronische ISSN: 1572-8153
DOI
https://doi.org/10.1007/s11036-019-01244-4

Weitere Artikel der Ausgabe 2/2020

Mobile Networks and Applications 2/2020 Zur Ausgabe