Top

The International Journal of Advanced Manufacturing Technology

10-05-2024 | ORIGINAL ARTICLE

A blind area information perception and AR assembly guidance method based on RGBD data for dynamic environments and user study

Authors: Zhenghang Hou, Qiang Zhang, Shuxia Wang, Weiping He, Shaohua Zhang

Published in: The International Journal of Advanced Manufacturing Technology

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In this research, a blind area information perception and guidance approach for dynamic context is proposed as a solution to the issue of difficult and time-consuming assembly in blind areas. The proposed approach involves the utilization of real-time RGBD data to perceive both blind area context and operator hand information. The resulting data is then used to visualize the blind area scene and provide assembly guidance through the application of augmented reality technology. Unlike conventional methods, the proposed solutions are based on dynamic RGBD data rather than static predefined CAD models, making it simpler to configure and adapt to more scenarios. A user study was designed and conducted to confirm the feasibility of the suggested approach. The results indicate that the suggested approach can decrease assembly time by 49.5%, greatly lower the percentage of assembly errors, reduce the mental load on the workers, and significantly enhance their operational experience.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Kayani A, Jamshidi J (2007) Measurement assisted assembly for large volume aircraft wing structures. In: 4th International conference on digital enterprise technology, pp 426–434

Hu SJ, Ko J, Weyand L, ElMaraghy HA, Lien TK, Koren Y, Bley H, Chryssolouris G, Nasr N, Shpitalni M (2011) Assembly system design and operations for product variety. CIRP Annals 60(2):715–733. https://doi.org/10.1016/j.cirp.2011.05.004CrossRef

Jung S, Bruder G, Wisniewski PJ, Sandor C, Hughes CE (2018) Over my hand: Using a personalized hand in vr to improve object size estimation, body ownership, and presence. In: Proceedings of the 2018 ACM Symposium on Spatial User Interaction. SUI ’18, pp 60–68. Association for Computing Machinery. https://doi.org/10.1145/3267782.3267920

State A, Hirota G, Chen DT, Garrett WF, Livingston MA (1996) Superior augmented reality registration by integrating landmark tracking and magnetic tracking. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 429–438

Martinetti A, Marques HC, Singh S, Dongen L (2019) Reflections on the limited pervasiveness of augmented reality in industrial sectors. Applied Sciences 9(16). https://doi.org/10.3390/app9163382

Nee AYC, Ong SK (2013) Virtual and augmented reality applications in manufacturing. IFAC Proceedings Volumes 46(9):15–26. https://doi.org/10.3182/20130619-3-RU-3018.00637. 7th IFAC Conference on Manufacturing Modelling, Management, and Control

Feng S, Liu Y, Zhang Q, He W, Zhang X, Wang S, Billinghurst M (2023) Parallel or cross? effects of two collaborative modes on augmented reality co-located operations. Int J Hum-Comput Interact, pp 1–12. https://doi.org/10.1080/10447318.2023.2202574

Zhang S, He W, Wang S, Feng S, Hou Z, Hu Y (2021) An AR-enabled see-through system for vision blind areas. In: Stephanidis C, Antona M, Ntoa S (eds) HCI International 2021 - Late Breaking Posters, pp 206–213. Springer. https://doi.org/10.1007/978-3-030-90176-9_28

Dargar S, Kennedy R, Lai W, Arikatla V, De S (2015) Towards immersive virtual reality (ivr): a route to surgical expertise. J Comput Surg 2(1):1–26. https://doi.org/10.1186/s40244-015-0015-8CrossRef

10.

Chu C-H, Ko C-H (2021) An experimental study on augmented reality assisted manual assembly with occluded components. J Manuf Syst 61:685–695. https://doi.org/10.1016/j.jmsy.2021.04.003CrossRef

11.

Gupta S, Chaudhary R, Gupta S, Kaur A, Mantri A (2019) A survey on tracking techniques in augmented reality based application. In: 2019 Fifth international conference on image information processing (ICIIP), pp 215–220. https://doi.org/10.1109/ICIIP47207.2019.8985779

12.

Thomas PC, David W (1992) Augmented reality: An application of heads-up display technology to manual manufacturing processes. In: Hawaii international conference on system sciences, vol 2. ACM SIGCHI Bulletin

13.

Rios H, González E, Rodriguez C, Siller HR, Contero M (2013) A mobile solution to enhance training and execution of troubleshooting techniques of the engine air bleed system on boeing 737. Procedia Comput Sci 25:161–170. https://doi.org/10.1016/j.procs.2013.11.020. 2013 International Conference on Virtual and Augmented Reality in Education

14.

Wang Z, Zhang S, Bai Q (2019) Augmented reality based product blind area assembly assistance. Manuf Autom 41(1):29–35. https://doi.org/10.3969/j.issn.1009-0134.2019.01.007CrossRef

15.

Feng S, He W, Zhang S, Billinghurst M (2022) Seeing is believing: AR-assisted blind area assembly to support hand-eye coordination. J Adv Manuf Technol 119(11–12):8149–8158. https://doi.org/10.1007/s00170-021-08546-6CrossRef

16.

Valentini PP (2018) Natural interface for interactive virtual assembly in augmented reality using leap motion controller. Int J Interact Des Manuf (IJIDeM) 12(4):1157–1165. https://doi.org/10.1007/s12008-018-0461-0CrossRef

17.

Wang Z, Yan Y, Han D, Bai X, Zhang S (2019) Product blind area assembly method based on augmented reality and machine vision. Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University 37(3):496–502CrossRef

18.

Johansson RS, Westling G, Bäckström A, Flanagan JR (2001) Eye-hand coordination in object manipulation. J Neurosci 21(17):6917–6932CrossRef

19.

Zhang F, Bazarevsky V, Vakunov A, Tkachenka A, Sung G, Chang C-L, Grundmann M (2020) Mediapipe hands: On-device real-time hand tracking. https://doi.org/10.48550/arXiv.2006.10214

20.

Khenak N, Vézien J, Bourdot P (2020) Effectiveness of augmented reality guides for blind insertion tasks. Front Virtual Real 1:588217. https://doi.org/10.3389/frvir.2020.588217CrossRef

21.

Lugaresi C, Tang J, Nash H, McClanahan C, Uboweja E, Hays M, Zhang F, Chang C-L, Yong MG, Lee J et al (2019) Mediapipe: A framework for building perception pipelines. https://doi.org/10.48550/arXiv.1906.08172

22.

Feng S, He X, He W, Li R, Zhang S (2023) First-and third-person perspectives in the AR-assisted blind area assembly

23.

Bangor A, Kortum PT, Miller JT (2008) An empirical evaluation of the system usability scale. Int J Hum-Comput Interact 24(6):574–594. https://doi.org/10.1080/10447310802205776CrossRef

24.

Zijlstra F, Doorn L (1985) The construction of a scale to measure perceived effort. Department of Philosophy and Social Sciences

25.

Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. In: Holzinger A (ed) HCI and usability for education and work. Springer, pp 63–76

Title: A blind area information perception and AR assembly guidance method based on RGBD data for dynamic environments and user study
Authors: Zhenghang Hou
Qiang Zhang
Shuxia Wang
Weiping He
Shaohua Zhang
Publication date: 10-05-2024
Publisher: Springer London
Published in: The International Journal of Advanced Manufacturing Technology
Print ISSN: 0268-3768
Electronic ISSN: 1433-3015
DOI: https://doi.org/10.1007/s00170-024-13599-4

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Premium Partners