nach oben

The International Journal of Advanced Manufacturing Technology

Erschienen in:

02.02.2022 | ORIGINAL ARTICLE

Gesture- and vision-based automatic grasping and flexible placement in teleoperation

verfasst von: Zhao Xue, Xiaoan Chen, Ye He, Hongli Cao, Shengli Tian

Erschienen in: The International Journal of Advanced Manufacturing Technology | Ausgabe 1/2022

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The teleoperation system has attracted a lot of attention because of its advantages in dangerous or unknown environments. It is very difficult to develop an operating system that can complete complex tasks in a completely autonomous. This paper proposes a robot arm control strategy based on gestures and visual perception. The strategy combines the advantages of humans and robots to obtain a convenient and flexible interaction model. The hand’s data were obtained by Leap-Motion. Then a neural network algorithm was used to classify the nine gestures, which were used for robots control through a finite state machine. The control mode switched between indicative control and mapping control. The robot acquired an autonomous grasp ability by incorporating YOLO 6D, depth data and a probabilistic roadmap planner algorithm. The robot completed most of the trajectory independently, while a few flexible trajectories required users to do mapping actions. This interactive mode reduces the burden of the users to a certain extent that makes up for the shortcomings of traditional teleoperation.

Vorheriger Artikel Development of a nozzle capillary viscometer for inline viscosity measurement of thermoplastics

Nächster Artikel An energy-concentrated synchrosqueezing transform using a reassignment operator for the analysis of nonstationary signals

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Tsitsimpelis I, Taylor CJ, Lennox B et al (2019) A review of ground-based robotic systems for the characterization of nuclear environments. Prog Nucl Energy 111:109–124. https://doi.org/10.1016/j.pnucene.2018.10.023

Birk A, Doernbach T, Mueller C et al (2018) Dexterous underwater manipulation from onshore locations: Streamlining efficiencies for remotely operated underwater vehicles. IEEE Robotics Automation Magazine 25:24–33. https://doi.org/10.1109/MRA.2018.2869523

Chen Z, Huang F, Yang C et al (2020) Adaptive Fuzzy Backstepping Control for Stable Nonlinear Bilateral Teleoperation Manipulators With Enhanced Transparency Performance. Ieee T Ind Electron 67:746–756. https://doi.org/10.1109/tie.2019.2898587

Bachmann ER, McGhee RB, Yun X et al (2001) Inertial and magnetic posture tracking for inserting humans into networked virtual environments Proceedings of the ACM symposium on Virtual reality software and technology. 9–16 https://doi.org/10.1145/505009.505011

Filippeschi A, Schmitz N, Miezal M et al (2017) Survey of motion tracking methods based on inertial sensors: A focus on upper limb human motion. Sensors 17:1257. https://doi.org/10.3390/s17061257

Qi J, Jiang G, Li G et al (2020) Surface EMG hand gesture recognition system based on PCA and GRNN. Neural Comput Appl 32:6343–6351. https://doi.org/10.1007/s00521-019-04142-8

Kebria PM, Abdi H, Dalvand MM et al (2018) Control methods for internet-based teleoperation systems: A review. IEEE Transactions on Human-Machine Systems 49:32–46. https://doi.org/10.1109/THMS.2018.2878815

Zhang Y, Huang Y, Sun X et al (2020) Static and dynamic human arm/hand gesture capturing and recognition via multiinformation fusion of flexible strain sensors. IEEE Sens J 20:6450–6459. https://doi.org/10.1109/JSEN.2020.2965580

Chakraborty BK, Sarma D, Bhuyan MK et al (2017) Review of constraints on vision-based gesture recognition for human–computer interaction. IET Comput Vision 12:3–15. https://doi.org/10.1049/iet-cvi.2017.0052

10.

Tsarouchi P, Athanasatos A, Makris S et al (2016) High level robot programming using body and hand gestures. Procedia Cirp 55:1–5. https://doi.org/10.1016/j.procir.2016.09.020

11.

Zeng H, Shen Y, Hu X et al (2020) Semi-autonomous robotic arm reaching with hybrid gaze–brain machine interface. Front Neurorobot 13:111. https://doi.org/10.3389/fnbot.2019.00111

12.

Lan T-S, Her M-G, Hsu K-S (2003) Virtual reality application for direct-drive robot with force feedback. Int J Adv Manuf Technol 21:66–71. https://doi.org/10.1007/s001700300007CrossRef

13.

Her M-G, Hsu K-S, Yu W-S (2002) Analysis and design of a haptic control system: virtual reality approach. Int J Adv Manuf Technol 19:743–751. https://doi.org/10.1007/s001700200085CrossRef

14.

Goodrich MA, Crandall JW, Barakova E (2013) Teleoperation and beyond for assistive humanoid robots. Rev Hum Factors Ergon 9:175–226. https://doi.org/10.1177/1557234X13502463CrossRef

15.

Liang JS, Chao K-M, Ivey P (2013) VR-based wheeled mobile robot in application of remote real-time assembly. Int J Adv Manuf Technol 64:1765–1779. https://doi.org/10.1007/s00170-012-4140-1CrossRef

16.

Hayati S, Venkataraman S (1989) Design and implementation of a robot control system with traded and shared control capability. IEEE Int Conf Robot Autom 1989:1310–1315. https://doi.org/10.1109/ROBOT.1989.100161CrossRef

17.

Bauer AS, Schmaus P, Albu-Schäffer A et al (2018) Inferring semantic state transitions during telerobotic manipulation. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS): 1–9. https://doi.org/10.1109/IROS.2018.8594458

18.

Du G, Yao G, Li C et al (2019) Natural human–robot interface using adaptive tracking system with the unscented Kalman filter. IEEE Transactions on Human-Machine Systems 50:42–54. https://doi.org/10.1109/THMS.2019.2947576

19.

Yang C, Wang X, Cheng L et al (2016) Neural-learning-based telerobot control with guaranteed performance. IEEE transactions on cybernetics 47:3148–3159. https://doi.org/10.1109/TCYB.2016.2573837

20.

Kavraki LE, Svestka P, Latombe J et al (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. Int Conf Robot Autom 12: 566–580. https://doi.org/10.1109/70.508439

21.

Xue Y, Ju Z, Xiang K et al (2018) Multimodal human hand motion sensing and analysis—A review. IEEE Transactions on Cognitive Developmental Systems 11:162–175. https://doi.org/10.1109/TCDS.2018.2800167

22.

Hernandez C, Bharatheesha M, Ko W et al (2016) Team delft’s robot winner of the amazon picking challenge. Robot World Cup. 613–624 https://doi.org/10.1007/978-3-319-68792-6_51

23.

Sahin C, Garcia-Hernando G, Sock J et al (2020) A review on object pose recovery: from 3d bounding box detectors to full 6d pose estimators. Image Vis Comput 96

24.

Redmon J, Divvala S, Girshick R et al (2016) You only look once: Unified, real-time object detection Proc IEEE Conf Comput Vis Pattern Recognit. 779–788 https://doi.org/10.1109/CVPR.2016.91

25.

Tekin B, Sinha SN, Fua P (2018) Real-time seamless single shot 6d object pose prediction Proc IEEE Conf Comput Vis Pattern Recognit. 292–301 https://doi.org/10.1109/CVPR.2018.00038

26.

Bachmann D, Weichert F, Rinkenauer G (2018) Review of three-dimensional human-computer interaction with focus on the leap motion controller. J Sensors 18:2194. https://doi.org/10.3390/s18072194CrossRef

27.

Du GL, Zhang P, Liu X (2016) Markerless Human-Manipulator Interface Using Leap Motion With Interval Kalman Filter and Improved Particle Filter. IEEE Trans Industr Inf 12:694–704. https://doi.org/10.1109/tii.2016.2526674CrossRef

28.

Luzanin O, Plancak M (2014) Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network. Assem Autom. 34: 94–105. https://doi.org/10.1108/AA-03-2013-020

29.

Wu B-X, Yang C-G, Zhong J-P (2021) Research on Transfer Learning of Vision-based Gesture Recognition. Int J Autom Comput. 1–10 https://doi.org/10.1007/s11633-020-1273-9

30.

Zhao X, He Y, Chen X et al (2021) Human-Robot Collaborative Assembly Based on Eye-Hand and a Finite State Machine in a Virtual Environment. Appl Sci 11:5754. https://doi.org/10.3390/app11125754

31.

Hodan T, Haluza P, Obdržálek Š et al (2017) T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV): 880–888. https://doi.org/10.1109/34.121791

32.

Engelcke M, Rao D, Wang DZ et al (2017) Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. 2017 IEEE Int Conf Robot Autom (ICRA): 1355–1361. https://doi.org/10.1109/ICRA.2017.7989161

33.

Fang B, Sun F, Liu H et al (2020). Wearable Technology for Robotic Manipulation and Learning Springer. https://doi.org/10.1007/978-981-15-5124-6

34.

Timmi A, Coates G, Fortin K et al (2018) Accuracy of a novel marker tracking approach based on the low-cost Microsoft Kinect v2 sensor. Med Eng Phys 59:63–69. https://doi.org/10.1016/j.medengphy.2018.04.020

35.

Besl PJ, Mckay HD (1992) A method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell. IEEE Trans Pattern Anal Mach Intell 14:239–256. https://doi.org/10.1109/34.121791CrossRef

36.

Zhang T, Su J (2018) Collision-free planning algorithm of motion path for the robot belt grinding system. Int J Adv Rob Syst 15:1729881418793778. https://doi.org/10.1177/1729881418793778CrossRef

37.

Ye G, Alterovitz R (2017) Demonstration-Guided Motion Planning. Int Symp Robot. 291–307. https://doi.org/10.1007/978-3-319-29363-9_17

38.

Ichter B, Harrison J, Pavone M (2018) Learning Sampling Distributions for Robot Motion Planning. Int Conf Robot Autom. 7087–7094. https://doi.org/10.1109/ICRA.2018.8460730

39.

Mu B, Giamou M, Paull L et al (2016) Information-based active SLAM via topological feature graphs. Conference on Decision and Control. 5583–5590 https://doi.org/10.1109/CDC.2016.7799127

40.

Wang P, Gao S, Li L et al (2019) Obstacle avoidance path planning design for autonomous driving vehicles based on an improved artificial potential field algorithm. Energies 12:2342. https://doi.org/10.3390/en12122342

41.

Francis A, Faust A, Chiang H-TL et al (2020) Long-range indoor navigation with prm-rl. IEEE Trans Rob 36:1115–1134. https://doi.org/10.1109/TRO.2020.2975428

42.

Zhao X, Chen X, He Y et al (2019) Varying Speed Rate Controller for Human-Robot Teleoperation Based on Muscle Electrical Signals. IEEE Access 7:143563–143572. https://doi.org/10.1109/ACCESS.2019.2944794

43.

Höglund T, Alander J, Mantere T (2018) A survey of telerobotic surface finishing. Nephron Clin Pract 8:156–161. https://doi.org/10.1515/eng-2018-0018CrossRef

44.

Katsanos C, Hadzilacos, et al (2015) Usability Guidelines for the Design of Robot Teleoperation: A Taxonomy. IEEE Trans Hum Mach Syst 25:256–262. https://doi.org/10.1109/THMS.2014.2371048

45.

Handa A, Van Wyk K, Yang W et al (2020) DexPilot: Vision-based teleoperation of dexterous robotic hand-arm system. 2020 IEEE Int Conf Robot Autom (ICRA): 9164–9170

46.

Song J, Chen Q, Li Z (2021) A peg-in-hole robot assembly system based on Gauss mixture model. Robot Comput Integr Manuf 67

47.

Kong L, Peng X, Chen Y et al (2020) Multi-sensor measurement and data fusion technology for manufacturing process monitoring: a literature review. Int J Extreme Manuf 2

48.

Zhang J, Zhang W, Song R et al (2020) Grasp for stacking via deep reinforcement learning. 2020 IEEE Int Conf Robot Autom (ICRA): 2543–2549. https://doi.org/10.1109/ICRA40945.2020.9197508

49.

Yang C, Lan X, Zhang H et al (2019) Task-oriented grasping in object stacking scenes with CRF-based semantic model. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS): 6427–6434. https://doi.org/10.1109/IROS40897.2019.8967992

50.

Asif U, Bennamoun M, Sohel F (2014) Model-free segmentation and grasp selection of unknown stacked objects. European Conference on Computer Vision (ECCV). 659–674 https://doi.org/10.1007/978-3-319-10602-1_43

Titel: Gesture- and vision-based automatic grasping and flexible placement in teleoperation
verfasst von: Zhao Xue
Xiaoan Chen
Ye He
Hongli Cao
Shengli Tian
Publikationsdatum: 02.02.2022
Verlag: Springer London
Erschienen in: The International Journal of Advanced Manufacturing Technology / Ausgabe 1/2022
Print ISSN: 0268-3768
Elektronische ISSN: 1433-3015
DOI: https://doi.org/10.1007/s00170-021-08585-z

Premium Partner

Marktübersichten

Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.

Zur Marktübersicht

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 1/2022

Novel monitoring method for belt wear state based on machine vision and image processing under grinding parameter variation

A systematic method for automated manufacturability analysis of machining parts

Robotic simple and fast drilling system for automated aircraft assembly

An integrated scheduling method for personalized products with no-wait constraints

Investigation on mechanism of ultrasonic welding AZ31B/5052 joint with laser texturing on mental surface

Point cloud model simplification method for ring forgings based on single median error metric

Premium Partner

Marktübersichten