Skip to main content
SpringerLink
Log in

Towards Human Activity Recognition Enhanced Robot Assisted Surgery

  • Chapter
  • First Online:
Robot Design

Part of the book series: Mechanisms and Machine Science ((Mechan. Machine Science,volume 123))

Abstract

Medical robotics have drawn increasing research interests in the past years. The potential of employing intelligence techniques is still not yet fully utilized to improve the capabilities of medical devices for assisting human beings during surgical operations. This book chapter presents a novel human activity recognition enhanced robot-assisted surgery to promote human enhancement with AI techniques. Novel-designed multisensory fusion systems can be used to provide more knowledge to boost the recognition rate and robustness. Then the identification results in the complex environment using deep learning can be used to determine the machine behavior using a hierarchical control framework. A detailed explanation is introduced in this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kwoh, Y.S., Hou, J., Jonckheere, E.A., Hayati, S.: A robot with improved absolute positioning accuracy for CT guided stereotactic brain surgery. IEEE Trans. Biomed. Eng. 35(2), 153–160 (1988)

    Article  Google Scholar 

  2. Buettner, R., Alena, R., Boos, A.: A systematic literature review of research in the surgical field of medical robotics. In: IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, pp. 517–522 (2020)

    Google Scholar 

  3. Ren, Y., Wu, Y.N., Yang, C.Y., Xu, T., Harvey, R.L., Zhang, L.Q.: Developing a wearable ankle rehabilitation robotic device for in-bed acute stroke rehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 25(6), 178–185 (2017)

    Article  Google Scholar 

  4. Okamura, M., Mataric, M.J., Christensen, H.I.: Medical and health-care robotics. IEEE Robot. Autom. Mag. 17(2), 26–37 (2010)

    Article  Google Scholar 

  5. Bosecker, C.J., Krebs, H.I.: “MIT-Skywalker”. In: IEEE International Conference on Rehabilitation Robotics, pp. 542–549 (2009)

    Google Scholar 

  6. Jezernik, S., Colombo, G., Keller, T., et al.: Robotic orthosis lokomat: a rehabilitation and research tool. Neuromodul. Technol. Neural Interf. 6(2), 108 (2003)

    Article  Google Scholar 

  7. Sawicki, G., Ferris, D.: A pneumatically powered knee-ankle-foot orthosis (KAFO) with myoelectric activation and inhibition. J. Neuroeng. Rehabil. 6(1), 23 (2009)

    Article  Google Scholar 

  8. Esquenazi, A., Talaty, M., Packel, A., et al.: The ReWalk powered exoskeleton to restore ambulatory function to individuals with thoracic-level motor-complete spinal cord injury. Am. J. Phys. Med. Rehabil. 91(11), 911–920 (2012)

    Article  Google Scholar 

  9. Sankai, Y.: HAL: hybrid assistive limb based on cybernics. Robot. Res. 25–34 (2010)

    Google Scholar 

  10. Khan, Z.H., Afifa, S., Chang, W.L.: Robotics utilization for healthcare digitization in global COVID-19 management. Int. J. Environ. Res. Publ. Health 17(11), 1–23 (2020)

    Article  Google Scholar 

  11. Omisore, O.M., Han, S., Xiong, J., Li, H., Li, Z., Wang, L.: A review on flexible robotic systems for minimally invasive surgery. IEEE Trans Syst Man Cybernet Syst. https://doi.org/10.1109/TSMC.2020.3026174 (2020)

  12. Wilson, T.G.: Advancement of technology and its impact on urologists: release of the da Vinci xi, a new surgical robot. Eur. Urol. 66(5), 793–794 (2014)

    Article  Google Scholar 

  13. Peters, B.S., Armijo, P.R., Krause, C., Choudhury, S.A., Oleynikov, D.: Review of emerging surgical robotic technology. Surg. Endosc. 32, 1636–1655 (2018)

    Article  Google Scholar 

  14. Su, H., Enayati, N., Vantadori, L., Spinoglio, A., Ferrigno, G., De Momi, E.: Online human-like redundancy optimization for tele-operated anthropomorphic manipulators. Int. J. Adv. Rob. Syst. 15(6), 1729881418814695 (2019)

    Google Scholar 

  15. Zhou, J., Ma, X., Xu, Z., Qi, Z.: Overview of medical robot technology development. In: 37th Chinese Control Conference (CCC), Wuhan, China, pp. 5169–5174 (2018)

    Google Scholar 

  16. Smith, J.R., Riches, P.E., Rowe, P.J.: Accuracy of a freehand sculpting tool for unicondylar knee replacement. Int. J. Med. Robot. Comp. Assisted Surg. 10(2), 162–169 (2014)

    Article  Google Scholar 

  17. Meister, D., Pokrandt, P., Both, A.: Milling accuracy in robot-assisted orthopaedic surgery. In: Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society, vol. 4, pp. 2502–2505 (2018)

    Google Scholar 

  18. Weisz, G., Metzger, D.C., Caputo, R.P., Delgado, J.A., Marshall, J.J., Vetrovec, G.W., Reisman, M., Waksman, R., Granada, J.F., Novack, V., Moses, J.W., Carrozza, J.P.: Safety and feasibility of robotic percutaneous coronary intervention: PRECISE (percutaneous robotically enhanced coronary intervention) study. J. Am. Coll. Cardiol. 61(15), 1596–1600 (2013)

    Article  Google Scholar 

  19. Singh, A.V., Ansari, M.H.D., Laux, P., Luch, A.: Micro-nanorobots: important considerations when developing novel drug delivery platforms. Expert Opin. Drug Deliv. 16(11), 1259–1275 (2019)

    Article  Google Scholar 

  20. Chałupniak, A., Morales-Narváez, E., Merkoçi, A.: Micro and nanomotors in diagnostics. Adv. Drug Deliv. Rev. 95, 104–116 (2015)

    Article  Google Scholar 

  21. Ullrich, F., Bergeles, C., Pokki, J., Ergeneman, O., Erni, S., Chatzipirpiridis, G., Pané, S., Framme, C., Nelson, B.J.: Mobility experiments with microrobots for minimally invasive intraocular surgery. Invest. Ophthalmol. Vis. Sci. 54(4), 2853–2863 (2013)

    Article  Google Scholar 

  22. Yang, J., Zhang, C., Wang, X., Wang, W., Xi, N., Liu, L.: Development of micro-and nanorobotics: a review. Sci. China Technol. Sci. 62(1), 1–20 (2019)

    Google Scholar 

  23. Medina-Sánchez, M., Xu, H., Schmidt, O.G.: Micro-and nano-motors: the new generation of drug carriers. Ther. Deliv. 9(4), 303–316 (2018)

    Article  Google Scholar 

  24. Su, H., Schmirander, Y., Li, Z., Zhou, X., Ferrigno, G., De Momi, E.: Bilateral teleoperation control of a redundant manipulator with an RCM kinematic constraint. In: 2020 IEEE International Conference on Robotics and Automation (ICRA) (2020)

    Google Scholar 

  25. Mukai, T., Hirano, S., Nakashima, H., Kato, Y., Sakaida, Y., Guo, S., Hosoe, S.: Development of a nursing-care assistant robot RIBA that can lift a human in its arms. IEEE/RSJ Int. Conf. Intell. Robots Syst. 5996–6001 (2010)

    Google Scholar 

  26. Gomes, P.: Surgical robotics: reviewing the past, analyzing the present, imagining the future. Robot. Comp. Integr. Manuf. 27(2), 261–266 (2011)

    Article  Google Scholar 

  27. Allan, M., Ourselin, S., Hawkes, D.J., Kelly, J.D., Stoyanov, D.V.: 3-D pose estimation of articulated instruments in robotic minimally invasive surgery. IEEE Trans. Med. Imaging 37(5), 1204–1213 (2018)

    Google Scholar 

  28. Hagn, U., Konietschke, R., Tobergte, A., Nickl, M., Jörg, S., Kübler, B., Passig, G., Gröger, M., Fröhlich, F., Seibold, U., Le-Tien, L., Albu-Schäffer, A., Nothhelfer, A., Hacker, F., Grebenstein, M., Hirzinger, G.: DLR MiroSurge: a versatile system for research in endoscopic telesurgery. Int. J. Comput. Assist. Radiol. Surg. 5(2), 183–193 (2010)

    Article  Google Scholar 

  29. Troccaz, J., Hagn, U., Nickl, M., Jörg, S., Passig, G., Bahls, T., Nothhelfer, A., Hacker, F., Le-Tien, L., Albu-Schäffer, A., Konietschke, R., Grebenstein, M., Warpup, R., Haslinger, R., Frommberger, M., Hirzinger, G.: The DLR MIRO: a versatile lightweight robot for surgical applications. Ind. Robot. 35, 324–336 (2008)

    Article  Google Scholar 

  30. Sandoval, J., Su, H., Vieyres, P., Poisson, G., Ferrigno, G., De Momi, E.: Collaborative framework for robot-assisted minimally invasive surgery using a 7-dof anthropomorphic robot. Robot. Auton. Syst. 106, 95–106 (2018)

    Article  Google Scholar 

  31. Su, H., Enayati, N., Vantadori, L., Spinoglio, A., Ferrigno, G., De Momi, E.: Online human-like redundancy optimization for tele-operated anthropomorphic manipulators. Int. J. Adv. Robot. Syst. 15(6) (2018b)

    Google Scholar 

  32. Su, H., Yang, C., Ferrigno, G., De Momi, E.: Improved human-robot collaborative control of redundant robot for teleoperated minimally invasive surgery. IEEE Robot. Autom. Lett. 4(2), 1447–1453 (2019)

    Article  Google Scholar 

  33. Su, H., Qi, W., Yang, C., Aliverti, A., Ferrigno, G., De Momi, E.: Deep neural network approach in human-like redundancy optimization for anthropomorphic manipulators. IEEE Access 7, 124207–124216 (2019)

    Article  Google Scholar 

  34. Su, H., Hu, Y., Li, Z., Knoll, A., Ferrigno, G., De Momi, E.: Reinforcement learning based manipulation skill transferring for robot-assisted minimally invasive surgery. In: Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA) (2020b)

    Google Scholar 

  35. Su, H., Schmirander, Y., Li, Z., Zhou, X., Ferrigno, G., De Momi, E.: Bilateral teleoperation control of a redundant manipulator with an RCM kinematic constraint. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 4477–4482 (2020c)

    Google Scholar 

  36. Su, H., Qi, W., Schmirander, Y., Ovur, S. E., Cai, S., & Xiong, X.: A human activity-aware shared control solution for medical human–robot interaction. Assembly Autom. 42(3), 88–394 (2022)

    Google Scholar 

  37. Enayati, N., De Momi, E., Ferrigno, G.: A quaternion-based unscented Kalman filter for robust optical/inertial motion tracking in computer-assisted surgery. IEEE Trans. Instrum. Meas. 64(8), 2291–2301 (2015)

    Article  Google Scholar 

  38. Qian, Y., Qiao, H., Xu, J.: Understanding deep neural network by filter sensitive area generation network. In: International Conference on Neural Information Processing, pp. 192–203, Springer, Cham (2018)

    Google Scholar 

  39. Huang, X., Nie, X., Qiao, H., Zhang, B.: Supervised Polsar image classification by combining multiple features. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 634–638. https://doi.org/10.1109/ICIP.2019.8803812 (2019)

  40. Qi, W., Liu, X., Zhang, L., Wu, L., Zang, W., Su, H.: Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation. Assem. Autom. (2018)

    Google Scholar 

  41. Qi, W., Su, H.: A cybertwin based multimodal network for ecg patterns monitoring using deep learning. IEEE Trans. Ind. Informat. 18(10), 663–6670 (2022)

    Google Scholar 

  42. Haidegger, T., Autonomy for surgical robots: concepts and paradigms. IEEE Trans. Med. Robot. Bionics. 1(2), 65–76 (2019)

    Google Scholar 

  43. Li, Z., Liu, J., Huang, Z., Peng, Y., Pu, H., Ding, L.: Adaptive impedance controlof human-robot cooperation using reinforcement learning. IEEE Trans. Industr. Electron. 64, 8013–8022 (2017)

    Article  Google Scholar 

  44. Marescaux, J., Leroy, J., Rubino, F., Vix, M., Simone, M., Mutter, D.: Transcontinental robot assisted remote telesurgery: feasibility and potential applications. Ann. Surg. 235, 487–492 (2002)

    Article  Google Scholar 

  45. Sheridan, T.B.: Teleoperation, telerobotics and telepresence: a progress report. Control. Eng. Pract. 3(2), 205–214 (1995)

    Article  Google Scholar 

  46. Puangmali, P., Althoefer, K., Seneviratne, L.D., Murphy, D., Dasgupta, P.: State-of-the-art in force and tactile sensing for minimally invasive surgery. IEEE Sens. J. 8(4), 371–380 (2008)

    Article  Google Scholar 

  47. Su, H., Qi, W., Yang, C., Sandoval, J., Ferrigno, G., De Momi, E.: Deep neural network approach in robot tool dynamics identification for bilateral teleoperation. IEEE Robot. Autom. Lett. 5(2), 2943–2949 (2020)

    Article  Google Scholar 

  48. Li, P., Yang, Z., Jiang, S.: Needle-tissue interactive mechanism and steering control in image-guided robot-assisted minimally invasive surgery: a review. Med. Biol. Eng. Comp. 56, 931–949 (2018)

    Article  Google Scholar 

  49. Stoll, J., Novotny, P., Howe, R., Dupont, P.: Real-time 3D ultrasoundbased servoing of a surgical instrument. In: Proceedings of 2006 IEEE International Conference on Robotics and Automation (ICRA), pp. 613–618 (2006)

    Google Scholar 

  50. Ren, H., Kazanzides, P.: Investigation of attitude tracking using an integrated inertial and magnetic navigation system for hand-held surgical instruments. IEEE/ASME Trans. Mechatron. 7(2), 210–217 (2012)

    Article  Google Scholar 

  51. Ren, H., Rank, D., Merdes, M., Stallkamp, J., Kazanzides, P.: Multisensor data fusion in an integrated tracking system for endoscopic surgery. IEEE Trans. Inf. Technol. Biomed. 16(1), 106–111 (2012)

    Article  Google Scholar 

  52. Qi, W., Su, H., Yang, C., Ferrigno, G., De Momi, E., Aliverti, A.: A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone. Sensors 19(17), 3731 (2019)

    Article  Google Scholar 

  53. Qi, W., Su, H., Aliverti, A.: A smartphone-based adaptive recognition and real-time monitoring system for human activities. IEEE Trans. Human Mach. Syst. 50(5), 414–423 (2020)

    Article  Google Scholar 

  54. Azizian, M., Khoshnam, M., Najmaei, N., Patel, R.V.: Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging—techniques and applications. Int. J. Med. Robot. Comp. Assisted Surg. 10(3), 263–274 (2014)

    Article  Google Scholar 

  55. Aghajani Pedram, S., Ferguson, P., Ma, J., Dutson, E., Rosen, J.: Autonomous suturing via surgical robot: an algorithm for optimal selection of needle diameter, shape, and path. In: Proceedings of 2017 International Conference on Robotics and Automation (ICRA) (2017)

    Google Scholar 

  56. Zhong, F., Wang, Y., Wang, Z., Liu, Y.-H.: Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing. IEEE Robot. Autom. Lett. 4(3), 2669–2676 (2019)

    Article  Google Scholar 

  57. Zhang, J., Li, P., Jin, C.-C., Zhang, W.-A., Liu, S.: A novel adaptive Kalman filtering approach to human motion tracking with magnetic-inertial sensors. IEEE Trans. Industr. Electron. 67(10), 8659–8669 (2019)

    Article  Google Scholar 

  58. Wang, J., Song, S., Ren, H., Lim, C.M., Meng Max, Q.-H.: Surgical instrument tracking by multiple monocular modules and a sensor fusion approach. IEEE Trans. Autom. Sci. Eng. 16(2), 629–639 (2018)

    Article  Google Scholar 

  59. Zhang, J., Li, P., Zhu, T., Zhang, W.-A., Liu, S.: Human motion capture based on kinect and IMUs and its application to human-robot collaboration. In: Proceedings of the 2020 5th International Conference on Advanced Robotics and Mechatronics (ICARM) (2020)

    Google Scholar 

  60. Su, H., Qi, W., Chen, J., Zhang, D.: Fuzzy Approximation-based Task-Space Control of Robot Manipulators with Remote Center of Motion Constraint. IEEE Trans. Fuzzy Syst. 30(6), 1564–1573 (2022)

    Google Scholar 

  61. Luo, J., Yang, C., Dai, S., Liu, Z.: Tremor attenuation for surgical robots using support vector machine with parameters optimization. In: 2018 Tenth International Conference on Advanced Computational Intelligence (ICACI). https://doi.org/10.1109/ICACI.2018.8377539(2018)

  62. Volkov, M., Hashimoto, D.A., Rosman, G., Meireles, O.R., Rus, D.: Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery. IEEE Int. Conf. Robot. Autom. (ICRA) (2017). https://doi.org/10.1109/ICRA.2017.7989093

    Article  Google Scholar 

  63. Fang, J., Sun, Q., Chen, Y., Tang, Y.: Quadrotor navigation in dynamic environments with deep reinforcement learning. Assem. Autom. (2021)

    Google Scholar 

  64. Shvets, A.A., Rakhlin, A., Kalinin, A.A., Iglovikov, V.I.: Automatic instrument segmentation in robot-assisted surgery using deep learning. In: 17th IEEE International Conference on Machine Learning and Applications (ICMLA) (2018)

    Google Scholar 

  65. Su, H., Hu, Y., Karimi, H.R., Knoll, A., Ferrigno, G., De Momi, E.: Improved recurrent neural network-based manipulator control with remote center of motion constraints: experimental results. Neural Netw. 131, 291–299 (2020)

    Article  MATH  Google Scholar 

  66. Yang, G.-Z., Cambias, J., Cleary, K., Daimler, E., Drake, J., Dupont, P.E., Hata, N., Kazanzides, P., Martel, S., Patel, R.V., Santos, V.J., Taylor, R.H.: Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2(4), eaam8638 (2017)

    Google Scholar 

  67. Su, H., Ovur, S.E., Li, Z., Hu, Y., Li, J., Knoll, A., Ferrigno, G., De Momi, E.: Internet of things (IoT)-based collaborative control of a redundant manipulator for teleoperated minimally invasive surgeries. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 9737–9742 (2020f)

    Google Scholar 

  68. Li, T., Chang, X., Wu, Z., Li, J., Shao, G., Deng, X., Qiu, J., Guo, B., Zhang, G., He, Q., Li, L., Wang, J.: Autonomous collision-free navigation of microvehicles in complex and dynamically changing environments. ACS Nano 11(9), 9268–9275 (2017)

    Article  Google Scholar 

  69. Morgan, P.S., Carter, T., Davis, S., Sepehri, A., Punt, J., Byrne, P., Moody, A., Finlay, P.: The application accuracy of the Pathfinder neurosurgical robot. Int. Congr. Ser. 1256, 561–567 (2003)

    Article  Google Scholar 

  70. Zhong, S., Zhou, J., Qiao, H.: Bioinspired gain-modulated recurrent neural network for controlling musculoskeletal robot. IEEE Trans. Neural Netw. Learn. Syst. (2021)

    Google Scholar 

  71. Huang, X., Yang, Q., Qiao, H.: Lightweight two-stream convolutional neural network for SAR target recognition. IEEE Geosci. Remote Sens. Lett. 18(4), 667–671 (2021). https://doi.org/10.1109/LGRS.2020.2983718

    Article  Google Scholar 

  72. Su, H., Yang, C., Ferrigno, G., De Momi, E.: Improved human-robot collaborative control of redundant robot for teleoperated minimally invasive surgery. IEEE Robot. Autom. Lett. 4, 1447–1453 (2019)

    Article  Google Scholar 

  73. Su, H., Sandoval, J., Vieyres, P., Poisson, G., Ferrigno, G., De Momi, E.: Safety-enhanced collaborative framework for tele-operated minimally invasive surgery using a 7-DoF torque-controlled robot. Int. J. Control Autom. Syst. 16(6), 2915–2923 (2019)

    Article  Google Scholar 

  74. Nicolis, D.: Human intention estimation based on neural networks for enhanced collaboration with robots. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Madrid, Spain, October 1–5 (2018)

    Google Scholar 

  75. Sin, Y.J., Chen, Y.L., Ko, T.H., Kai, T.S.: Shared control design of a walking-assistant robot. IEEE Trans. Control Syst. Technol. 25(6), 2143–2150 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Building 32.2, Via Giuseppe Colombo, 40 20133, Milano, Italy

    Hang Su & Elena De Momi

Authors
  1. Hang Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elena De Momi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hang Su .

Editor information

Editors and Affiliations

  1. DIMEG, University of Calabria, Arcavacata di Rende, Italy

    Giuseppe Carbone

  2. SP2MI - Site du Futuroscope, University of Poitiers, Poitiers, France

    Med Amine Laribi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Su, H., De Momi, E. (2023). Towards Human Activity Recognition Enhanced Robot Assisted Surgery. In: Carbone, G., Laribi, M.A. (eds) Robot Design. Mechanisms and Machine Science, vol 123. Springer, Cham. https://doi.org/10.1007/978-3-031-11128-0_7

Download citation

Publish with us

Policies and ethics

Search

Navigation