Skip to main content
SpringerLink
Log in

Review of methods for objective surgical skill evaluation

  • Review
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Rising health and financial costs associated with iatrogenic errors have drawn increasing attention to the dexterity of surgeons. With the advent of new technologies, such as robotic surgical systems and medical simulators, researchers now have the tools to analyze surgical motion with the goal of differentiating the level of technical skill in surgeons.

Methods

The review for this paper is obtained from a Google Scholar and PubMed search of the key words “objective surgical skill evaluation.” Only studies that included motion analysis were used.

Results

In this paper, we provide a clinical motivation for the importance of surgical skill evaluation. We review the current methods of tracking surgical motion and the available data-collection systems. We also survey current methods of surgical skill evaluation and show that most approaches fall into one of three methods: (1) structured human grading; (2) descriptive statistics; or (3) statistical language models of surgical motion. We discuss the need for an encompassing approach to model human skill through statistical models to allow for objective skill evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Reznick RK, MacRae H (2006) Teaching surgical skills-changes in the wind. N Engl J Med 355(25):2664–2669

    Article  CAS  PubMed  Google Scholar 

  2. DeFrances CJ, Hall MJ (2005) National hospital discharge survey. Adv Data Vital Health Stat 385:1–19

    Google Scholar 

  3. Anderson P, Gelijns A, Moskowitz A, Arons R, Gupta L, Weinberg A, Faries P, Nowygrod R, Kent K (2004) Understanding trends in inpatient surgical volume: vascular interventions, 1980–2000. J Vasc Surg 39(6):1200–1208

    Article  PubMed  Google Scholar 

  4. Zhan C, Miller M (2003) Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA 290(14):1868–1874

    Article  CAS  PubMed  Google Scholar 

  5. Peracchia A (2001) Surgical education in the third millennium. Ann Surg 234(6):709–712

    Article  CAS  PubMed  Google Scholar 

  6. Carter BN (1952) The fruition of Halsted’s concept of surgical training. Surgery 32:518–527

    CAS  PubMed  Google Scholar 

  7. Darzi A, Smith S, Taffinder N (1999) Assessing operative skill needs to become more objective. Br Med J 318:887–888

    CAS  Google Scholar 

  8. Barden CB, Specht MC, McCarter MD, Daly JM, Fahey TJ (2003) Effects of limited work hours on surgical training. Obstet Gynecol Surv 58(4):244–245

    Article  Google Scholar 

  9. Ericsson KA (2004) Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domain. Acad Med 79:70–81

    Article  Google Scholar 

  10. Bann SD, Datta VK, Khan MS, Ridgway PF, Darzi AW (2005) Attitudes towards skills examinations for basic surgical trainees. Int J Clin Pract 59(1):107–113

    Article  CAS  PubMed  Google Scholar 

  11. Kopta JA (1971) An approach to the evaluation of operative skills. Surgery 70:297–303

    CAS  PubMed  Google Scholar 

  12. Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165:358–361

    Article  CAS  PubMed  Google Scholar 

  13. Lin HC, Shafran I, Yuh DD, Hager GD (2006) Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comp Aid Surg 11(5):220–223

    Article  Google Scholar 

  14. Bridgewater B, Grayson AD, Jackson M, Brooks N, Grotte GJ, Keenan DJM, Millner R, Fabri BM, Jones M (2003) Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 327:13–17

    Article  PubMed  Google Scholar 

  15. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. Br Med Assoc 327:1032–1037

    Google Scholar 

  16. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W (1997) Testing technical skill via an innovative bench station examination. Am J Surg 173(3):226–230

    Article  CAS  PubMed  Google Scholar 

  17. Doyle J, Webber E, Sidhu R (2007) A universal global rating scale for the evaluation of technical skills in the operating room. Am J Surg 193(5):551–555

    Article  PubMed  Google Scholar 

  18. Neumann M, Hahn C, Horbach T, Schneide I, Meining A, Heldwein W, Rsch T (2003) Score card endoscopy: a multicenter study to evaluate learning curves in 1-week courses using the Erlangen endotrainer. Endoscopy 35:515–520

    Article  CAS  PubMed  Google Scholar 

  19. Moorthy K, Munz Y, Dosis A, Hernandez J, Martin S, Bello F, Rockall T, Darzi A (2004) Dexterity enhancement with robotic surgery. Surg Endosc 18:790–795

    CAS  PubMed  Google Scholar 

  20. Shah J, Darzi A (2001) Surgical skills assessment: an ongoing debate. BJU Int 88(7):655–660

    Article  CAS  PubMed  Google Scholar 

  21. Rosenbaum DA (1992) Reaching and grasping. Human Motor Control 197–225

  22. Chmarra MK, Grimbergen CA, Dankelman J (2007) Systems for tracking minimally invasive surgical instruments. Min Inv Ther All Tech 16(6):328–340

    Article  CAS  Google Scholar 

  23. Riedl S (2002) Modern operating room management in the workflow of surgery, spectrum of tasks and challenges of the future. Anaesthesist 73:105–110

    CAS  Google Scholar 

  24. Thompson CA (2004) Jcaho gears up to survey sterile compounding practices. Am J Health, Syst Pharm

    Google Scholar 

  25. Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B, Hoffman K (2004) Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 135:21–27

    Article  PubMed  Google Scholar 

  26. Figert PL, Park AE, Witzke DB, Schwartz RW (2001) Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 193(5):533–537

    Article  CAS  PubMed  Google Scholar 

  27. Moltenbrey K (2006) Tools of the trade. Comp Graph World 29(11)

  28. 5DT Data Glove Ulta

  29. Francis N (2002) The performance of master surgeons on the advanced Dundee endoscopic psychomotor tester. Arch Surg 137:841–844

    Article  PubMed  Google Scholar 

  30. Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in laboratory-based model. J Am Coll Surg 193:479–485

    Article  CAS  PubMed  Google Scholar 

  31. Taffinder N, Smith S, Mair J, Russel RC, Darzi A (1999) Can a computer measure surgical precision? Reliability, validity and feasibility of the ICSAD. Surg Endosc 13(1):81

    Article  Google Scholar 

  32. Xin H, Zelek JS, Carnahan H (2006) Laparoscopic surgery, perceptual limitations and force: a review. In: First joint eurohaptics conference and symposium on haptic interfaces for virtual environment

  33. Guthart GS, Salisbury JK (2000) The intuitive telesurgery system: overview and application. In: IEEE International conference on robotics automation, pp 618–621

  34. Intuitive Surgical Inc (2007) Intuitive surgical annual report

  35. Nawrat Z, Kostka P (2007) The robin heart vision, telemanipulator for camera holding––preliminary test results. J Automat Mobile Robotics Intelligent Syst 1(1):48–53

    Google Scholar 

  36. Mayer H, Gomex F, Wierstra D, Nagy I, Knoll A, Schmidhuber J (2006) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. ICRA, pp 543–548

  37. Global Strategic Business (2007) Computer assisted surgical systems. A global strategic business report

  38. Richards C, Rosen J, Hannaford B, Pellegrini C, Sinanan M (2000) Skills evaluation in minimally invasive surgery using force/torque signatures. Surg Endosc 14:791–798

    Article  CAS  PubMed  Google Scholar 

  39. Rosen J, Brown JD, Chang L, Sinanan MN, Hannaford B (2006) Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans Biomed Eng 53(3):399–413

    Article  PubMed  Google Scholar 

  40. Pugh CM, Youngblood P (2002) Development and validation of assessment measures for a newly developed physical examination simulator. J Am Med Inform Assoc 9(5):448–460

    Article  PubMed  Google Scholar 

  41. Ko S, Kwon D (2004) A surgical knowledge based interaction method for a laparoscopic assistant robot. In: RO-MAN––IEEE, International workshop on robot and human interactive communication, pp 313–318

  42. Ibbotson JA, MacKenzie CL, Cao CG, Lomax AJ (1999) Gaze patterns in laparoscopic surgery. In: MMVR, pp 154–160

  43. Rosen J, Hannaford B, Richards CG, Sinanan MN (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5):579–591

    Article  CAS  PubMed  Google Scholar 

  44. Yamauchi Y, Yamashita J, Morikawa O, Hashimoto R (2002) Surgical skill evaluation by force data for endoscopic sinus surgery training system. In: MICCAI, pp 44–51

  45. Judkins TN, Oleynikov D, Stergiou N (2008) Objective evaluation of expert performance during human robotic surgical procedures. J Robotics Surg 1(4):307–312

    Article  Google Scholar 

  46. Van Sickle KR, McClusky DA, Gallagher AG, Smith CD (2005) Construct validation of the promise simulator using a novel laparoscopic suturing task. Surg Endosc 19(9):1227–1231

    Article  PubMed  Google Scholar 

  47. Acosta E, Temkin B (2005) Haptic laparoscopic skills trainer with practical user evaluation metrics. MMVR 111:8–11

    Google Scholar 

  48. Hernandez JD, Bann SD, Munz Y, Moorthy K, Datta V (2004) Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surg Endosc 372–378

  49. Rosen J, Solazzo M, Hannaford B, Sinanan MN (2002) Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden Markov model. Comp Aid Surg 7(1):49–61

    Article  Google Scholar 

  50. Cao CG, MacKenzie RD, Payandeh S (1996) Task and motion analyses in endoscopic surgery. In: ASME IMECE, pp 583–590

  51. Padoy N, Blum T, Essa IA, Feussne H, Berger MO, Navab N (2007) A boosted segmentation method for surgical workflow analysis. In: MICCAI, pp 102–109

  52. Datta V, Bann S, Hernandez J, Darzi A (2007) Objective assessment comparing hand-assisted and conventional laparoscopic surgery. Surg Endosc 21:414–417

    Article  CAS  PubMed  Google Scholar 

  53. Datta V, Mandalia M, Mackay S, Chang A, Cheshire N, Darzi A (2002) Relationship between skill and outcome in the laboratory based model. Surgery 131:318–323

    Article  PubMed  Google Scholar 

  54. Fried GM, Feldman LS, Vassiliou MC, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528

    Article  PubMed  Google Scholar 

  55. Acosta E, Temkin B (2005) Dynamic generation of surgery specific simulators: a feasibility study. StudHealth Technol Inform 111:1–7

    Google Scholar 

  56. Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2002) Assessment of technical surgical skills. Eur J Surg 168:139–144

    Article  PubMed  Google Scholar 

  57. Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A (2004) Laparoscopic virtual reality and box trainers: is one superior to the other? Surg Endosc 18:485–494

    Article  CAS  PubMed  Google Scholar 

  58. Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P, Rattner D, Dawson S (2002) Metrics for laparoscopic skills trainers: the weakest link. MICCAI 2488:35–43

    Google Scholar 

  59. O’Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, Bell WJ, Raibert M (1999) Measuring and developing suturing technique with a VR surgical simulator. Am Coll Surg 189(1):114–127

    Article  Google Scholar 

  60. Liu A, Tendick F, Cleary K, Kaufmann C (2003) A survey of surgical simulation: applications, technology, and education. Presence Teleop Virt Environ 12(6):599–614

    Article  Google Scholar 

  61. Issenberg SB, McGaghie WC (2002) Clinical skills training practice makes perfect. Med Educat 36(3):210–211

    Article  Google Scholar 

  62. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241(2):364–372

    Article  PubMed  Google Scholar 

  63. Seymour NE, Anthony G, Sanziana A, O’Brien MK, Bansal VK, Andersen DK (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458–464

    Article  PubMed  Google Scholar 

  64. Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on mist VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 81:281286

    Google Scholar 

  65. Nagy I, Mayer H, Knoll A (2003) The Endopar system for minimally invasive robotic surgery. Technical report, TUM

  66. MacKenzie CL, Ibbotson JA, Cao CLGL, Lomax AJ (2001) Hierarchical decomposition of laparoscopic surgery: a human factors approach to investigating the operating room environment. Minim Invas Ther Allied Technol 10(3):121–127

    Article  Google Scholar 

  67. Ahmadi SA, Sielhorst T, Stauder R, Horn M, Feussner H, Navab N (2006) Recovery of surgical workflow without explicit models. Med Imag Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv 9(1):420–428

    Google Scholar 

  68. Mackel T, Rosen J, Pugh C (2007) Application of hidden Markov modeling to objective medical skill evaluation. Stud Health Technol Inform 125:316–318

    PubMed  Google Scholar 

  69. Leong JH, Nicolaou M, Atallah L, Mylonas GP, Darzi AW, Yang G (2006) Hmm assessment of quality of movement trajectory in laparoscopic surgery. MICCAI 4190:752–759

    Google Scholar 

  70. Speidel S, Delles ML, Gutt C, Dillmann R (2006) Tracking of instruments in minimally invasive surgery for surgical skill analysis. Med Imag Augment Real 4091:148–155

    Article  Google Scholar 

  71. Hundtofte CS, Hager GD, Okamura AM (2002) Building a task language for segmentation and recognition of user input to cooperative manipulation systems. Haptic Interf Virt Environ Teleop Syst, pp 225–230

  72. Murphy TE, Vignes CM, Yuh DD, Okamura AM (2003) Automatic motion recognition and skill evaluation for dynamic tasks. Eurohaptics 363–373

  73. Reiley CE, Lin HC, Varadarajan B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Med Virt Real 132:396–401

    Google Scholar 

  74. Castellani A, Botturi D, Bicego M, Fiorini P (2004) Hybrid hmm/svm model for the analysis and segmentation of teleoperation tasks. IEEE Int Conf Robotics Autom 3:2918–2923

    Google Scholar 

  75. Li M, Okamura AM (2003) Recognition of operator motions for realtime assistance using virtual fixtures. Int Sympos Haptic Interf Virt Environ Teleop Syst 1954–1959

  76. Kragic D, Marayong P, Li M, Okamura AM, Hager GD (2005) Human-machine collaborative systems for microsurgical applications. Int J Robotics Res 24(9):731–741

    Article  Google Scholar 

  77. Intrafocus. http://www.intra-focus.com

  78. Okamura AM (2004) Methods for haptic feedback in teleoperated robot-assisted surgery. Industr Robot Int J 31:499–508

    Article  Google Scholar 

Download references

Disclosures

The authors, Carol Reiley, Henry Lin, and Dr. Gregory Hager, own some Intuitive Surgical stock in their individual, managed, or retirement accounts less than $10,000. Dr. David Yuh has no conflict of interest or financial ties to disclose. Our access to Intuitive Surgical da Vinci API is governed by the terms of a Collaborative agreement that does not include any financial support for this project. Supported by grants from the National Science Foundation under Grant No. 0534359, 0941362, 0931805; National Institute of Health R21 1R21EB009143-01A1; and a National Science Foundation Graduate Research Fellowship.

Author information

Authors and Affiliations

  1. Laboratory of Computational Sensing and Robotics Department of Computer Science, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA

    Carol E. Reiley, Henry C. Lin & Gregory D. Hager

  2. Program for Outcomes Research, Department of Surgery, Johns Hopkins School of Medicine, 600 North Wolfe Street, Baltimore, MD, 21287, USA

    David D. Yuh

  3. Division of Cardiac Surgery, Johns Hopkins Medical Institutions, 600 North Wolfe Street, Baltimore, MD, 21287, USA

    David D. Yuh

Authors
  1. Carol E. Reiley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henry C. Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David D. Yuh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gregory D. Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carol E. Reiley.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reiley, C.E., Lin, H.C., Yuh, D.D. et al. Review of methods for objective surgical skill evaluation. Surg Endosc 25, 356–366 (2011). https://doi.org/10.1007/s00464-010-1190-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-010-1190-z

Keywords

Search

Navigation