Skip to main content

Advertisement

SpringerLink
Log in

Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Artificial intelligence and computer vision have revolutionized laparoscopic surgical video analysis. However, there is no multi-center study focused on deep learning-based laparoscopic cholecystectomy phases recognizing. This work aims to apply artificial intelligence in recognizing and analyzing phases in laparoscopic cholecystectomy videos from multiple centers.

Methods

This observational cohort-study included 163 laparoscopic cholecystectomy videos collected from four medical centers. Videos were labeled by surgeons and a deep-learning model was developed based on 90 videos. Thereafter, the performance of the model was tested in additional ten videos by comparing it with the annotated ground truth of the surgeon. Deep-learning models were trained to identify laparoscopic cholecystectomy phases. The performance of models was measured using precision, recall, F1 score, and overall accuracy. With a high overall accuracy of the model, additional 63 videos as an analysis set were analyzed by the model to identify different phases.

Results

Mean concordance correlation coefficient for annotations of the surgeons across all operative phases was 92.38%. Also, the overall phase recognition accuracy of laparoscopic cholecystectomy by the model was 91.05%. In the analysis set, there was an average surgery time of 2195 ± 896 s, with a huge individual variance of different surgical phases. Notably, laparoscopic cholecystectomy in acute cholecystitis cases had prolonged overall durations, and the surgeon would spend more time in mobilizing the hepatocystic triangle phase.

Conclusion

A deep-learning model based on multiple centers data can identify phases of laparoscopic cholecystectomy with a high degree of accuracy. With continued refinements, artificial intelligence could be utilized in huge data surgery analysis to achieve clinically relevant future applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Hashimoto DA, Rosman G, Rus D, Meireles OR (2018) Artificial intelligence in surgery: promises and perils. Ann Surg 268:70–76

    Article  Google Scholar 

  2. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118

    Article  CAS  Google Scholar 

  3. Zappella L, Bejar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17:732–745

    Article  Google Scholar 

  4. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36:86–97

    Article  Google Scholar 

  5. Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW, Lillemoe KD, Rus DL, Meireles OR (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 270:414–421

    Article  Google Scholar 

  6. Morita S, Tabuchi H, Masumoto H, Yamauchi T, Kamiura N (2019) Real-time extraction of important surgical phases in cataract surgery videos. Sci Rep 9:16590

    Article  Google Scholar 

  7. Dergachyova O, Bouget D, Huaulme A, Morandi X, Jannin P (2016) Automatic data-driven real-time segmentation and recognition of surgical workflow. Int J Comput Assist Radiol Surg 11:1081–1089

    Article  Google Scholar 

  8. Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu CW, Heng PA (2018) SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37:1114–1126

    Article  Google Scholar 

  9. Wakabayashi G, Iwashita Y, Hibi T, Takada T, Strasberg SM, Asbun HJ, Endo I, Umezawa A, Asai K, Suzuki K, Mori Y, Okamoto K, Pitt HA, Han HS, Hwang TL, Yoon YS, Yoon DS, Choi IS, Huang WS, Gimenez ME, Garden OJ, Gouma DJ, Belli G, Dervenis C, Jagannath P, Chan ACW, Lau WY, Liu KH, Su CH, Misawa T, Nakamura M, Horiguchi A, Tagaya N, Fujioka S, Higuchi R, Shikata S, Noguchi Y, Ukai T, Yokoe M, Cherqui D, Honda G, Sugioka A, de Santibanes E, Supe AN, Tokumura H, Kimura T, Yoshida M, Mayumi T, Kitano S, Inomata M, Hirata K, Sumiyama Y, Inui K, Yamamoto M (2018) Tokyo Guidelines 2018: surgical management of acute cholecystitis: safe steps in laparoscopic cholecystectomy for acute cholecystitis (with videos). J Hepatobiliary Pancreat Sci 25:73–86

    Article  Google Scholar 

  10. Iwashita Y, Ohyama T, Honda G, Hibi T, Yoshida M, Miura F, Takada T, Han HS, Hwang TL, Shinya S, Suzuki K, Umezawa A, Yoon YS, Choi IS, Huang WS, Chen KH, Watanabe M, Abe Y, Misawa T, Nagakawa Y, Yoon DS, Jang JY, Yu HC, Ahn KS, Kim SC, Song IS, Kim JH, Yun SS, Choi SH, Jan YY, Sheen-Chen SM, Shan YS, Ker CG, Chan DC, Lee KT, Toyota N, Higuchi R, Nakamura Y, Mizuguchi Y, Takeda Y, Ito M, Norimizu S, Yamada S, Matsumura N, Shindoh J, Sunagawa H, Hasegawa H, Rikiyama T, Sata N, Kano N, Kitano S, Tokumura H, Yamashita Y, Watanabe G, Nakagawa K, Kimura T, Yamakawa T, Wakabayashi G, Endo I, Miyazaki M, Yamamoto M (2016) What are the appropriate indicators of surgical difficulty during laparoscopic cholecystectomy? Results from a Japan–Korea–Taiwan multinational survey. J Hepatobiliary Pancreat Sci 23:533–547

    Article  Google Scholar 

  11. Kitaguchi D, Takeshita N, Matsuzaki H, Takano H, Owada Y, Enomoto T, Oda T, Miura H, Yamanashi T, Watanabe M, Sato D, Sugomori Y, Hara S, Ito M (2019) Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc 34:4924

    Article  Google Scholar 

  12. Yokoe M, Hata J, Takada T, Strasberg SM, Asbun HJ, Wakabayashi G, Kozaka K, Endo I, Deziel DJ, Miura F, Okamoto K, Hwang TL, Huang WS, Ker CG, Chen MF, Han HS, Yoon YS, Choi IS, Yoon DS, Noguchi Y, Shikata S, Ukai T, Higuchi R, Gabata T, Mori Y, Iwashita Y, Hibi T, Jagannath P, Jonas E, Liau KH, Dervenis C, Gouma DJ, Cherqui D, Belli G, Garden OJ, Gimenez ME, de Santibanes E, Suzuki K, Umezawa A, Supe AN, Pitt HA, Singh H, Chan ACW, Lau WY, Teoh AYB, Honda G, Sugioka A, Asai K, Gomi H, Itoi T, Kiriyama S, Yoshida M, Mayumi T, Matsumura N, Tokumura H, Kitano S, Hirata K, Inui K, Sumiyama Y, Yamamoto M (2018) Tokyo Guidelines 2018: diagnostic criteria and severity grading of acute cholecystitis (with videos). J Hepatobiliary Pancreat Sci 25:41–54

    Article  Google Scholar 

  13. Meeuwsen FC, van Luyn F, Blikkendaal MD, Jansen FW, van den Dobbelsteen JJ (2019) Surgical phase modelling in minimal invasive surgery. Surg Endosc 33:1426–1432

    Article  CAS  Google Scholar 

  14. Kannan S, Yengera G, Mutter D, Marescaux J, Padoy N (2020) Future-state predicting LSTM for early surgery type recognition. IEEE Trans Med Imaging 39:556–566

    Article  Google Scholar 

  15. Gutt CN, Encke J, Koninger J, Harnoss JC, Weigand K, Kipfmuller K, Schunter O, Gotze T, Golling MT, Menges M, Klar E, Feilhauer K, Zoller WG, Ridwelski K, Ackmann S, Baron A, Schon MR, Seitz HK, Daniel D, Stremmel W, Buchler MW (2013) Acute cholecystitis: early versus delayed cholecystectomy, a multicenter randomized trial (ACDC study, NCT00447304). Ann Surg 258:385–393

    Article  Google Scholar 

  16. Strasberg SM, Brunt LM (2010) Rationale and use of the critical view of safety in laparoscopic cholecystectomy. J Am Coll Surg 211:132–138

    Article  Google Scholar 

  17. Strasberg SM (2019) A three-step conceptual roadmap for avoiding bile duct injury in laparoscopic cholecystectomy: an invited perspective review. J Hepatobiliary Pancreat Sci 26:123–127

    Article  Google Scholar 

  18. Yokoe M, Takada T, Strasberg SM, Solomkin JS, Mayumi T, Gomi H, Pitt HA, Gouma DJ, Garden OJ, Buchler MW, Kiriyama S, Kimura Y, Tsuyuguchi T, Itoi T, Yoshida M, Miura F, Yamashita Y, Okamoto K, Gabata T, Hata J, Higuchi R, Windsor JA, Bornman PC, Fan ST, Singh H, de Santibanes E, Kusachi S, Murata A, Chen XP, Jagannath P, Lee S, Padbury R, Chen MF, Tokyo Guidelines Revision C (2012) New diagnostic criteria and severity assessment of acute cholecystitis in revised Tokyo Guidelines. J Hepatobiliary Pancreat Sci 19:578–585

    Article  Google Scholar 

  19. Huang L, Li TJ (2018) Laparoscopic surgery for gastric cancer: where are we now and where are we going? Expert Rev Anticancer Ther 18:1145–1157

    Article  CAS  Google Scholar 

  20. Palter VN, Grantcharov TP (2012) A prospective study demonstrating the reliability and validity of two procedure-specific evaluation tools to assess operative competence in laparoscopic colorectal surgery. Surg Endosc 26:2489–2503

    Article  Google Scholar 

  21. De Siqueira JR, Gough MJ (2016) Correlation between experience targets and competence for general surgery certification. Br J Surg 103:921–927

    Article  Google Scholar 

  22. Langerman A, Grantcharov TP (2017) Are we ready for our close-up?: why and how we must embrace video in the OR. Ann Surg 266:934–936

    Article  Google Scholar 

  23. Hu YY, Mazer LM, Yule SJ, Arriaga AF, Greenberg CC, Lipsitz SR, Gawande AA, Smink DS (2017) Complementing operating room teaching with video-based coaching. JAMA Surg 152:318–325

    Article  Google Scholar 

  24. Korndorffer JR Jr, Dunne JB, Sierra R, Stefanidis D, Touchard CL, Scott DJ (2005) Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg 201:23–29

    Article  Google Scholar 

  25. Stefanidis D, Korndorffer JR Jr, Markley S, Sierra R, Heniford BT, Scott DJ (2007) Closing the gap in operative performance between novices and experts: does harder mean better for laparoscopic simulator training? J Am Coll Surg 205:307–313

    Article  Google Scholar 

  26. Curtis NJ, Foster JD, Miskovic D, Brown CSB, Hewett PJ, Abbott S, Hanna GB, Stevenson ARL, Francis NK (2020) Association of surgical skill assessment with clinical outcomes in cancer surgery. JAMA Surg 155:590

    Article  Google Scholar 

  27. Tang B, Hanna GB, Carter F, Adamson GD, Martindale JP, Cuschieri A (2006) Competence assessment of laparoscopic operative and cognitive skills: objective structured clinical examination (OSCE) or observational clinical human reliability assessment (OCHRA). World J Surg 30:527–534

    Article  CAS  Google Scholar 

  28. Foster JD, Miskovic D, Allison AS, Conti JA, Ockrim J, Cooper EJ, Hanna GB, Francis NK (2016) Application of objective clinical human reliability analysis (OCHRA) in assessment of technical performance in laparoscopic rectal cancer surgery. Tech Coloproctol 20:361–367

    Article  CAS  Google Scholar 

  29. Pugh CM, Hashimoto DA, Korndorffer JR Jr (2020) The what? How? And who? Of video based assessment. Am J Surg 221:13

    Article  Google Scholar 

  30. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ, Michigan Bariatric Surgery C (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We are thankful to the contribution that all of the authors have made.

Funding

This work is supported by key project of Health Commission of Sichuan Province, China (20ZD003).

Author information

Author notes

  1. Ke Cheng and Jiaying You are equal contribution to the work.

Authors and Affiliations

  1. West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China

    Ke Cheng, Jiaying You, Shangdi Wu, Zixin Chen, Zijian Zhou, Bing Peng & Xin Wang

  2. Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China

    Ke Cheng, Jiaying You, Shangdi Wu, Zixin Chen, Zijian Zhou, Bing Peng & Xin Wang

  3. ChengDu Withai Innovations Technology Company, Chengdu, China

    Jingye Guan

Authors
  1. Ke Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaying You
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shangdi Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zixin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zijian Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jingye Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bing Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Bing Peng or Xin Wang.

Ethics declarations

Disclosure

Dr. Ke Cheng, Dr. Jiaying You, Dr. Shangdi Wu, Dr. Zixin Chen, Dr. Zijian Zhou, Mr. Jingye Guan, Prof. Bing Peng and Dr. Xin Wang have no conflict of interest or financial ties to disclose. This work is supported by key project of Health Commission of Sichuan Province, China (20ZD003).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, K., You, J., Wu, S. et al. Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc 36, 3160–3168 (2022). https://doi.org/10.1007/s00464-021-08619-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-021-08619-3

Keywords

Search

Navigation