nach oben

International Journal of Computer Assisted Radiology and Surgery

Erschienen in:

16.11.2018 | Original Article

“Deep-Onto” network for surgical workflow and context recognition

verfasst von: Hirenkumar Nakawala, Roberto Bianchi, Laura Erica Pescatori, Ottavio De Cobelli, Giancarlo Ferrigno, Elena De Momi

Erschienen in: International Journal of Computer Assisted Radiology and Surgery | Ausgabe 4/2019

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Purpose

Surgical workflow recognition and context-aware systems could allow better decision making and surgical planning by providing the focused information, which may eventually enhance surgical outcomes. While current developments in computer-assisted surgical systems are mostly focused on recognizing surgical phases, they lack recognition of surgical workflow sequence and other contextual element, e.g., “Instruments.” Our study proposes a hybrid approach, i.e., using deep learning and knowledge representation, to facilitate recognition of the surgical workflow.

Methods

We implemented “Deep-Onto” network, which is an ensemble of deep learning models and knowledge management tools, ontology and production rules. As a prototypical scenario, we chose robot-assisted partial nephrectomy (RAPN). We annotated RAPN videos with surgical entities, e.g., “Step” and so forth. We performed different experiments, including the inter-subject variability, to recognize surgical steps. The corresponding subsequent steps along with other surgical contexts, i.e., “Actions,” “Phase” and “Instruments,” were also recognized.

Results

The system was able to recognize 10 RAPN steps with the prevalence-weighted macro-average (PWMA) recall of 0.83, PWMA precision of 0.74, PWMA F1 score of 0.76, and the accuracy of 74.29% on 9 videos of RAPN.

Conclusion

We found that the combined use of deep learning and knowledge representation techniques is a promising approach for the multi-level recognition of RAPN surgical workflow.

Vorheriger Artikel A novel robotic system for vascular intervention: principles, performances, and applications

Nächster Artikel A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Available at https://doi.org/10.5281/zenodo.1066831.

Blum T, Feussner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: MICCAI international conference on medical image computing and computer-assisted-intervention, 2010. MICCAI 2010, vol 13(3), pp 400–407. https://doi.org/10.1007/978-3-642-15711-0_50

Khurshid AG, Esfahani ET, Raza SJ, Bhat R, Wang K, Hammond Y, Wilding G, Peabody JO, Chowriappa AJ (2015) Cognitive skills assessment during robot-assisted surgery: separating the wheat from the chaff. BJUI 155(1):166–174. https://doi.org/10.1111/bju.12657 CrossRef

Flin R, Youngson G, Yule S (2007) How do surgeons make intraoperative decisions? Qual Saf Health Care 16(3):235–239. https://doi.org/10.1136/qshc.2006.020743 CrossRefPubMedPubMedCentral

Ferlay J, Steliarova-Foucher E, Lortet-Tieulent J, Rosso S, Coebergh JW, Comber H, Forman D, Bray F (2012) Cancer incidence and mortality patterns in Europe: estimates for 40 countries in 2012. Eur J Cancer 49(6):1374–403. https://doi.org/10.1016/j.ejca.2012.12.027 CrossRef

Abel EJ, Culp SH, Meissner M, Matin SF, Tamboli P, Wood CG (2010) Identifying the risk of disease progression after surgery for localized renal cell carcinoma. BJU Int 106(9):1227–83. https://doi.org/10.1111/j.1464-410x.2010.09337.x CrossRef

Hu JC, Treat E, Filson CP, McLaren I, Xiong S, Stepanian S, Hafez KS, Weizer AZ, Porter J (2014) Technique and outcomes of robot-assisted retroperitoneoscopic partial nephrectomy: a multicenter study. Eur J Urol 66:542–549. https://doi.org/10.1016/j.eururo.2014.04.028 CrossRef

Government of Alberta (2018) Robot-assisted partial nephrectomy for renal cell carcinoma: mini review. https://open.alberta.ca/dataset/0e172257-2820-4eba-9915-f0add1d14f0d/resource/0e537ff8-f84a-4f7f-a00b-ebfbdea0289e/download/ahtdp-partial-nephrectomy-2017.pdf. Accessed 02 May 2018

Lin HC, Shafran I, Murphy TE, Okamura AM, Yuh DD, Hager GD (2005) Automatic detection and segmentation of robot-assisted surgical motions. In: International conference on medical image computing and computer-assisted intervention, vol 8(Pt 1), pp 802–810. https://doi.org/10.1007/11566465_99

Katić D, Julliard C, Wekerle AL, Kenngott H, Möller-Stich BP, Dillmann R, Speidel S, Jannin P, Gibaud B (2015) LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition. Int J Comput Assist Radiol Surg 10(9):1427–34. https://doi.org/10.1007/s11548-015-1222-1 CrossRefPubMed

10.

Nakawala H, Ferrigno G, De Momi E (2018) Development of an intelligent surgical training system for thoracentesis. Artif Intell Med 84:50–63. https://doi.org/10.1016/j.artmed.2017.10.004 CrossRefPubMed

11.

Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97. https://doi.org/10.1109/TMI.2016.2593957 CrossRefPubMed

12.

Jin Y, Doi Q, Chen H, Yu L, Qin J, Fu C-W, Heng P-A (2018) SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37(5):1114–1126. https://doi.org/10.1109/TMI.2017.2787657 CrossRefPubMed

13.

Cadene R, Robert T, Thome N, Cord M (2016) M2CAI workflow challenge: convolutional neural network with time smoothing and hidden Markov model for video frames classification. arXiv preprint arXiv:1610.05541

14.

Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2014) Long-term recurrent convolutional networks for visual recognition and description. arXiv:1411.4389. https://doi.org/10.21236/ada623249

15.

Szegedy C, Vanhoucke V, loffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/cvpr.2016.308

16.

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y CrossRef

17.

Kingma DP, Ba J (2017) Adam: a method for stochastic optimization. arXiv:1412.6980

18.

Kaouk JH, Khalifeh A, Hillyer S, Haber G-P, Stein RJ, Autorino R (2012) Robot-assisted laparoscopic partial nephrectomy: step-by-step contemporary technique and surgical outcomes at a single high-volume institution. Eur Urol 62(3):553–561. https://doi.org/10.1016/j.eururo.2012.05.021 CrossRefPubMed

19.

Neumuth T, Strau G, Meixensberger J, Lemke HU, Burgert O (2006) Acquisition of process descriptions from surgical interventions. In: Bressan S, Kung J, Wagner R (eds) DEXA 2006 LNCS, vol 4080. Springer, Heidelberg, pp 602–611

20.

Gibaud B, Forestier G, Feldmann C, Ferrigno G, Gonçalves P, Haidegger T, Julliard C, Katić D, Kenngott H, Maier-Hein L, März K, de Momi E, Nagy DA, Nakawala H, Neumann J, Neumuth T, Balderrama JR, Speidel S, Wagner M, Jannin P (2018) Toward a standard ontology of surgical process models. Int J Comput Assist Radiol Surg 13(9):1397–1408. https://doi.org/10.1007/s11548-018-1824-5 CrossRefPubMed

21.

Grenon P, Smith B (2004) SNAP and SPAN: towards dynamic spatial ontology. Spat Cogn Comput 4(1):69–104. https://doi.org/10.1207/s15427633scc0401_5 CrossRef

22.

Rosse C, Mejino JLV (2007) The foundational model of anatomy ontology. In: Burger A, Davidson D, Baldock R (eds) Anatomy ontologies for bioinformatics. Computational biology, vol 6. Springer, London, pp 59–117. https://doi.org/10.1007/978-1-84628-885-2_4

23.

Information Artifact Ontology. https://code.google.com/p/information-artifact-ontology. Accessed 17 Aug 2016

24.

W3C Time Ontology. https://www.w3.org/TR/owl-time/. Accessed 20 Aug 2016

25.

Xiang Z, Courtot M, Brinkman RR, Ruttenberg A, He Y (2010) OntoFox: web-based support for ontology reuse. BMC Res Notes 3(1):175. https://doi.org/10.1186/1756-0500-3-175 CrossRefPubMedPubMedCentral

26.

Protégé, Stanford Center for Biomedical Informatics Research. http://protege.stanford.edu. Accessed 12 Jan 2016

27.

Horrocks I, Patel-Scheider P, Boley H, Tabet S, Grosof B, Dean M (2017) SWRL: a semantic web rule language combining OWL and RuleML. W3C Member Submission 2004. https://www.w3.org/Submission/SWRL. Accessed 23 Feb 2017

28.

Chollet F (2015) Keras. https://github.com/fchollet/keras. Accessed 05 May 2017

29.

Horridge M, Bechhofer S (2011) The OWL API: a Java API for OWL ontologies. Semant Web J 2(1):11–21

30.

Sirin E, Parsia B, Grau BC, Kalyanpur A, Katz Y (2007) Pellet: a pratical OWL-DL reasoner. Web Semant Sci Serv Agents World Wide Web 5(2):51–53. https://doi.org/10.1016/j.websem.2007.03.004 CrossRef

31.

Kipp M (2007) Anvil—a generic annotation tool for multimodal dialogue. In: 7th European conference on speech communication and technology (Eurospeech), pp 1367–1370

32.

Sun R, Giles CL (2001) Sequence learning: from recognition and prediction to sequential decision making. IEEE Intell Syst 16(4):67–70CrossRef

33.

Maaten LV, Hinton G (2008) Visualize data using t-SNE. J Mach Learn Res 9:2579–2605

34.

Nakawala N, De Momi E, Pescatori LE, Morelli A, Ferrigno G (2017) Inductive learning of the surgical workflow model through video annotations. In: IEEE 30th international symposium on computer-based medical systems, CBMS 2017, Thessaloniki, Greece. https://doi.org/10.1109/CBMS.2017.91

Titel: “Deep-Onto” network for surgical workflow and context recognition
verfasst von: Hirenkumar Nakawala
Roberto Bianchi
Laura Erica Pescatori
Ottavio De Cobelli
Giancarlo Ferrigno
Elena De Momi
Publikationsdatum: 16.11.2018
Verlag: Springer International Publishing
Erschienen in: International Journal of Computer Assisted Radiology and Surgery / Ausgabe 4/2019
Print ISSN: 1861-6410
Elektronische ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-018-1882-8

Springer Professional

Abstract

Purpose

Methods

Results

Conclusion

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 4/2019

Breast tumor classification using different features of quantitative ultrasound parametric images

Wearable technology-based metrics for predicting operator performance during cardiac catheterisation

The Adaptive Hermite Fractal Tree (AHFT): a novel surgical 3D path planning approach with curvature and heading constraints

Joint reconstruction and classification of tumor cells and cell interactions in melanoma tissue sections with synthesized training data

Predicting underestimation of ductal carcinoma in situ: a comparison between radiomics and conventional approaches

Early esophageal adenocarcinoma detection using deep learning methods