Skip to main content
Erschienen in: International Journal of Computer Assisted Radiology and Surgery 8/2016

01.08.2016 | Original Article

Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors

verfasst von: Masoud S. Nosrati, Alborz Amir-Khalili, Jean-Marc Peyrat, Julien Abinahed, Osama Al-Alao, Abdulla Al-Ansari, Rafeef Abugharbieh, Ghassan Hamarneh

Erschienen in: International Journal of Computer Assisted Radiology and Surgery | Ausgabe 8/2016

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Purpose

Despite great advances in medical image segmentation, the accurate and automatic segmentation of endoscopic scenes remains a challenging problem. Two important aspects have to be considered in segmenting an endoscopic scene: (1) noise and clutter due to light reflection and smoke from cutting tissue, and (2) structure occlusion (e.g. vessels occluded by fat, or endophytic tumours occluded by healthy kidney tissue).

Methods

In this paper, we propose a variational technique to augment a surgeon’s endoscopic view by segmenting visible as well as occluded structures in the intraoperative endoscopic view. Our method estimates the 3D pose and deformation of anatomical structures segmented from 3D preoperative data in order to align to and segment corresponding structures in 2D intraoperative endoscopic views. Our preoperative to intraoperative alignment is driven by, first, spatio-temporal, signal processing based vessel pulsation cues and, second, machine learning based analysis of colour and textural visual cues. To our knowledge, this is the first work that utilizes vascular pulsation cues for guiding preoperative to intraoperative registration. In addition, we incorporate a tissue-specific (i.e. heterogeneous) physically based deformation model into our framework to cope with the non-rigid deformation of structures that occurs during the intervention.

Results

We validated the utility of our technique on fifteen challenging clinical cases with 45 % improvements in accuracy compared to the state-of-the-art method.

Conclusions

A new technique for localizing both visible and occluded structures in an endoscopic view was proposed and tested. This method leverages both preoperative data, as a source of patient-specific prior knowledge, as well as vasculature pulsation and endoscopic visual cues in order to accurately segment the highly noisy and cluttered environment of an endoscopic video. Our results on in vivo clinical cases of partial nephrectomy illustrate the potential of the proposed framework for augmented reality applications in minimally invasive surgeries.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Agudo A, Agapito L, Calvo B, Montiel J (2014) Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1558–1565 Agudo A, Agapito L, Calvo B, Montiel J (2014) Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1558–1565
2.
Zurück zum Zitat Amir-Khalili A, Hamarneh G, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R (2015) Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video. Med Image Anal 25(1):103–110 Amir-Khalili A, Hamarneh G, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R (2015) Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video. Med Image Anal 25(1):103–110
3.
Zurück zum Zitat Amir-Khalili A, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Hamarneh G, Abugharbieh R (2014) Auto localization and segmentation of occluded vessels in robot-assisted partial nephrectomy. In: Medical image computing and computer-assisted intervention (MICCAI), pp 407–414 Amir-Khalili A, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Hamarneh G, Abugharbieh R (2014) Auto localization and segmentation of occluded vessels in robot-assisted partial nephrectomy. In: Medical image computing and computer-assisted intervention (MICCAI), pp 407–414
4.
Zurück zum Zitat Andrews S, Hamarneh G (2015) The generalized log-ratio transformation: learning shape and adjacency priors for simultaneous thigh muscle segmentation. IEEE Trans Med Imaging 34(9):1773–1787 Andrews S, Hamarneh G (2015) The generalized log-ratio transformation: learning shape and adjacency priors for simultaneous thigh muscle segmentation. IEEE Trans Med Imaging 34(9):1773–1787
5.
Zurück zum Zitat Andrews S, McIntosh C, Hamarneh G (2011) Convex multi-region probabilistic segmentation with shape prior in the isometric log-ratio transformation space. In: IEEE international conference on computer vision (ICCV). IEEE, pp 2096–2103 Andrews S, McIntosh C, Hamarneh G (2011) Convex multi-region probabilistic segmentation with shape prior in the isometric log-ratio transformation space. In: IEEE international conference on computer vision (ICCV). IEEE, pp 2096–2103
6.
Zurück zum Zitat Brown E, Chan T, Bresson X (2009) Convex formulation and exact global solutions for multi-phase piecewise constant Mumford–Shah image segmentation. UCLA CAM report, pp 09–66 Brown E, Chan T, Bresson X (2009) Convex formulation and exact global solutions for multi-phase piecewise constant Mumford–Shah image segmentation. UCLA CAM report, pp 09–66
7.
Zurück zum Zitat Chan TF, Vese L et al (2001) Active contours without edges. IEEE Trans Image Process 10(2):266–277CrossRefPubMed Chan TF, Vese L et al (2001) Active contours without edges. IEEE Trans Image Process 10(2):266–277CrossRefPubMed
8.
Zurück zum Zitat Crane NJ et al (2010) Visual enhancement of laparoscopic partial nephrectomy with 3-charge coupled device camera: assessing intraoperative tissue perfusion and vascular anatomy by visible hemoglobin spectral response. J Urol 184(4):1279–1285CrossRefPubMed Crane NJ et al (2010) Visual enhancement of laparoscopic partial nephrectomy with 3-charge coupled device camera: assessing intraoperative tissue perfusion and vascular anatomy by visible hemoglobin spectral response. J Urol 184(4):1279–1285CrossRefPubMed
9.
Zurück zum Zitat Delong A, Boykov Y (2009) Globally optimal segmentation of multi-region objects. In: ieee international conference on computer vision (IEEE ICCV), pp 285–292 Delong A, Boykov Y (2009) Globally optimal segmentation of multi-region objects. In: ieee international conference on computer vision (IEEE ICCV), pp 285–292
10.
Zurück zum Zitat Escudier B, Kataja V et al (2010) Renal cell carcinoma: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol 21(Suppl 5):v137–v139CrossRefPubMed Escudier B, Kataja V et al (2010) Renal cell carcinoma: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol 21(Suppl 5):v137–v139CrossRefPubMed
11.
Zurück zum Zitat Estépar RSJ, Westin CF, Vosburgh KG (2009) Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures. Int J Comput Assist Radiol Surg 4(6):549–560CrossRefPubMedCentral Estépar RSJ, Westin CF, Vosburgh KG (2009) Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures. Int J Comput Assist Radiol Surg 4(6):549–560CrossRefPubMedCentral
12.
Zurück zum Zitat Figueiredo IN, Figueiredo PN, Stadler G, Ghattas O, Araujo A (2010) Variational image segmentation for endoscopic human colonic aberrant crypt foci. IEEE Trans Med Imaging 29(4):998–1011CrossRefPubMed Figueiredo IN, Figueiredo PN, Stadler G, Ghattas O, Araujo A (2010) Variational image segmentation for endoscopic human colonic aberrant crypt foci. IEEE Trans Med Imaging 29(4):998–1011CrossRefPubMed
13.
Zurück zum Zitat Figueiredo IN, Moreno JC, Prasath VBS, Figueiredo PN (2012) A segmentation model and application to endoscopic images. In: Image analysis and recognition. Springer, pp 164–171 Figueiredo IN, Moreno JC, Prasath VBS, Figueiredo PN (2012) A segmentation model and application to endoscopic images. In: Image analysis and recognition. Springer, pp 164–171
14.
Zurück zum Zitat Gill IS, Desai MM, Kaouk JH, Meraney AM, Murphy DP, Sung GT, Novick AC (2002) Laparoscopic partial nephrectomy for renal tumor: duplicating open surgical techniques. J Urol 167(2):469–476CrossRefPubMed Gill IS, Desai MM, Kaouk JH, Meraney AM, Murphy DP, Sung GT, Novick AC (2002) Laparoscopic partial nephrectomy for renal tumor: duplicating open surgical techniques. J Urol 167(2):469–476CrossRefPubMed
15.
Zurück zum Zitat Gill IS, Kavoussi LR, Lane BR, Blute ML, Babineau D, Colombo JR Jr, Frank I, Permpongkosol S, Weight CJ, Kaouk JH et al (2007) Comparison of 1,800 laparoscopic and open partial nephrectomies for single renal tumors. J Urol 178(1):41–46CrossRefPubMed Gill IS, Kavoussi LR, Lane BR, Blute ML, Babineau D, Colombo JR Jr, Frank I, Permpongkosol S, Weight CJ, Kaouk JH et al (2007) Comparison of 1,800 laparoscopic and open partial nephrectomies for single renal tumors. J Urol 178(1):41–46CrossRefPubMed
16.
Zurück zum Zitat Gill S, Abolmaesumi P, Vikal S, Mousavi P, Fichtinger G (2008) Intraoperative prostate tracking with slice-to-volume registration in MR. In: International conference of the society for medical innovation and technology, pp 154–158 Gill S, Abolmaesumi P, Vikal S, Mousavi P, Fichtinger G (2008) Intraoperative prostate tracking with slice-to-volume registration in MR. In: International conference of the society for medical innovation and technology, pp 154–158
17.
Zurück zum Zitat Hernes N, Toril A, Lindseth F, Selbekk T, Wollf A, Solberg OV, Harg E, Rygh OM, Tangen GA, Rasmussen I et al (2006) Computer-assisted 3D ultrasound-guided neurosurgery: technological contributions, including multimodal registration and advanced display, demonstrating future perspectives. Int J Med Robot Comput Assist Surg 2(1):45–59CrossRef Hernes N, Toril A, Lindseth F, Selbekk T, Wollf A, Solberg OV, Harg E, Rygh OM, Tangen GA, Rasmussen I et al (2006) Computer-assisted 3D ultrasound-guided neurosurgery: technological contributions, including multimodal registration and advanced display, demonstrating future perspectives. Int J Med Robot Comput Assist Surg 2(1):45–59CrossRef
18.
Zurück zum Zitat Hummel J, Figl M, Bax M, Bergmann H, Birkfellner W (2008) 2D/3D registration of endoscopic ultrasound to CT volume data. Phys Med Biol 53(16):4303CrossRefPubMed Hummel J, Figl M, Bax M, Bergmann H, Birkfellner W (2008) 2D/3D registration of endoscopic ultrasound to CT volume data. Phys Med Biol 53(16):4303CrossRefPubMed
19.
Zurück zum Zitat McIntosh C, Hamarneh G (2009) Optimal weights for convex functionals in medical image segmentation. In: Advances in visual computing. Springer, pp 1079–1088 McIntosh C, Hamarneh G (2009) Optimal weights for convex functionals in medical image segmentation. In: Advances in visual computing. Springer, pp 1079–1088
20.
Zurück zum Zitat Merritt SA, Rai L, Higgins WE (2006) Real-time CT-video registration for continuous endoscopic guidance. In: Medical imaging. International Society for Optics and Photonics, pp 614313–614313 Merritt SA, Rai L, Higgins WE (2006) Real-time CT-video registration for continuous endoscopic guidance. In: Medical imaging. International Society for Optics and Photonics, pp 614313–614313
21.
Zurück zum Zitat Mewes PW, Neumann D, Licegevic O, Simon J, Juloski AL, Angelopoulou E (2011) Automatic region-of-interest segmentation and pathology detection in magnetically guided capsule endoscopy. In: Medical image computing and computer-assisted intervention (MICCAI 2011). Springer, pp 141–148 Mewes PW, Neumann D, Licegevic O, Simon J, Juloski AL, Angelopoulou E (2011) Automatic region-of-interest segmentation and pathology detection in magnetically guided capsule endoscopy. In: Medical image computing and computer-assisted intervention (MICCAI 2011). Springer, pp 141–148
22.
Zurück zum Zitat Nosrati MS, Andrews S, Hamarneh G (2013) Bounded labeling function for global segmentation of multi-part objects with geometric constraints. In: IEEE international conference on computer vision (ICCV), pp 2032–2039 Nosrati MS, Andrews S, Hamarneh G (2013) Bounded labeling function for global segmentation of multi-part objects with geometric constraints. In: IEEE international conference on computer vision (ICCV), pp 2032–2039
23.
Zurück zum Zitat Nosrati MS, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R, Hamarneh G (2014) Efficient multi-organ segmentation in multi-view endoscopic videos using pre-operative priors. In: Medical image computing and computer-assisted intervention (MICCAI), pp 324–331 Nosrati MS, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R, Hamarneh G (2014) Efficient multi-organ segmentation in multi-view endoscopic videos using pre-operative priors. In: Medical image computing and computer-assisted intervention (MICCAI), pp 324–331
24.
Zurück zum Zitat Pentland A, Sclaroff S (1991) Closed-form solutions for physically based shape modeling and recognition. IEEE Trans Pattern Anal Mach Intell (IEEE TPAMI) 7:715–729CrossRef Pentland A, Sclaroff S (1991) Closed-form solutions for physically based shape modeling and recognition. IEEE Trans Pattern Anal Mach Intell (IEEE TPAMI) 7:715–729CrossRef
25.
Zurück zum Zitat Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2D–3D image registration. In: Engineering in medicine and biology society. EMBC 2009. Annual international conference of the IEEE. IEEE, pp 5821–5824 Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2D–3D image registration. In: Engineering in medicine and biology society. EMBC 2009. Annual international conference of the IEEE. IEEE, pp 5821–5824
26.
Zurück zum Zitat Pratt P, Mayer E, Vale J, Cohen D, Edwards E, Darzi A, Yang GZ (2012) An effective visualisation and registration system for image-guided robotic partial nephrectomy. J Robot Surg 6(1):23–31CrossRef Pratt P, Mayer E, Vale J, Cohen D, Edwards E, Darzi A, Yang GZ (2012) An effective visualisation and registration system for image-guided robotic partial nephrectomy. J Robot Surg 6(1):23–31CrossRef
27.
Zurück zum Zitat Puerto-Souza GA, Mariottini GL (2013) Toward long-term and accurate augmented-reality display for minimally-invasive surgery. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 5384–5389 Puerto-Souza GA, Mariottini GL (2013) Toward long-term and accurate augmented-reality display for minimally-invasive surgery. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 5384–5389
28.
Zurück zum Zitat Singh I (2009) Robot-assisted laparoscopic partial nephrectomy: current review of the technique and literature. J Min Access Surg 5(4):87 Singh I (2009) Robot-assisted laparoscopic partial nephrectomy: current review of the technique and literature. J Min Access Surg 5(4):87
29.
Zurück zum Zitat Teber D et al (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56(2):332–338CrossRefPubMed Teber D et al (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56(2):332–338CrossRefPubMed
30.
Zurück zum Zitat Tobis S et al (2011) Near infrared fluorescence imaging with robotic assisted laparoscopic partial nephrectomy: initial clinical experience for renal cortical tumors. J Urol 186(1):47–52CrossRefPubMed Tobis S et al (2011) Near infrared fluorescence imaging with robotic assisted laparoscopic partial nephrectomy: initial clinical experience for renal cortical tumors. J Urol 186(1):47–52CrossRefPubMed
31.
Zurück zum Zitat Urban BA et al (2001) Three-dimensional volume-rendered CT angiography of the renal arteries and veins: normal anatomy, variants, and clinical applications. RadioGraphics 21(2):373–386CrossRefPubMed Urban BA et al (2001) Three-dimensional volume-rendered CT angiography of the renal arteries and veins: normal anatomy, variants, and clinical applications. RadioGraphics 21(2):373–386CrossRefPubMed
32.
Zurück zum Zitat Yim Y, Wakid M, Kirmizibayrak C, Bielamowicz S, Hahn J (2010) Registration of 3D CT data to 2D endoscopic image using a gradient mutual information based viewpoint matching for image-guided medialization laryngoplasty. J Comput Sci Eng 4(4):368–387CrossRef Yim Y, Wakid M, Kirmizibayrak C, Bielamowicz S, Hahn J (2010) Registration of 3D CT data to 2D endoscopic image using a gradient mutual information based viewpoint matching for image-guided medialization laryngoplasty. J Comput Sci Eng 4(4):368–387CrossRef
33.
Zurück zum Zitat Zikic D, Glocker B, Kutter O, Groher M, Komodakis N, Khamene A, Paragios N, Navab N (2010) Markov random field optimization for intensity-based 2D–3D registration. In: SPIE medical imaging. International Society for Optics and Photonics, pp 762334–762334 Zikic D, Glocker B, Kutter O, Groher M, Komodakis N, Khamene A, Paragios N, Navab N (2010) Markov random field optimization for intensity-based 2D–3D registration. In: SPIE medical imaging. International Society for Optics and Photonics, pp 762334–762334
Metadaten
Titel
Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors
verfasst von
Masoud S. Nosrati
Alborz Amir-Khalili
Jean-Marc Peyrat
Julien Abinahed
Osama Al-Alao
Abdulla Al-Ansari
Rafeef Abugharbieh
Ghassan Hamarneh
Publikationsdatum
01.08.2016
Verlag
Springer Berlin Heidelberg
Erschienen in
International Journal of Computer Assisted Radiology and Surgery / Ausgabe 8/2016
Print ISSN: 1861-6410
Elektronische ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-015-1331-x

Weitere Artikel der Ausgabe 8/2016

International Journal of Computer Assisted Radiology and Surgery 8/2016 Zur Ausgabe