Skip to main content
Erschienen in: International Journal of Computer Assisted Radiology and Surgery 7/2017

15.05.2017 | Original Article

Deep monocular 3D reconstruction for assisted navigation in bronchoscopy

verfasst von: Marco Visentini-Scarzanella, Takamasa Sugiura, Toshimitsu Kaneko, Shinichiro Koto

Erschienen in: International Journal of Computer Assisted Radiology and Surgery | Ausgabe 7/2017

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Purpose

In bronchoschopy, computer vision systems for navigation assistance are an attractive low-cost solution to guide the endoscopist to target peripheral lesions for biopsy and histological analysis. We propose a decoupled deep learning architecture that projects input frames onto the domain of CT renderings, thus allowing offline training from patient-specific CT data.

Methods

A fully convolutional network architecture is implemented on GPU and tested on a phantom dataset involving 32 video sequences and \(\sim \)60k frames with aligned ground truth and renderings, which is made available as the first public dataset for bronchoscopy navigation.

Results

An average estimated depth accuracy of 1.5 mm was obtained, outperforming conventional direct depth estimation from input frames by 60%, and with a computational time of \(\le \)30 ms on modern GPUs. Qualitatively, the estimated depth and renderings closely resemble the ground truth.

Conclusions

The proposed method shows a novel architecture to perform real-time monocular depth estimation without losing patient specificity in bronchoscopy. Future work will include integration within SLAM systems and collection of in vivo datasets.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Fußnoten
2
While in our experiments the material \(\textit{violet-rubber}\) was uniformly used, any other physically meaningful BRDF can be used for the purpose.
 
Literatur
1.
Zurück zum Zitat Asano F, Eberhardt R, Herth FJF (2014) Virtual bronchoscopic navigation for peripheral pulmonary lesions. Respiration 88(5):430–440CrossRefPubMed Asano F, Eberhardt R, Herth FJF (2014) Virtual bronchoscopic navigation for peripheral pulmonary lesions. Respiration 88(5):430–440CrossRefPubMed
2.
Zurück zum Zitat Dosovitskiy A, Fischery P, Ilg E, Husser P, Hazirbas C, Golkov V, vd Smagt P, Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: IEEE International conference on computer vision (ICCV), pp 2758–2766 Dosovitskiy A, Fischery P, Ilg E, Husser P, Hazirbas C, Golkov V, vd Smagt P, Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: IEEE International conference on computer vision (ICCV), pp 2758–2766
3.
Zurück zum Zitat Eberhardt R, Kahn N, Gompelmann D, Schumann M, Heussel CP, Herth FJ (2010) Lungpoint—a new approach to peripheral lesions. J Thorac Oncol 5(10):1559–1563CrossRefPubMed Eberhardt R, Kahn N, Gompelmann D, Schumann M, Heussel CP, Herth FJ (2010) Lungpoint—a new approach to peripheral lesions. J Thorac Oncol 5(10):1559–1563CrossRefPubMed
4.
Zurück zum Zitat Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: IEEE International conference on computer vision (ICCV), pp 2650–2658 Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: IEEE International conference on computer vision (ICCV), pp 2650–2658
5.
Zurück zum Zitat Engel J, Schops T, Cremers D (2014) Lsd-slam: large-scale direct monocular slam. In: European conference in computer vision (ECCV), pp 834–849 Engel J, Schops T, Cremers D (2014) Lsd-slam: large-scale direct monocular slam. In: European conference in computer vision (ECCV), pp 834–849
6.
Zurück zum Zitat Garrido-Jurado S, noz Salinas RM, Madrid-Cuevas F, Marín-Jiménez M (2014) Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit 47(6):2280–2292CrossRef Garrido-Jurado S, noz Salinas RM, Madrid-Cuevas F, Marín-Jiménez M (2014) Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit 47(6):2280–2292CrossRef
7.
Zurück zum Zitat Gilbert C, Akulian J, Ortiz R, Lee H, Yarmus L (2014) Novel bronchoscopic strategies for the diagnosis of peripheral lung lesions: present techniques and future directions. Respirology 19(5):636–644CrossRefPubMed Gilbert C, Akulian J, Ortiz R, Lee H, Yarmus L (2014) Novel bronchoscopic strategies for the diagnosis of peripheral lung lesions: present techniques and future directions. Respirology 19(5):636–644CrossRefPubMed
9.
Zurück zum Zitat Hayashi Y, Misawa K, Oda M, Hawkes DJ, Mori K (2016) Clinical application of a surgical navigation system based on virtual laparoscopy in laparoscopic gastrectomy for gastric cancer. Int J Comput Assist Radiol Surg 11(5):827–836CrossRefPubMed Hayashi Y, Misawa K, Oda M, Hawkes DJ, Mori K (2016) Clinical application of a surgical navigation system based on virtual laparoscopy in laparoscopic gastrectomy for gastric cancer. Int J Comput Assist Radiol Surg 11(5):827–836CrossRefPubMed
10.
Zurück zum Zitat Herth FJ, Eberhardt R, Sterman D, Silvestri GA, Hoffmann H, Shah PL (2015) Bronchoscopic transparenchymal nodule access (btpna): first in human trial of a novel procedure for sampling solitary pulmonary nodules. Thorax 70(4):326–332CrossRefPubMed Herth FJ, Eberhardt R, Sterman D, Silvestri GA, Hoffmann H, Shah PL (2015) Bronchoscopic transparenchymal nodule access (btpna): first in human trial of a novel procedure for sampling solitary pulmonary nodules. Thorax 70(4):326–332CrossRefPubMed
11.
Zurück zum Zitat Kazhdan M, Bolitho M, Hoppe H (2006) Poisson surface reconstruction. In: Eurographics symposium on geometry processing, pp 61–70 Kazhdan M, Bolitho M, Hoppe H (2006) Poisson surface reconstruction. In: Eurographics symposium on geometry processing, pp 61–70
12.
Zurück zum Zitat Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR) Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR)
13.
Zurück zum Zitat Leong S, Ju H, Marshall H, Bowman R, Yang I, Ree AM, Saxon C, Fong KM (2012) Electromagnetic navigation bronchoscopy: a descriptive analysis. J Thorac Dis 4(2):173–185PubMedPubMedCentral Leong S, Ju H, Marshall H, Bowman R, Yang I, Ree AM, Saxon C, Fong KM (2012) Electromagnetic navigation bronchoscopy: a descriptive analysis. J Thorac Dis 4(2):173–185PubMedPubMedCentral
14.
Zurück zum Zitat Liu F, Shen C, Lin G, Reid I (2016) Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans Pattern Anal Mach Intell 38(10):2024–2039CrossRefPubMed Liu F, Shen C, Lin G, Reid I (2016) Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans Pattern Anal Mach Intell 38(10):2024–2039CrossRefPubMed
15.
Zurück zum Zitat Lorensen, W.E., Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. In: ACM SIGGRAPH, pp 163–169 Lorensen, W.E., Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. In: ACM SIGGRAPH, pp 163–169
16.
Zurück zum Zitat Luo X, Feuerstein M, Deguchi D, Kitasaka T, Takabatake H, Mori K (2012) Development and comparison of new hybrid motion tracking for bronchoscopic navigation. Med Image Anal 16(3):577–596CrossRefPubMed Luo X, Feuerstein M, Deguchi D, Kitasaka T, Takabatake H, Mori K (2012) Development and comparison of new hybrid motion tracking for bronchoscopic navigation. Med Image Anal 16(3):577–596CrossRefPubMed
17.
Zurück zum Zitat Mahmoud N, Cirauqui I, Hostettler A, Doignon C, Soler L, Marescaux J, Montiel JMM (2017) Orbslam-based endoscope tracking and 3d reconstruction. In: International workshop on computer-assisted and robotic endoscopy (CARE), pp 72–83 Mahmoud N, Cirauqui I, Hostettler A, Doignon C, Soler L, Marescaux J, Montiel JMM (2017) Orbslam-based endoscope tracking and 3d reconstruction. In: International workshop on computer-assisted and robotic endoscopy (CARE), pp 72–83
18.
Zurück zum Zitat Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996CrossRefPubMed Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996CrossRefPubMed
19.
Zurück zum Zitat Malti A, Bartoli A (2014) Combining conformal deformation and cook-torrance shading for 3-d reconstruction in laparoscopy. IEEE Trans Biomed Eng 61(6):1684–1692CrossRefPubMed Malti A, Bartoli A (2014) Combining conformal deformation and cook-torrance shading for 3-d reconstruction in laparoscopy. IEEE Trans Biomed Eng 61(6):1684–1692CrossRefPubMed
20.
Zurück zum Zitat Matusik W, Pfister H, Brand M, McMillan L (2003) A data-driven reflectance model. ACM Trans Graph 22(3):759–769CrossRef Matusik W, Pfister H, Brand M, McMillan L (2003) A data-driven reflectance model. ACM Trans Graph 22(3):759–769CrossRef
21.
Zurück zum Zitat Merritt SA, Khare R, Bascom R, Higgins WE (2013) Interactive ct-video registration for the continuous guidance of bronchoscopy. IEEE Trans Med Imaging 32(8):1376–1396CrossRefPubMedPubMedCentral Merritt SA, Khare R, Bascom R, Higgins WE (2013) Interactive ct-video registration for the continuous guidance of bronchoscopy. IEEE Trans Med Imaging 32(8):1376–1396CrossRefPubMedPubMedCentral
22.
Zurück zum Zitat Mirota D, Wang H, Taylor R, Ishii M, Gallia G, Hager G (2012) A system for video-based navigation for endoscopic endonasal skull base surgery. IEEE Trans Med Imaging 31(4):963–976CrossRefPubMed Mirota D, Wang H, Taylor R, Ishii M, Gallia G, Hager G (2012) A system for video-based navigation for endoscopic endonasal skull base surgery. IEEE Trans Med Imaging 31(4):963–976CrossRefPubMed
23.
Zurück zum Zitat Mur-Artal R, Montiel JMM, Tardos JD (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Trans Robot 31(5):1147–1163CrossRef Mur-Artal R, Montiel JMM, Tardos JD (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Trans Robot 31(5):1147–1163CrossRef
24.
Zurück zum Zitat Mura M, Abu-Kheil Y, Ciuti G, Visentini-Scarzanella M, Menciassi A, Dario P, Dias J, Seneviratne L (2016) Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept. J Micro Bio Robot 11(1):35–45CrossRef Mura M, Abu-Kheil Y, Ciuti G, Visentini-Scarzanella M, Menciassi A, Dario P, Dias J, Seneviratne L (2016) Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept. J Micro Bio Robot 11(1):35–45CrossRef
25.
Zurück zum Zitat Reiter, A., Leondard, S., Sinha, A., Ishii, M., Taylor, R.H., Hager, G.D.: Endoscopic-ct: learning-based photometric reconstruction for endoscopic surgery. In: SPIE medical imaging, pp 1–6 (2016) Reiter, A., Leondard, S., Sinha, A., Ishii, M., Taylor, R.H., Hager, G.D.: Endoscopic-ct: learning-based photometric reconstruction for endoscopic surgery. In: SPIE medical imaging, pp 1–6 (2016)
26.
Zurück zum Zitat Siegel RL, Miller KD, Jemal A (2016) Cancer statistics, 2016. CA A Cancer J Clin 66(1):7–30CrossRef Siegel RL, Miller KD, Jemal A (2016) Cancer statistics, 2016. CA A Cancer J Clin 66(1):7–30CrossRef
27.
Zurück zum Zitat Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE Trans Pattern Anal Mach Intell 30(6):1068–1080CrossRefPubMed Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE Trans Pattern Anal Mach Intell 30(6):1068–1080CrossRefPubMed
28.
Zurück zum Zitat Tagliasacchi A, Alhashim I, Olson M, Zhang H (2012) Mean curvature skeletons. Comput Graph Forum 31(5):1735–1744CrossRef Tagliasacchi A, Alhashim I, Olson M, Zhang H (2012) Mean curvature skeletons. Comput Graph Forum 31(5):1735–1744CrossRef
29.
Zurück zum Zitat Umeyama S (1991) Least-squares estimation of transformation parameters between two point patterns. IEEE Trans Pattern Anal Mach Intell 13(4):376–380CrossRef Umeyama S (1991) Least-squares estimation of transformation parameters between two point patterns. IEEE Trans Pattern Anal Mach Intell 13(4):376–380CrossRef
30.
Zurück zum Zitat Visentini-Scarzanella M, Kawasaki H (2015) Simultaneous camera, light position and radiant intensity distribution calibration. In: Pacific rim symposium on image and video technology (PSIVT), pp 557–571 Visentini-Scarzanella M, Kawasaki H (2015) Simultaneous camera, light position and radiant intensity distribution calibration. In: Pacific rim symposium on image and video technology (PSIVT), pp 557–571
31.
Zurück zum Zitat Visentini-Scarzanella M, Mylonas GP, Stoyanov D, Yang GZ: i-brush: A gaze-contingent virtual paintbrush for dense 3d reconstruction in robotic assisted surgery. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 353–360 Visentini-Scarzanella M, Mylonas GP, Stoyanov D, Yang GZ: i-brush: A gaze-contingent virtual paintbrush for dense 3d reconstruction in robotic assisted surgery. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 353–360
32.
Zurück zum Zitat Weisstein EW (2002) Sphere point picking. Tech. rep, Wolfram MathWorld Weisstein EW (2002) Sphere point picking. Tech. rep, Wolfram MathWorld
33.
Zurück zum Zitat Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRef Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRef
34.
Zurück zum Zitat Zhao Q, Price T, Pizer S, Niethammer M, Alterovitz R, Rosenman J (2016) The endoscopogram: a 3d model reconstructed from endoscopic video frames. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 439–447 Zhao Q, Price T, Pizer S, Niethammer M, Alterovitz R, Rosenman J (2016) The endoscopogram: a 3d model reconstructed from endoscopic video frames. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 439–447
Metadaten
Titel
Deep monocular 3D reconstruction for assisted navigation in bronchoscopy
verfasst von
Marco Visentini-Scarzanella
Takamasa Sugiura
Toshimitsu Kaneko
Shinichiro Koto
Publikationsdatum
15.05.2017
Verlag
Springer International Publishing
Erschienen in
International Journal of Computer Assisted Radiology and Surgery / Ausgabe 7/2017
Print ISSN: 1861-6410
Elektronische ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-017-1609-2

Weitere Artikel der Ausgabe 7/2017

International Journal of Computer Assisted Radiology and Surgery 7/2017 Zur Ausgabe