Skip to main content
SpringerLink
Log in

Inter-foetus Membrane Segmentation for TTTS Using Adversarial Networks

  • Original Article
  • Published:
Annals of Biomedical Engineering Aims and scope Submit manuscript

Abstract

Twin-to-Twin Transfusion Syndrome is commonly treated with minimally invasive laser surgery in fetoscopy. The inter-foetal membrane is used as a reference to find abnormal anastomoses. Membrane identification is a challenging task due to small field of view of the camera, presence of amniotic liquid, foetus movement, illumination changes and noise. This paper aims at providing automatic and fast membrane segmentation in fetoscopic images. We implemented an adversarial network consisting of two Fully-Convolutional Neural Networks. The former (the segmentor) is a segmentation network inspired by U-Net and integrated with residual blocks, whereas the latter acts as critic and is made only of the encoding path of the segmentor. A dataset of 900 images acquired in 6 surgical cases was collected and labelled to validate the proposed approach. The adversarial networks achieved a median Dice similarity coefficient of 91.91% with Inter-Quartile Range (IQR) of 4.63%, overcoming approaches based on U-Net (82.98%-IQR: 14.41%) and U-Net with residual blocks (86.13%-IQR: 13.63%). Results proved that the proposed architecture could be a valuable and robust solution to assist surgeons in providing membrane identification while performing fetoscopic surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5

Similar content being viewed by others

Notes

  1. https://pytorch.org/.

References

  1. Almoussa, N., B. Dutra, B. Lampe, P. Getreuer, T. Wittman, C. Salafia, and L. Vese. Automated vasculature extraction from placenta images. In: Medical Imaging 2011: Image Processing, International Society for Optics and Photonics, vol. 7962, p. 79621, 2011.

  2. Badrinarayanan, V., A. Kendall, and R. Cipolla. Segnet: adeep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39:2481–2495, 2017.

    Article  Google Scholar 

  3. Bano, S., F. Vasconcelos, M. Tella, G. Dwyer, C. Gruijthuijsen, J. Deprest, S. Ourselin, E. Vander Poorten, T. Vercauteren and D. Stoyanov. Deep sequential mosaicking of fetoscopic videos. In: Medical Image Computing and Computer Assisted Intervention, 2019

  4. Bolch, C., M. Fahey, D. Reddihough, K. Williams, S. Reid, A. Guzys, S. Cole, A. Edwards, A. Fung, R. Hodges et al. Twin-to-twin transfusion syndrome neurodevelopmental follow-up study (neurodevelopmental outcomes for children whose twin-to-twin transfusion syndrome was treated with placental laser photocoagulation). BMC pediatrics 18:256, 2018.

    Article  Google Scholar 

  5. Brandao, P., O. Zisimopoulos, E. Mazomenos, G. Ciuti, J. Bernal, M. Visentini-Scarzanella, A. Menciassi, P. Dario, A. Koulaouzidis, A. Arezzo et al. Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks. Journal of Medical Robotics Research 3:1840002, 2018.

    Article  Google Scholar 

  6. Chhatkuli, A., A. Bartoli, A. Malti, and T. Collins. Live image parsing in uterine laparoscopy. In: IEEE International Symposium on Biomedical Imaging, pp. 1263–1266, IEEE 2014.

  7. Colleoni, E., S. Moccia, X. Du, E. De Momi, and D. Stoyanov. Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers. IEEE Robotics and Automation Letters 4:2714–2721, 2019.

    Article  Google Scholar 

  8. Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: a Large-Scale Hierarchical Image Database. In: CVPR09. 2009.

  9. Deprest, J. A., D. V. Schoubroeck, P. P. V. Ballaer, H. Flageole, F. A. V. Assche, and K. Vandenberghe. Alternative technique for nd : YAG laser coagulation in twin-to-twin transfusion syndrome with anterior placenta. Ultrasound in Obstetrics and Gynecology 11:347–352, 1998.

    Article  CAS  Google Scholar 

  10. Drozdzal, M., E. Vorontsov, G. Chartrand, S. Kadoury, and C. Pal. The importance of skip connections in biomedical image segmentation. In: Deep Learning and Data Labeling for Medical Applications, pp. 179–187, Springer, 2016.

  11. Du, X., T. Kurmann, P. Chang, M. Allan, S. Ourselin, R. Sznitman, J. D. Kelly, and D. Stoyanov. Articulated multi-instrument 2d pose estimation using fully convolutional networks. IEEE Transactions on Medical Imaging 37:1276–1287, 2018.

    Article  Google Scholar 

  12. Everingham, M., L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88:303–338, 2010.

    Article  Google Scholar 

  13. Gaisser, F., S. H. Peeters, B. Lenseigne, P. Jonker, and D. Oepkes. Stable image registration for in-vivo fetoscopic panorama reconstruction. Journal of Imaging 4, 2018.

    Article  Google Scholar 

  14. Ghosh, T., L. Li, and J. Chakareski. Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images. In: IEEE International Conference on Image Processing, pp. 3034–3038, IEEE, 2018.

  15. Goodfellow, I., Y. Bengio, and A. Courville. Deep learning, MIT press, Cambridge, 2016.

    Google Scholar 

  16. Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In: Advances in Neural Information Processing Systems 27, edited by Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. c, pp. 2672–2680, Curran Associates, Inc., Red Hook, 2014.

    Google Scholar 

  17. He, K., X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. 2016.

  18. Huber, A., A. A. Baschat, T. Bregenzer, A. Diemert, M. Tchirikov, B. J. Hackelöer, and K. Hecher. Laser coagulation of placental anastomoses with a 30° fetoscope in severe mid-trimester twin–twin transfusion syndrome with anterior placenta. Ultrasound in Obstetrics and Gynecology 31:412–416, 2008.

    Article  CAS  Google Scholar 

  19. Ioffe, S. and C. Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. 2015.

  20. Kingma, D. P. and J. Ba. Adam: A method for stochastic optimization. In: International Conference on Learning Representation. 2015.

  21. Litjens, G., T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez. A survey on deep learning in medical image analysis. Medical Image Analysis 42:60–88, 2017.

    Article  Google Scholar 

  22. Long, J., E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440. 2015.

  23. Luc, P., C. Couprie, S. Chintala, and J. Verbeek. Semantic segmentation using adversarial networks. In: Neural Information Processing Systems Workshop on Adversarial Training. 2016.

  24. Maas, A. L., A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In: International Conference on Machine Learning Workshop on Deep Learning for Audio, Speech and Language Processing. 2013.

  25. Mewes, P. W., D. Neumann, O. Licegevic, J. Simon, A. L. Juloski, and E. Angelopoulou. Automatic region-of-interest segmentation and pathology detection in magnetically guided capsule endoscopy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 141–148, Springer, 2011.

  26. Miller, W., J. Novotny, D. Laidlaw, F. L. Luks, D. Merck, and S. Collins. Virtually visualizing vessels : a study of the annotation of placental vasculature from mri in large-scale virtual reality for surgical planning. In: Brown University, Providence. 2016.

  27. Moccia, S., G. O. Vanone, E. De Momi, A. Laborai, L. Guastini, G. Peretti, and L. S. Mattos. Learning-based classification of informative laryngoscopic frames. Computer Methods and Programs in Biomedicine 158:21–30, 2018.

    Article  Google Scholar 

  28. Moccia, S., S. J. Wirkert, H. Kenngott, A. S. Vemuri, M. Apitz, B. Mayer, E. De Momi, L. S. Mattos, and L. Maier-Hein. Uncertainty-aware organ classification for surgical data science applications in laparoscopy. IEEE Transactions on Biomedical Engineering 65:2649–2659, 2018.

    Article  Google Scholar 

  29. Odena, A., V. Dumoulin, and C. Olah. Deconvolution and checkerboard artifacts. Distill , 2016.

    Article  Google Scholar 

  30. Pratt, R., J. Deprest, T. Vercauteren, S. Ourselin, and A. L. David. Computer-assisted surgical planning and intraoperative guidance in fetal surgery: a systematic review. Prenatal Diagnosis 35:1159–1166, 2015.

    Article  Google Scholar 

  31. Prokopetc, K., T. Collins, and A. Bartoli. Automatic detection of the uterus and fallopian tube junctions in laparoscopic images. In: International Conference on Information Processing in Medical Imaging, pp. 552–563, Springer, 2015.

  32. Quintero, R. A. Twin–twin transfusion syndrome. Clinics in Perinatology 30:591–600, 2003.

    Article  Google Scholar 

  33. Roberts, D., J. P. Neilson, M. D. Kilby, and S. Gates. Interventions for the treatment of twin-twin transfusion syndrome. Cochrane Database of Systematic Reviews, 2014.

    Google Scholar 

  34. Ronneberger, O., P. Fischer, and T. Brox. U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention pp. 234–241, 2015.

    Google Scholar 

  35. Sadda, P., M. Imamoglu, M. Dombrowski, X. Papademetris, M. O. Bahtiyar, and J. Onofrey. Deep-learned placental vessel segmentation for intraoperative video enhancement in fetoscopic surgery. International Journal of Computer Assisted Radiology and Surgery 14:227–235, 2019.

    Article  Google Scholar 

  36. Senat, M.-V., J. Deprest, M. Boulvain, A. Paupe, N. Winer, and Y. Ville. Endoscopic laser surgery versus serial amnioreduction for severe twin-to-twin transfusion syndrome. New England Journal of Medicine 351:136–144, 2004.

    Article  CAS  Google Scholar 

  37. Shen, D., G. Wu, and H.-I. Suk. Deep learning in medical image analysis. Annual Review of Biomedical Engineering 19:221, 2017.

    Article  CAS  Google Scholar 

  38. Tella, M., P. Daga, F. Chadebecq, S. Thompson, D. I. Shakir, G. Dwyer, R. Wimalasundera, J. Deprest, D. Stoyanov, T. Vercauteren, and S. Ourselin. A combined em and visual tracking probabilistic model for robust mosaicking: application to fetoscopy. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016.

  39. Vasconcelos, F., P. Brandão, T. Vercauteren, S. Ourselin, J. Deprest, D. Peebles, and D. Stoyanov. Towards computer-assisted TTTS: Laser ablation detection for workflow segmentation from fetoscopic video. International Journal of Computer Assisted Radiology and Surgery 13:1661–1670, 2018.

    Article  Google Scholar 

  40. Wickstrøm, K., M. Kampffmeyer, and R. Jenssen. Uncertainty modeling and interpretability in convolutional neural networks for polyp segmentation. In: IEEE International Workshop on Machine Learning for Signal Processing, pp. 1–6, IEEE, 2018.

  41. Xu, B., N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 , 2015.

  42. Xue, Y., T. Xu, H. Zhang, L. R. Long, and X. Huang. Segan: Adversarial network with multi-scale l1 loss for medical image segmentation. Neuroinformatics 16:383–392, 2018.

    Article  Google Scholar 

  43. Xue, Y., T. Xu, and X. Huang. Adversarial learning with multi-scale loss for skin lesion segmentation. In: IEEE International Symposium on Biomedical Imaging, IEEE, 2018.

  44. Yu, L., H. Chen, Q. Dou, J. Qin, and P. A. Heng. Integrating online and offline three-dimensional deep learning for automated polyp detection in colonoscopy videos. IEEE Journal of Biomedical and Health Informatics 21:65–75, 2016.

    Article  Google Scholar 

  45. Zhu, X., X. Zhang, X.-Y. Zhang, Z. Xue, and L. Wang. A novel framework for semantic segmentation with generative adversarial network. Journal of Visual Communication and Image Representation 58:532 – 543, 2019.

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Alessandro Casella and Sara Moccia have contributed equally to this work.

Authors and Affiliations

  1. Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy

    Alessandro Casella & Elena De Momi

  2. Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy

    Alessandro Casella, Sara Moccia & Leonardo S. Mattos

  3. Department of Information Engineering, Università Politecnica delle Marche, Ancona, Italy

    Sara Moccia & Emanuele Frontoni

  4. Department of Foetal and Perinatal Medicine, Istituto Giannina Gaslini, Genoa, Italy

    Dario Paladini

Authors
  1. Alessandro Casella
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sara Moccia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emanuele Frontoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dario Paladini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Elena De Momi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leonardo S. Mattos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessandro Casella.

Ethics declarations

Conflict of interest

No benefits in any form have been or will be received from a commercial party related directly or indirectly to the subjects of this manuscript.

Ethical Standards

This article followed the Ethics Guidelines for Trustworthy Artificial Intelligence, recently published by the European Commission (https://ec.europa.eu/futurium/en/ai-alliance-consultation).

Additional information

Associate Editor Daniel Elson oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Casella, A., Moccia, S., Frontoni, E. et al. Inter-foetus Membrane Segmentation for TTTS Using Adversarial Networks. Ann Biomed Eng 48, 848–859 (2020). https://doi.org/10.1007/s10439-019-02424-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10439-019-02424-9

Keywords

Search

Navigation