Skip to main content

Advertisement

SpringerLink
Log in

CTumorGAN: a unified framework for automatic computed tomography tumor segmentation

  • Original Article
  • Published:
European Journal of Nuclear Medicine and Molecular Imaging Aims and scope Submit manuscript

Abstract

Purpose

Unlike the normal organ segmentation task, automatic tumor segmentation is a more challenging task because of the existence of similar visual characteristics between tumors and their surroundings, especially on computed tomography (CT) images with severe low contrast resolution, as well as the diversity and individual characteristics of data acquisition procedures and devices. Consequently, most of the recently proposed methods have become increasingly difficult to be applied on a different tumor dataset with good results, and moreover, some tumor segmentors usually fail to generalize beyond those datasets and modalities used in their original evaluation experiments.

Methods

In order to alleviate some of the problems with the recently proposed methods, we propose a novel unified and end-to-end adversarial learning framework for automatic segmentation of any kinds of tumors from CT scans, called CTumorGAN, consisting of a Generator network and a Discriminator network. Specifically, the Generator attempts to generate segmentation results that are close to their corresponding golden standards, while the Discriminator aims to distinguish between generated samples and real tumor ground truths. More importantly, we deliberately design different modules to take into account the well-known obstacles, e.g., severe class imbalance, small tumor localization, and the label noise problem with poor expert annotation quality, and then use these modules to guide the CTumorGAN training process by utilizing multi-level supervision more effectively.

Results

We conduct a comprehensive evaluation on diverse loss functions for tumor segmentation and find that mean square error is more suitable for the CT tumor segmentation task. Furthermore, extensive experiments with multiple evaluation criteria on three well-established datasets, including lung tumor, kidney tumor, and liver tumor databases, also demonstrate that our CTumorGAN achieves stable and competitive performance compared with the state-of-the-art approaches for CT tumor segmentation.

Conclusion

In order to overcome those key challenges arising from CT datasets and solve some of the main problems existing in the current deep learning-based methods, we propose a novel unified CTumorGAN framework, which can be effectively generalized to address any kinds of tumor datasets with superior performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Siegel RL, Miller KD, Jemal A. Cancer statistics. CA Cancer J Clin. 2019;69:7–34.

    Article  Google Scholar 

  2. Wenzel M, Milletari F, Krüger J, Lange C, Schenk M, Apostolova I, et al. Automatic classification of dopamine transporter SPECT: deep convolutional neural networks can be trained to be robust with respect to variable image characteristics. Eur J Nucl Med Mol Imaging. 2019:1–12.

  3. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44 1038/nature14539.

    Article  CAS  Google Scholar 

  4. Isensee F, Petersen J, Kohl SA, Jäger PF, Maier-Hein KH. nnU-Net: breaking the spell on successful medical image segmentation. arXiv preprint arXiv:1904.08128. 2019.

  5. Mohammadi A, Afshar P, Asif A, Farahani K, Kirby J, Oikonomou A, et al. Lung cancer radiomics: highlights from the IEEE video and image processing cup 2018 student competition [SP competitions]. IEEE Signal Process Mag. 2018;36:164–73.

    Article  Google Scholar 

  6. Aerts HJWL, Rios Velazquez E, Leijenaar RT, Parmar C, Grossmann P, Carvalho S, Lambin P. Data from NSCLC-radiomics. The cancer imaging archive. https://wiki.cancerimagingarchive.net/display/Public/NSCLC-Radiomics. 2015.

  7. 2019 kidney tumor segmentation challenge. https://kits19.grand-challenge.org/home/. Accessed 03 Oct 2019.

  8. Heller N, Sathianathen N, Kalapara A, Walczak E, Moore K, Kaluzniak H, Rosenberg J, Blake P, Rengel Z, Oestreich M, Dean J. The KiTS19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes. arXiv preprint arXiv:1904.00445. 2019.

  9. LiTS – liver tumor segmentation challenge. https://competitions.codalab.org/competitions/17094#results. Accessed 12 Oct 2019.

  10. Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, Fu CW, Han X, Heng PA, Hesser J, Kadoury S. The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056. 2019.

  11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27; 2014. p. 2672–80.

    Google Scholar 

  12. Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017. p. 5967–76. https://doi.org/10.1109/CVPR.2017.632.

    Chapter  Google Scholar 

  13. Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 2015;9351:234–241. https://doi.org/10.1007/978-3-319-24574-4_28.

  14. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, et al. CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging. 2019;38:2281–92. https://doi.org/10.1109/TMI.2019.2903562.

    Article  PubMed  Google Scholar 

  15. Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell. 2017;39:2481–95. https://doi.org/10.1109/TPAMI.2016.2644615.

    Article  PubMed  Google Scholar 

  16. Drozdzal M, Chartrand G, Vorontsov E, Shakeri M, Di Jorio L, Tang A, et al. Learning normalized inputs for iterative estimation in medical image segmentation. Med Image Anal. 2018;44:1–13. https://doi.org/10.1016/j.media.2017.11.005.

    Article  PubMed  Google Scholar 

  17. Reed S, Akata Z, Yan X, Logeswaran L, Schiele B, Lee H. Generative adversarial text to image synthesis. In: International Conference on Machine Learning; 2016. p. 1060–9.

    Google Scholar 

  18. Vorontsov E, Tang A, Pal C, Kadoury S. Liver lesion segmentation informed by joint liver segmentation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018): IEEE; 2018. p. 1332–5.

  19. Li X, Chen H, Qi X, Dou Q, Fu CW, Heng PA. H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Trans Med Imaging. 2018;37:2663–74. https://doi.org/10.1109/TMI.2018.2845918.

    Article  PubMed  Google Scholar 

  20. Xue Y, Xu T, Zhang H, Long LR, Huang X. Segan: adversarial network with multi-scale l 1 loss for medical image segmentation. Neuroinformatics. 2018;16:383–92. https://doi.org/10.1007/s12021-018-9377-x.

    Article  PubMed  Google Scholar 

  21. Zhao A, Balakrishnan G, Durand F, Guttag JV, Dalca AV. Data augmentation using learned transformations for one-shot medical image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2019. p. 8543–53.

    Google Scholar 

  22. Yang D, Xu D, Zhou SK, Georgescu B, Chen M, Grbic S, et al. Automatic liver segmentation using an adversarial image-to-image network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer; 2017. p. 507–15. https://doi.org/10.1007/978-3-319-66179-7_58.

    Chapter  Google Scholar 

  23. Milletari F, Navab N, Ahmadi SA. V-net: fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV). IEEE; 2016. P. 565–571. https://doi.org/10.1109/3DV.2016.79.

  24. Myronenko A. 3D MRI brain tumor segmentation using autoencoder regularization. In: International MICCAI Brainlesion workshop. Springer: Cham; 2018. p. 311–20. https://doi.org/10.1007/978-3-030-11726-9_28.

    Chapter  Google Scholar 

  25. Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: a review. Med Image Anal. 2019;58:101552. https://doi.org/10.1016/j.media.2019.101552.

    Article  PubMed  Google Scholar 

  26. McGuinness K, O’connor NE. A comparative evaluation of interactive segmentation algorithms. Pattern Recogn. 2010;43:434–44. https://doi.org/10.1016/j.patcog.2009.03.008.

    Article  Google Scholar 

  27. Zhu X, Rangayyan RM. Detection of the optic disc in images of the retina using the Hough transform. In: 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society: IEEE; 2008. p. 3546–9.

  28. Aquino A, Gegúndez-Arias ME, Marín D. Detecting the optic disc boundary in digital fundus images using morphological, edge detection, and feature extraction techniques. IEEE Trans Med Imaging. 2010;29:1860–9. https://doi.org/10.1109/TMI.2010.2053042.

    Article  PubMed  Google Scholar 

  29. Lee Y, Hara T, Fujita H, Itoh S, Ishigaki T. Automated detection of pulmonary nodules in helical CT images based on an improved template-matching technique. IEEE Trans Med Imaging. 2001;20:595–04. https://doi.org/10.1109/42.932744.

    Article  CAS  PubMed  Google Scholar 

  30. Chen W, Smith R, Ji SY, Ward KR, Najarian K. Automated ventricular systems segmentation in brain CT images by combining low-level segmentation and high-level template matching. BMC Med Inform Decis Mak. 2009;9:S4. https://doi.org/10.1186/1472-6947-9-S1-S4.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Mihaylova A, Georgieva V. Spleen segmentation in MRI sequence images using template matching and active contours. Procedia Comput Sci. 2018;131:15–22.

    Article  Google Scholar 

  32. Boykov YY, Jolly MP. Interactive graph cuts for optimal boundary & region segmentation of objects in ND images. In: Proceedings eighth IEEE International Conference on Computer Vision, vol. 1; 2001. p. 105–12.

    Google Scholar 

  33. Boykov Y, Funka-Lea G. Graph cuts and efficient ND image segmentation. Int J Comput Vis. 2006;70:109–31.

    Article  Google Scholar 

  34. Chen X, Udupa JK, Bagci U, Zhuge Y, Yao J. Medical image segmentation by combining graph cuts and oriented active appearance models. IEEE Trans Image Process. 2012;21:2035–46. https://doi.org/10.1109/TIP.2012.2186306.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Zhang Y, Brady M, Smith S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans Med Imaging. 2001;20:45–57. https://doi.org/10.1109/42.906424.

    Article  CAS  Google Scholar 

  36. Lee CH, Schmidt M, Murtha A, Bistritz A, Sander J, Greiner R. Segmenting brain tumors with random fields and support vector machines. In: In International Workshop on Computer Vision for Biomedical Image Applications, vol. 3765; 2005. p. 469–78.

    Chapter  Google Scholar 

  37. Christ PF, Elshaer MEA, Ettlinger F, Tatavarty S, Bickel M, Bilic P, et al. Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer; 2016. p. 415–23. https://doi.org/10.1007/978-3-319-46723-8_48.

    Chapter  Google Scholar 

  38. Ng HP, Ong SH, Foong KWC, Goh PS, Nowinski WL. Medical image segmentation using k-means clustering and improved watershed algorithm. In 2006 IEEE Southwest Symposium on Image Analysis and Interpretation. IEEE; 2006. P. 61–65.

  39. Kanimozhi M, Bindu CH. Brain MR image segmentation using self organizing map. Brain. 2013;2.

  40. Aganj I, Harisinghani MG, Weissleder R, Fischl B. Unsupervised medical image segmentation based on the local center of mass. Sci Rep. 2018;8:13012. https://doi.org/10.1038/s41598-018-31333-5.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Abramoff MD, Alward WL, Greenlee EC, Shuba L, Kim CY, Fingert JH, et al. Automated segmentation of the optic disc from stereo color photographs using physiologically plausible features. Invest Ophthalmol Vis Sci. 2007;48:1665–73. https://doi.org/10.1167/iovs.06-1081.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Cobzas D, Birkbeck N, Schmidt M, Jagersand M, Murtha A. 3D variational brain tumor segmentation using a high dimensional feature set. In 2007 IEEE 11th International Conference on Computer Vision. IEEE; 2007. P. 1–8.

  43. Geremia E, Clatz O, Menze BH, Konukoglu E, Criminisi A, Ayache N. Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images. NeuroImage. 2011;57:378–90. https://doi.org/10.1016/j.neuroimage.2011.03.080.

    Article  PubMed  Google Scholar 

  44. Cheng J, Liu J, Xu Y, Yin F, Wong DWK, Tan NM, et al. Superpixel classification based optic disc and optic cup segmentation for glaucoma screening. IEEE Trans Med Imaging. 2013;32:1019–32. https://doi.org/10.1109/TMI.2013.2247770.

    Article  PubMed  Google Scholar 

  45. Tian Z, Liu L, Zhang Z, Fei B. Superpixel-based segmentation for 3D prostate MR images. IEEE Trans Med Imaging. 2015;35:791–801. https://doi.org/10.1109/TMI.2015.2496296.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Ciresan D, Giusti A, Gambardella LM, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. In: Advances in Neural Information Processing Systems; 2012. p. 2843–51.

    Google Scholar 

  47. Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging. 2016;35:1240–51. https://doi.org/10.1109/TMI.2016.2538465.

    Article  PubMed  Google Scholar 

  48. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, et al. Brain tumor segmentation with deep neural networks. Med Image Anal. 2017;35:18–31. https://doi.org/10.1016/j.media.2016.05.004.

    Article  PubMed  Google Scholar 

  49. Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. 2017;36:61–78. https://doi.org/10.1016/j.media.2016.10.004.

    Article  PubMed  Google Scholar 

  50. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. Unet++: a nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, vol. 11045; 2018. p. 3–11. https://doi.org/10.1007/978-3-030-00889-5_1.

    Chapter  Google Scholar 

  51. Lin F, Liu C, Xie H, Zha ZJ, Zhang Y. Semantic-embedding and shape-aware u-net for ultrasound eyeball segmentation. In: 2019 IEEE International Conference on Multimedia and Expo (ICME); 2019. p. 892–7. https://doi.org/10.1109/ICME.2019.00158.

    Chapter  Google Scholar 

  52. Ibtehaz N, Rahman MS. MultiResUNet: rethinking the u-net architecture for multimodal biomedical image segmentation. Neural Netw. 2019;121:74–87. https://doi.org/10.1016/j.neunet.2019.08.025.

    Article  PubMed  Google Scholar 

  53. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B. Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999. 2018.

  54. Cui J, Gong K, Guo N, Wu C, Meng X, Kim K, et al. PET image denoising using unsupervised deep learning. Eur J Nucl Med Mol Imaging. 2019:1–10. https://doi.org/10.1007/s00259-019-04468-4.

  55. Goris ML, Zhu HJ, Robinson TE. A critical discussion of computer analysis in medical imaging. Proc Am Thorac Soc. 2007;4:347–9. https://doi.org/10.1513/pats.200701-014HT.

    Article  PubMed  Google Scholar 

  56. Sollini M, Antunovic L, Chiti A, Kirienko M. Towards clinical application of image mining: a systematic review on artificial intelligence and radiomics. Eur J Nucl Med Mol Imaging. 2019;46:2656–72. https://doi.org/10.1007/s00259-019-04372-x.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Kirienko M, Sollini M, Silvestri G, Mognetti S, Voulaz E, Antunovic L, et al. Convolutional neural networks promising in lung cancer T-parameter assessment on baseline FDG-PET/CT. Contrast Media Mol Imaging. 2018;2018. https://doi.org/10.1155/2018/1382309.

  58. Goris ML. Medical image acquisition and processing: clinical validation. Open J Med Imaging. 2014;4:52593. https://doi.org/10.4236/ojmi.2014.44028.

    Article  Google Scholar 

Download references

Funding

Shuchao Pang has been supported in part by an International Macquarie University Research Excellence Scholarship (iMQRES: 2018150). And this work is also supported in part by the National Natural Science Foundation of China (No. 61472416).

Author information

Authors and Affiliations

  1. Department of Computing, Macquarie University, Sydney, NSW, 2109, Australia

    Shuchao Pang, Mehmet A. Orgun, Yan Wang & Guanfeng Liu

  2. School of Electrical and Data Engineering, University of Technology Sydney, Ultimo, NSW, 2007, Australia

    Anan Du

  3. Faculty of Information Technology, Macau University of Science and Technology, Avenida Wai Long, Taipa, 999078, Macau, China

    Mehmet A. Orgun

  4. School of Data and Computer Science, Shandong Women’s University, Jinan, 250014, China

    Zhenmei Yu

  5. Department of Anesthesiology, China-Japan Union Hospital of Jilin University, Changchun, 130012, China

    Yunyun Wang

Authors
  1. Shuchao Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anan Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mehmet A. Orgun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenmei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yunyun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guanfeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Shuchao Pang, Anan Du, Mehmet A. Orgun, and Yunyun Wang. The first draft of the manuscript was written by Shuchao Pang, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mehmet A. Orgun.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Advanced Image Analyses (Radiomics and Artificial Intelligence)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pang, S., Du, A., Orgun, M.A. et al. CTumorGAN: a unified framework for automatic computed tomography tumor segmentation. Eur J Nucl Med Mol Imaging 47, 2248–2268 (2020). https://doi.org/10.1007/s00259-020-04781-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00259-020-04781-3

Keywords

Search

Navigation