Skip to main content
Top

Hint

Swipe to navigate through the chapters of this book

2021 | OriginalPaper | Chapter

Hybrid Aggregation Network for Survival Analysis from Whole Slide Histopathological Images

Authors : Jia-Ren Chang, Ching-Yi Lee, Chi-Chung Chen, Joachim Reischl, Talha Qaiser, Chao-Yuan Yeh

Published in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021

Publisher: Springer International Publishing

share
SHARE

Abstract

Understanding of prognosis and mortality is crucial for evaluating the treatment plans for patients. Recent developments of digital pathology and deep learning bring the possibility of predicting survival time using histopathology whole slide images (WSIs). However, most prevalent methods usually rely on a small set of patches sampled from a WSI and are unable to directly learn from an entire WSI. We argue that a small patch set cannot fully represent patients’ survival risks due to the heterogeneity of tumors; moreover, multiple WSIs from one patient need to be evaluated together. In this paper, we propose a Hybrid Aggregation Network (HANet) to adaptively aggregate information from multiple WSIs of one patient for survival analysis. Specifically, we first extract features from WSIs using a convolutional neural network trained in a self-supervised manner, and further aggregate feature maps using two proposed aggregation modules. The self-aggregation module propagates informative features to the entire WSI, and further abstract features to region representations. The WSI-aggregation module fuses all the region representations from different WSIs of one patient to predict patient-level survival risk. We conduct experiments on two WSI datasets that have accompanying survival data, i.e., NLST and TCGA-LUSC. The proposed method achieves state-of-the-art performances with concordance indices of 0.734 for NLST and 0.668 for TCGA-LUSC, outperforming existing approaches.
Appendix
Available only for authorised users
Literature
1.
go back to reference Bejnordi, B.E., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22), 2199–2210 (2017) CrossRef Bejnordi, B.E., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22), 2199–2210 (2017) CrossRef
2.
go back to reference Courtiol, P., et al.: Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25(10), 1519–1525 (2019) CrossRef Courtiol, P., et al.: Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25(10), 1519–1525 (2019) CrossRef
3.
go back to reference Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
4.
go back to reference Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997) CrossRef Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997) CrossRef
5.
go back to reference Ganeshan, B., Miles, K.A., Young, R.C., Chatwin, C.R.: Hepatic enhancement in colorectal cancer: texture analysis correlates with hepatic hemodynamics and patient survival. Acad. Radiol. 14(12), 1520–1530 (2007) CrossRef Ganeshan, B., Miles, K.A., Young, R.C., Chatwin, C.R.: Hepatic enhancement in colorectal cancer: texture analysis correlates with hepatic hemodynamics and patient survival. Acad. Radiol. 14(12), 1520–1530 (2007) CrossRef
6.
go back to reference Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 317–326 (2016) Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 317–326 (2016)
7.
go back to reference Hao, J., Kim, Y., Mallavarapu, T., Oh, J.H., Kang, M.: Interpretable deep neural network for cancer survival analysis by integrating genomic and clinical data. BMC Med. Genomics 12(10), 1–13 (2019) Hao, J., Kim, Y., Mallavarapu, T., Oh, J.H., Kang, M.: Interpretable deep neural network for cancer survival analysis by integrating genomic and clinical data. BMC Med. Genomics 12(10), 1–13 (2019)
8.
go back to reference He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:​1911.​05722 (2019) He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:​1911.​05722 (2019)
9.
go back to reference He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015) CrossRef He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015) CrossRef
10.
go back to reference He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
11.
go back to reference Kandoth, C., et al.: Mutational landscape and significance across 12 major cancer types. Nature 502(7471), 333–339 (2013) CrossRef Kandoth, C., et al.: Mutational landscape and significance across 12 major cancer types. Nature 502(7471), 333–339 (2013) CrossRef
12.
go back to reference Kim, J.H., et al.: Breast cancer heterogeneity: Mr imaging texture analysis and survival outcomes. Radiology 282(3), 665–675 (2017) CrossRef Kim, J.H., et al.: Breast cancer heterogeneity: Mr imaging texture analysis and survival outcomes. Radiology 282(3), 665–675 (2017) CrossRef
14.
go back to reference Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019) Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)
15.
go back to reference Steck, H., Krishnapuram, B., Dehing-Oberije, C., Lambin, P., Raykar, V.C.: On ranking in survival analysis: Bounds on the concordance index. In: Advances in Neural InformationProcessing Systems, pp. 1209–1216 (2008) Steck, H., Krishnapuram, B., Dehing-Oberije, C., Lambin, P., Raykar, V.C.: On ranking in survival analysis: Bounds on the concordance index. In: Advances in Neural InformationProcessing Systems, pp. 1209–1216 (2008)
16.
go back to reference Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomographic screening. New England J. Med. 365(5), 395–409 (2011) Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomographic screening. New England J. Med. 365(5), 395–409 (2011)
17.
go back to reference Tibshirani, R.: The lasso method for variable selection in the cox model. Stat. Med. 16(4), 385–395 (1997) CrossRef Tibshirani, R.: The lasso method for variable selection in the cox model. Stat. Med. 16(4), 385–395 (1997) CrossRef
18.
go back to reference Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018) Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
19.
go back to reference Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One 15(6), e0233678 (2020) Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One 15(6), e0233678 (2020)
23.
go back to reference Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 544–547. IEEE (2016) Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 544–547. IEEE (2016)
24.
go back to reference Zhu, X., Yao, J., Zhu, F., Huang, J.: Wsisa: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017) Zhu, X., Yao, J., Zhu, F., Huang, J.: Wsisa: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017)
Metadata
Title
Hybrid Aggregation Network for Survival Analysis from Whole Slide Histopathological Images
Authors
Jia-Ren Chang
Ching-Yi Lee
Chi-Chung Chen
Joachim Reischl
Talha Qaiser
Chao-Yuan Yeh
Copyright Year
2021
DOI
https://doi.org/10.1007/978-3-030-87240-3_70

Premium Partner