Skip to main content
main-content

Tipp

Weitere Kapitel dieses Buchs durch Wischen aufrufen

2020 | OriginalPaper | Buchkapitel

Are Pathologist-Defined Labels Reproducible? Comparison of the TUPAC16 Mitotic Figure Dataset with an Alternative Set of Labels

verfasst von: Christof A. Bertram, Mitko Veta, Christian Marzahl, Nikolas Stathonikos, Andreas Maier, Robert Klopfleisch, Marc Aubreville

Erschienen in: Interpretable and Annotation-Efficient Learning for Medical Image Computing

Verlag: Springer International Publishing

share
TEILEN

Abstract

Pathologist-defined labels are the gold standard for histopathological data sets, regardless of well-known limitations in consistency for some tasks. To date, some datasets on mitotic figures are available and were used for development of promising deep learning-based algorithms. In order to assess robustness of those algorithms and reproducibility of their methods it is necessary to test on several independent datasets. The influence of different labeling methods of these available datasets is currently unknown. To tackle this, we present an alternative set of labels for the images of the auxiliary mitosis dataset of the TUPAC16 challenge. Additional to manual mitotic figure screening, we used a novel, algorithm-aided labeling process, that allowed to minimize the risk of missing rare mitotic figures in the images. All potential mitotic figures were independently assessed by two pathologists. The novel, publicly available set of labels contains 1,999 mitotic figures (+28.80%) and additionally includes 10,483 labels of cells with high similarities to mitotic figures (hard examples). We found significant difference comparing \(F_1\) scores between the original label set (0.549) and the new alternative label set (0.735) using a standard deep learning object detection architecture. The models trained on the alternative set showed higher overall confidence values, suggesting a higher overall label consistency. Findings of the present study show that pathologists-defined labels may vary significantly resulting in notable difference in the model performance. Comparison of deep learning-based algorithms between independent datasets with different labeling methods should be done with caution.
Literatur
2.
Zurück zum Zitat Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35(5), 1313–1321 (2016) CrossRef Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35(5), 1313–1321 (2016) CrossRef
5.
Zurück zum Zitat Bertram, C.A., Aubreville, M., Marzahl, C., Maier, A., Klopfleisch, R.: A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci. Data 6(274), 1–9 (2019) Bertram, C.A., Aubreville, M., Marzahl, C., Maier, A., Klopfleisch, R.: A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci. Data 6(274), 1–9 (2019)
6.
Zurück zum Zitat Li, C., Wang, X., Liu, W., Latecki, L.J., Wang, B., Huang, J.: Weakly supervised mitosis detection in breast histopathology images using concentric loss. Med. Imaging Anal. 53, 165–178 (2019) CrossRef Li, C., Wang, X., Liu, W., Latecki, L.J., Wang, B., Huang, J.: Weakly supervised mitosis detection in breast histopathology images using concentric loss. Med. Imaging Anal. 53, 165–178 (2019) CrossRef
7.
Zurück zum Zitat Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017) Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
8.
Zurück zum Zitat Malon, C., et al.: Mitotic figure recognition: agreement among pathologists and computerized detector. Anal. Cell Pathol. 35(2), 97–100 (2012) CrossRef Malon, C., et al.: Mitotic figure recognition: agreement among pathologists and computerized detector. Anal. Cell Pathol. 35(2), 97–100 (2012) CrossRef
9.
Zurück zum Zitat Marzahl, C., et al.: Are fast labeling methods reliable? A case study of computer-aided expert annotations on microscopy slides. Preprint, arXiv:​2004.​05838 (2020) Marzahl, C., et al.: Are fast labeling methods reliable? A case study of computer-aided expert annotations on microscopy slides. Preprint, arXiv:​2004.​05838 (2020)
10.
Zurück zum Zitat Roux, L., et al.: MITOS & ATYPIA - detection of mitosis and evaluation of nuclear atypia score in breast cancer histological images. Agency Sci., Technol. Res. Inst. Infocom Res., Singapore, Image Pervasive Access Lab IPAL (2014) Roux, L., et al.: MITOS & ATYPIA - detection of mitosis and evaluation of nuclear atypia score in breast cancer histological images. Agency Sci., Technol. Res. Inst. Infocom Res., Singapore, Image Pervasive Access Lab IPAL (2014)
11.
Zurück zum Zitat Roux, L., et al.: Mitosis detection in breast cancer histological images An ICPR 2012 contest. J. Pathol. Inf. 4(1), 8 (2013) CrossRef Roux, L., et al.: Mitosis detection in breast cancer histological images An ICPR 2012 contest. J. Pathol. Inf. 4(1), 8 (2013) CrossRef
12.
Zurück zum Zitat Tellez, D., et al.: Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans. Med. Imaging 37(9), 2126–2136 (2018) CrossRef Tellez, D., et al.: Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans. Med. Imaging 37(9), 2126–2136 (2018) CrossRef
13.
Zurück zum Zitat Veta, M., et al.: Assessment of algorithms for mitosis detection in breast cancerhistopathology images. Med. Imaging Anal. 20(1), 237–248 (2015) CrossRef Veta, M., et al.: Assessment of algorithms for mitosis detection in breast cancerhistopathology images. Med. Imaging Anal. 20(1), 237–248 (2015) CrossRef
14.
Zurück zum Zitat Veta, M., et al.: Predicting breast tumor proliferation from whole-slide images - the TUPAC16 challenge. Med. Imaging Anal. 54, 111–121 (2019) CrossRef Veta, M., et al.: Predicting breast tumor proliferation from whole-slide images - the TUPAC16 challenge. Med. Imaging Anal. 54, 111–121 (2019) CrossRef
15.
Zurück zum Zitat Veta, M., Van Diest, P.J., Jiwa, M., Al-Janabi, S., Pluim, J.P.: Mitosis counting in breast cancer: object-level interobserver agreement and comparison to an automatic method. PloS One 11(8), e0161286 (2016) CrossRef Veta, M., Van Diest, P.J., Jiwa, M., Al-Janabi, S., Pluim, J.P.: Mitosis counting in breast cancer: object-level interobserver agreement and comparison to an automatic method. PloS One 11(8), e0161286 (2016) CrossRef
16.
Zurück zum Zitat Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I. IEEE (2001) Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I. IEEE (2001)
Metadaten
Titel
Are Pathologist-Defined Labels Reproducible? Comparison of the TUPAC16 Mitotic Figure Dataset with an Alternative Set of Labels
verfasst von
Christof A. Bertram
Mitko Veta
Christian Marzahl
Nikolas Stathonikos
Andreas Maier
Robert Klopfleisch
Marc Aubreville
Copyright-Jahr
2020
DOI
https://doi.org/10.1007/978-3-030-61166-8_22

Premium Partner