Abstract
Fusing of information, obtained during the histological slides staining can be helpful for diagnosing and further patient treatment. However, when the slides are being prepared, tissues are subjected to deformations and registration is highly required. Automatic histological image registration is one of the most challenging parts of a histological tissues analysis. The situation is exacerbated by the lack of data and its diversity when the neural networks are susceptible to overfitting and low generalization ability. One of the core sub-problems of histological image registration is the calculation of the initial affine transformation before the final nonrigid registration. We propose a new method of affine registration that requires no histological data to learn and is based on knowledge transfer from nature domain. The results show that it outperforms existing methods by a large margin on most-commonly used histological images registration benchmark in terms of target registration error and produces less outliers. The code is available at https://github.com/VladPyatov/ImgRegWithTransformers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
I. Arganda-Carreras, R. Fernandez-Gonzalez, and C. Ortiz-de-Solorzano, “Automatic registration of serial mammary gland sections,” in 26th Int. Conf. of the IEEE Engineering in Medicine and Biology Society, San Francisco, 2004 (IEEE, 2004), pp. 1691–1694. https://doi.org/10.1109/IEMBS.2004.1403509
J. Borovec, J. Kybic, I. Arganda-Carreras, D. V. Sorokin, G. Bueno, A. V. Khvostikov, S. Bakas, E. I-C. Chang, S. Heldmann, K. Kartasalo, L. Latonen, J. Lotz, M. Noga, S. Pati, K. Punithakumar, P. Ruusuvuori, A. Skalski, N. Tahmasebi, M. Valkonen, L. Venet, Y. Wang, N. Weiss, M. Wodzinski, Yu Xiang, Ya. Xu, Ya. Yan, P. Yushkevich, S. Zhao, and A. Muñoz-Barrutia, “ANHIR: Automatic non-rigid histological image registration challenge,” IEEE Trans. Med. Imaging 39, 3042–3052 (2020). https://doi.org/10.1109/TMI.2020.2986331
C. Ceritoglu, L. Wang, L. D. Selemon, J. G. Csernansky, M. I. Miller, and J. T. Ratnanather, “Large deformation diffeomorphic metric mapping registration of reconstructed 3D histological section images and in vivo MR images,” Front. Human Neurosci. 4, 43 (2010). https://doi.org/10.3389/fnhum.2010.00043
E. Chee, and Z. Wu, “AIRNet: Self-supervised affine registration for 3D medical images using neural networks,” (2018) arXiv:1810.02583 [cs.CV]
L. Cooper, O. Sertel, J. Kong, G. Lozanski, K. Huang, and M. Gurcan, “Feature-based registration of histopathology images with different stains: An application for computerized follicular lymphoma prognosis,” Comput. Methods Programs Biomed. 96, 182–192 (2009). https://doi.org/10.1016/j.cmpb.2009.04.012
B. D. De Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, and I. Išgum, “A deep learning framework for unsupervised affine and deformable image registration,” Med. Image Anal. 52, 128–143 (2019). https://doi.org/10.1016/j.media.2018.11.010
M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM 24, 381–395 (1981). https://doi.org/10.1145/358669.358692
L. Gupta, B. M. Klinkhammer, P. Boor, D. Merhof, and M. Gadermayr, “Stain independent segmentation of whole slide images: A case study in renal histology,” in IEEE 15th Int. Symp. on Biomedical Imaging (ISBI 2018), Washington, D.C., 2018 (IEEE, 2018), pp. 1360–1364. https://doi.org/10.1109/ISBI.2018.8363824
C. D. Kuglin, “The phase correlation image alignment method,” in Proc. Int. Conf. Cybernetics Society (1975), pp. 163–165.
Z. Li, and N. Snavely, “Megadepth: Learning single-view depth prediction from internet photos,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, Utah, 2018 (IEEE, 2018), pp. 2041–2050. https://doi.org/10.1109/CVPR.2018.00218
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision 60, 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94
D. F. G. Obando, A. Frafjord, I. Øynebråten, A. Corthay, J. C. Olivo-Marin, and V. Meas-Yedid, “Multi-staining registration of large histology images,” in IEEE 14th Int. Symp. on Biomedical Imaging (ISBI 2017), Melbourne, Australia, 2017 (IEEE, 2017), pp. 345–348. https://doi.org/10.1109/ISBI.2017.7950534
J. Pichat, J. E. Iglesias, T. Yousry, S. Ourselin, and M. Modat, “A survey of methods for 3D histology reconstruction,” Med. Image Anal. 46, 73–105 (2018). https://doi.org/10.1016/j.media.2018.02.004
J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “LoFTR: Detector-free local feature matching with transformers,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Nashville, Tenn., 2021 (IEEE, 2021), pp. 8922–8931. https://doi.org/10.1109/CVPR46437.2021.00881
S. Tang, J. Zhang, S. Zhu, and P. Tan, “QuadTree attention for vision transformers,” (2022). arXiv:2201.02767 [cs.CV]
M. Wodzinski and H. Müller, “DeepHistReg: Unsupervised deep learning registration framework for differently stained histology samples,” Comput. Methods Programs Biomed. 198, 105799 (2021). https://doi.org/10.1016/j.cmpb.2020.105799
M. Wodzinski and H. Müller, “Learning-based affine registration of histological images,” in Biomedical Image Registration, Ed. by Ž. Špiclin, J. McClelland, J. Kybic, and O. Goksel, Lecture Notes in Computer Science, Vol. 12120 (Springer, 2020), pp. 12–22. https://doi.org/10.1007/978-3-030-50120-4_2
Funding
The work was supported by Russian Science Foundation, grant no. 22-41-02002.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
COMPLIANCE WITH ETHICAL STANDARDS
This article is a completely original work of its authors; it has not been published before and will not be sent to other publications until the PRIA Editorial Board decides not to accept it for publication.
Conflict of Interest
The authors declare that they have no conflicts of interest.
Additional information
Vladislav A. Pyatov received the Bachelor’s degree with honors in applied mathematics and computer science from Lomonosov Moscow State University, Russia, in 2021. Since 2022 he is a master student at the at the Laboratory of Mathematical Methods of Image Processing, Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University. His research interests are optical flow estimation, image registration, biomedical image analysis, computer vison, and deep learning.
Dmitry V. Sorokin received the specialist degree (MSc analog) and Cand. Sci. degree in applied mathematics from Lomonosov Moscow State University, Russia, in 2008 and 2011. From 2012 till 2017 he was a Post-Doc Researcher at the Centre for Biomedical Image Analysis, Masaryk University, Brno, Czech Republic. Since 2018 he works at the Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University as a Senior Researcher at the Laboratory of Mathematical Methods of Image Processing. He won the Prize of the Lomonosov Moscow State University for Young Teachers and Researchers in 2018, 2019, and 2020. In 2018 he won the Moscow Government Award for Young Scientists. He is a member of program committees of several annual international conferences and serves as a reviewer for IEEE Transaction of Medical Imaging. His research interests are mathematical methods of image processing and analysis, image registration, biomedical image analysis, and deep learning.
Rights and permissions
About this article
Cite this article
Pyatov, V.A., Sorokin, D.V. Affine Registration of Histological Images Using Transformer-Based Feature Matching. Pattern Recognit. Image Anal. 32, 626–630 (2022). https://doi.org/10.1134/S1054661822030324
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1054661822030324