Skip to main content
Top
Published in: Journal of Visualization 6/2021

19-08-2021 | Regular Paper

IsoExplorer: an isosurface-driven framework for 3D shape analysis of biomedical volume data

Authors: Haoran Dai, Yubo Tao, Xiangyang He, Hai Lin

Published in: Journal of Visualization | Issue 6/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The high-resolution scanning devices developed in recent decades provide biomedical volume datasets that support the study of molecular structure and drug design. Isosurface analysis is an important tool in these studies, and the key is to construct suitable description vectors to support subsequent tasks, such as classification and retrieval. Traditional methods based on handcrafted features are insufficient for dealing with complex structures, while deep learning-based approaches have high memory and computation costs when dealing directly with volume data. To address these problems, we propose IsoExplorer, an isosurface-driven framework for 3D shape analysis of biomedical volume data. We first extract isosurfaces from volume data and split them into individual 3D shapes according to their connectivity. Then, we utilize octree-based convolution to design a variational autoencoder model that learns the latent representations of the shape. Finally, these latent representations are used for low-dimensional isosurface representation and shape retrieval. We demonstrate the effectiveness and usefulness of IsoExplorer via isosurface similarity analysis, shape retrieval of real-world data, and comparison with existing methods.

Graphic abstract

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literature
go back to reference Bai S, Bai X, Zhou Z, Zhang Z, Tian Q, Latecki LJ (2017) Gift: towards scalable 3d shape retrieval. IEEE Trans Multim 19(6):1257–1271CrossRef Bai S, Bai X, Zhou Z, Zhang Z, Tian Q, Latecki LJ (2017) Gift: towards scalable 3d shape retrieval. IEEE Trans Multim 19(6):1257–1271CrossRef
go back to reference Berger M, Li J, Levine JA (2018) A generative model for volume rendering. IEEE Trans Vis Comput Graph 25(4):1636–1650CrossRef Berger M, Li J, Levine JA (2018) A generative model for volume rendering. IEEE Trans Vis Comput Graph 25(4):1636–1650CrossRef
go back to reference Bruckner S, Möller T (2010) Isosurface similarity maps. Comput Graph Forum 29(3):773–782CrossRef Bruckner S, Möller T (2010) Isosurface similarity maps. Comput Graph Forum 29(3):773–782CrossRef
go back to reference Castillo-Barnes D, Martinez-Murcia FJ, Ortiz A, Salas-Gonzalez D, RamÍrez J, Górriz JM (2020) Morphological characterization of functional brain imaging by isosurface analysis in parkinson’s disease. Int J Neural Syst 30(9):2050044 Castillo-Barnes D, Martinez-Murcia FJ, Ortiz A, Salas-Gonzalez D, RamÍrez J, Górriz JM (2020) Morphological characterization of functional brain imaging by isosurface analysis in parkinson’s disease. Int J Neural Syst 30(9):2050044
go back to reference Cheng HC, Cardone A, Jain S, Krokos E, Narayan K, Subramaniam S, Varshney A (2018) Deep-learning-assisted volume visualization. IEEE Trans Vis Comput Graph 25(2):1378–1391CrossRef Cheng HC, Cardone A, Jain S, Krokos E, Narayan K, Subramaniam S, Varshney A (2018) Deep-learning-assisted volume visualization. IEEE Trans Vis Comput Graph 25(2):1378–1391CrossRef
go back to reference El-Baz A, Nitzken M, Khalifa F, Elnakib A, Gimel’farb G, Falk R, El-Ghar MA (2011) 3d shape analysis for early diagnosis of malignant lung nodules. In: Information processing in medical imaging, pp 772–783 El-Baz A, Nitzken M, Khalifa F, Elnakib A, Gimel’farb G, Falk R, El-Ghar MA (2011) 3d shape analysis for early diagnosis of malignant lung nodules. In: Information processing in medical imaging, pp 772–783
go back to reference Engel D, Ropinski T (2021) Deep volumetric ambient occlusion. IEEE Trans Vis Comput Graph 27(2):1268–1278CrossRef Engel D, Ropinski T (2021) Deep volumetric ambient occlusion. IEEE Trans Vis Comput Graph 27(2):1268–1278CrossRef
go back to reference Gao L, Lai YK, Liang D, Chen SY, Xia S (2016) Efficient and flexible deformation representation for data-driven surface modeling. ACM Trans Graph 35(5):1–17CrossRef Gao L, Lai YK, Liang D, Chen SY, Xia S (2016) Efficient and flexible deformation representation for data-driven surface modeling. ACM Trans Graph 35(5):1–17CrossRef
go back to reference Gobeil S, Janowska K, McDowell S, Mansouri K, Parks R, Stalls V, Kopp MF, Manne K, Saunders KO, Edwards RJ et al (2021) Effect of natural mutations of sars-cov-2 on spike structure, conformation and antigenicity. bioRxiv Gobeil S, Janowska K, McDowell S, Mansouri K, Parks R, Stalls V, Kopp MF, Manne K, Saunders KO, Edwards RJ et al (2021) Effect of natural mutations of sars-cov-2 on spike structure, conformation and antigenicity. bioRxiv
go back to reference Han J, Tao J, Wang C (2018) Flownet: a deep learning framework for clustering and selection of streamlines and stream surfaces. IEEE Trans Vis Comput Graph 26(4):1732–1744 Han J, Tao J, Wang C (2018) Flownet: a deep learning framework for clustering and selection of streamlines and stream surfaces. IEEE Trans Vis Comput Graph 26(4):1732–1744
go back to reference Han Z, Liu Z, Han J, Vong CM, Bu S, Chen CLP (2019) Unsupervised learning of 3-d local features from raw voxels based on a novel permutation voxelization strategy. IEEE Trans Cybern 49(2):481–494CrossRef Han Z, Liu Z, Han J, Vong CM, Bu S, Chen CLP (2019) Unsupervised learning of 3-d local features from raw voxels based on a novel permutation voxelization strategy. IEEE Trans Cybern 49(2):481–494CrossRef
go back to reference Han J, Zheng H, Xing Y, Chen DZ, Wang C (2021) V2v: a deep learning approach to variable-to-variable selection and translation for multivariate time-varying data. IEEE Trans Vis Comput Graph 27(2):1290–1300CrossRef Han J, Zheng H, Xing Y, Chen DZ, Wang C (2021) V2v: a deep learning approach to variable-to-variable selection and translation for multivariate time-varying data. IEEE Trans Vis Comput Graph 27(2):1290–1300CrossRef
go back to reference He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: Proceedings of the European conference on computer vision (ECCV), pp 630–645 He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: Proceedings of the European conference on computer vision (ECCV), pp 630–645
go back to reference He W, Wang J, Guo H, Wang KC, Shen HW, Raj M, Nashed YS, Peterka T (2019) Insitunet: deep image synthesis for parameter space exploration of ensemble simulations. IEEE Trans Vis Comput Graph 26(1):23–33 He W, Wang J, Guo H, Wang KC, Shen HW, Raj M, Nashed YS, Peterka T (2019) Insitunet: deep image synthesis for parameter space exploration of ensemble simulations. IEEE Trans Vis Comput Graph 26(1):23–33
go back to reference Higgins I, Matthey L, Pal A, Burgess C, Glorot X, Botvinick M, Mohamed S, Lerchner A (2017) beta-vae: learning basic visual concepts with a constrained variational framework. In: International conference on learning representations Higgins I, Matthey L, Pal A, Burgess C, Glorot X, Botvinick M, Mohamed S, Lerchner A (2017) beta-vae: learning basic visual concepts with a constrained variational framework. In: International conference on learning representations
go back to reference Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456 Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456
go back to reference Kirchdoerfer RN, Wang N, Pallesen J, Wrapp D, Turner HL, Cottrell CA, Corbett KS, Graham BS, McLellan JS, Ward AB (2018) Stabilized coronavirus spikes are resistant to conformational changes induced by receptor recognition or proteolysis. Sci Rep 8(1):1–11 Kirchdoerfer RN, Wang N, Pallesen J, Wrapp D, Turner HL, Cottrell CA, Corbett KS, Graham BS, McLellan JS, Ward AB (2018) Stabilized coronavirus spikes are resistant to conformational changes induced by receptor recognition or proteolysis. Sci Rep 8(1):1–11
go back to reference Kruger J, Westermann R (2003) Acceleration techniques for gpu-based volume rendering. In: IEEE visualization, pp 287–292 Kruger J, Westermann R (2003) Acceleration techniques for gpu-based volume rendering. In: IEEE visualization, pp 287–292
go back to reference Lawson CL, Patwardhan A, Baker ML, Hryc C, Garcia ES, Hudson BP, Lagerstedt I, Ludtke SJ, Pintilie G, Sala R et al (2016) Emdatabank unified data resource for 3dem. Nucleic Acids Res 44(D1):D396–D403CrossRef Lawson CL, Patwardhan A, Baker ML, Hryc C, Garcia ES, Hudson BP, Lagerstedt I, Ludtke SJ, Pintilie G, Sala R et al (2016) Emdatabank unified data resource for 3dem. Nucleic Acids Res 44(D1):D396–D403CrossRef
go back to reference Leskovec J, Rajaraman A, Ullman JD (2020) Mining of massive data sets. Cambridge University Press, CambridgeCrossRef Leskovec J, Rajaraman A, Ullman JD (2020) Mining of massive data sets. Cambridge University Press, CambridgeCrossRef
go back to reference Li Y, Wang C, Shene CK (2014) Streamline similarity analysis using bag-of-features. In: Visualization and data analysis, vol 9017, p 90170N Li Y, Wang C, Shene CK (2014) Streamline similarity analysis using bag-of-features. In: Visualization and data analysis, vol 9017, p 90170N
go back to reference Liu J, Gao Y, Shan G, Chi X (2019) Vasem: visual analytics system for electron microscopy data bank. J Vis 22(6):1145–1159CrossRef Liu J, Gao Y, Shan G, Chi X (2019) Vasem: visual analytics system for electron microscopy data bank. J Vis 22(6):1145–1159CrossRef
go back to reference Loncaric S (1998) A survey of shape analysis techniques. Patt Recogn 31(8):983–1001CrossRef Loncaric S (1998) A survey of shape analysis techniques. Patt Recogn 31(8):983–1001CrossRef
go back to reference Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. ACM Siggraph Comput Graph 21(4):163–169CrossRef Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. ACM Siggraph Comput Graph 21(4):163–169CrossRef
go back to reference Melero R, Sorzano COS, Foster B, Vilas JL, Martínez M, Marabini R, Ramírez-Aportela E, Sanchez-Garcia R, Herreros D, Del Caño L et al (2020) Continuous flexibility analysis of sars-cov-2 spike prefusion structures. IUCrJ 7(6) Melero R, Sorzano COS, Foster B, Vilas JL, Martínez M, Marabini R, Ramírez-Aportela E, Sanchez-Garcia R, Herreros D, Del Caño L et al (2020) Continuous flexibility analysis of sars-cov-2 spike prefusion structures. IUCrJ 7(6)
go back to reference Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: International conference on machine learning, pp 807–814 Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: International conference on machine learning, pp 807–814
go back to reference Park YJ, Walls AC, Wang Z, Sauer MM, Li W, Tortorici MA, Bosch BJ, DiMaio F, Veesler D (2019) Structures of mers-cov spike glycoprotein in complex with sialoside attachment receptors. Nat Struct Mol Biol 26(12):1151–1157CrossRef Park YJ, Walls AC, Wang Z, Sauer MM, Li W, Tortorici MA, Bosch BJ, DiMaio F, Veesler D (2019) Structures of mers-cov spike glycoprotein in complex with sialoside attachment receptors. Nat Struct Mol Biol 26(12):1151–1157CrossRef
go back to reference Porter WP, Xing Y, von Ohlen BR, Han J, Wang C (2019) A deep learning approach to selecting representative time steps for time-varying multivariate data. In: IEEE visualization conference (VIS), pp 1–5 Porter WP, Xing Y, von Ohlen BR, Han J, Wang C (2019) A deep learning approach to selecting representative time steps for time-varying multivariate data. In: IEEE visualization conference (VIS), pp 1–5
go back to reference Rostami R, Bashiri FS, Rostami B, Yu Z (2019) A survey on data-driven 3d shape descriptors. Comput Graph Forum 38(1):356–393CrossRef Rostami R, Bashiri FS, Rostami B, Yu Z (2019) A survey on data-driven 3d shape descriptors. Comput Graph Forum 38(1):356–393CrossRef
go back to reference Shang J, Zheng Y, Yang Y, Liu C, Geng Q, Tai W, Du L, Zhou Y, Zhang W, Li F (2018) Cryo-electron microscopy structure of porcine deltacoronavirus spike protein in the prefusion state. J Virol 92(4) Shang J, Zheng Y, Yang Y, Liu C, Geng Q, Tai W, Du L, Zhou Y, Zhang W, Li F (2018) Cryo-electron microscopy structure of porcine deltacoronavirus spike protein in the prefusion state. J Virol 92(4)
go back to reference Shinagawa Y, Kunii TL, Kergosien YL (1991) Surface coding based on morse theory. IEEE Comput Graph Appl 11(5):66–78CrossRef Shinagawa Y, Kunii TL, Kergosien YL (1991) Surface coding based on morse theory. IEEE Comput Graph Appl 11(5):66–78CrossRef
go back to reference Tabia H, Laga H, Picard D, Gosselin PH (2014) Covariance descriptors for 3d shape matching and retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) Tabia H, Laga H, Picard D, Gosselin PH (2014) Covariance descriptors for 3d shape matching and retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
go back to reference Tao J, Imre M, Wang C, Chawla NV, Guo H, Sever G, Kim SH (2019) Exploring time-varying multivariate volume data using matrix of isosurface similarity maps. IEEE Trans Vis Comput Graph 25(1) Tao J, Imre M, Wang C, Chawla NV, Guo H, Sever G, Kim SH (2019) Exploring time-varying multivariate volume data using matrix of isosurface similarity maps. IEEE Trans Vis Comput Graph 25(1)
go back to reference Thomas DM, Natarajan V (2011) Symmetry in scalar field topology. IEEE Trans Vis Comput Graph 17(12):2035–2044CrossRef Thomas DM, Natarajan V (2011) Symmetry in scalar field topology. IEEE Trans Vis Comput Graph 17(12):2035–2044CrossRef
go back to reference Tkachev G, Frey S, Ertl T (2021) Local prediction models for spatiotemporal volume visualization. IEEE Trans Vis Comput Graph 27(7):3091–3108CrossRef Tkachev G, Frey S, Ertl T (2021) Local prediction models for spatiotemporal volume visualization. IEEE Trans Vis Comput Graph 27(7):3091–3108CrossRef
go back to reference Wang PS, Liu Y, Guo YX, Sun CY, Tong X (2017) O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Trans Graph 36(4):1–11CrossRef Wang PS, Liu Y, Guo YX, Sun CY, Tong X (2017) O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Trans Graph 36(4):1–11CrossRef
go back to reference Wang Z, Esturo JM, Seidel HP, Weinkauf T (2017) Stream line-based pattern search in flows. Comput Graph Forum 36(8):7–18CrossRef Wang Z, Esturo JM, Seidel HP, Weinkauf T (2017) Stream line-based pattern search in flows. Comput Graph Forum 36(8):7–18CrossRef
go back to reference Wang Z, Seidel HP, Weinkauf T (2015) Multi-field pattern matching based on sparse feature sampling. IEEE Trans Vis Comput Graph 22(1):807–816CrossRef Wang Z, Seidel HP, Weinkauf T (2015) Multi-field pattern matching based on sparse feature sampling. IEEE Trans Vis Comput Graph 22(1):807–816CrossRef
go back to reference Weiss S, Chu M, Thuerey N, Westermann R (2019) Volumetric isosurface rendering with deep learning-based super-resolution. arXiv preprint arXiv:1906.06520 Weiss S, Chu M, Thuerey N, Westermann R (2019) Volumetric isosurface rendering with deep learning-based super-resolution. arXiv preprint arXiv:​1906.​06520
go back to reference Xiao YP, Lai YK, Zhang FL, Li C, Gao L (2020) A survey on deep geometry learning: from a representation perspective. Comput Vis Media 6(2):113–133CrossRef Xiao YP, Lai YK, Zhang FL, Li C, Gao L (2020) A survey on deep geometry learning: from a representation perspective. Comput Vis Media 6(2):113–133CrossRef
go back to reference Xie J, Fang Y, Zhu F, Wong E (2015) Deepshape: deep learned shape descriptor for 3d shape matching and retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1275–1283 Xie J, Fang Y, Zhu F, Wong E (2015) Deepshape: deep learned shape descriptor for 3d shape matching and retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1275–1283
go back to reference Yang C, Li Y, Liu C, Yuan X (2019) Deep learning-based viewpoint recommendation in volume visualization. J Vis 22(5):991–1003CrossRef Yang C, Li Y, Liu C, Yuan X (2019) Deep learning-based viewpoint recommendation in volume visualization. J Vis 22(5):991–1003CrossRef
go back to reference Yip KM, Fischer N, Paknia E, Chari A, Stark H (2020) Atomic-resolution protein structure determination by cryo-em. Nature 587(7832):157–161CrossRef Yip KM, Fischer N, Paknia E, Chari A, Stark H (2020) Atomic-resolution protein structure determination by cryo-em. Nature 587(7832):157–161CrossRef
go back to reference Zaharescu A, Boyer E, Varanasi K, Horaud R (2009) Surface feature detection and description with applications to mesh matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 373–380 Zaharescu A, Boyer E, Varanasi K, Horaud R (2009) Surface feature detection and description with applications to mesh matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 373–380
Metadata
Title
IsoExplorer: an isosurface-driven framework for 3D shape analysis of biomedical volume data
Authors
Haoran Dai
Yubo Tao
Xiangyang He
Hai Lin
Publication date
19-08-2021
Publisher
Springer Berlin Heidelberg
Published in
Journal of Visualization / Issue 6/2021
Print ISSN: 1343-8875
Electronic ISSN: 1875-8975
DOI
https://doi.org/10.1007/s12650-021-00770-2

Other articles of this Issue 6/2021

Journal of Visualization 6/2021 Go to the issue

Premium Partner