nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

Fast Approximate Light Field Volume Rendering: Using Volume Data to Improve Light Field Synthesis via Convolutional Neural Networks

verfasst von : Seán Bruton, David Ganter, Michael Manzke

Erschienen in: Computer Vision, Imaging and Computer Graphics Theory and Applications

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Volume visualization pipelines have the potential to be improved by the use of light field display technology, allowing enhanced perceptual qualities. However, these displays will require a significant increase in pixels to be rendered at interactive rates. Volume rendering makes use of ray-tracing techniques, which makes this resolution increase challenging for modest hardware. We demonstrate in this work an approach to synthesize the majority of the viewpoints in the light field using a small set of rendered viewpoints via a convolutional neural network. We show that synthesis performance can be further improved by allowing the network access to the volume data itself. To perform this efficiently, we propose a range of approaches and evaluate them against two datasets collected for this task. These approaches all improve synthesis performance and avoid the use of expensive 3D convolutional operations. With this approach, we improve light field volume rendering times by a factor of 8 for our test case.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel A Layered Approach to Lightweight Toolchaining in Visual Analytics

Nächstes Kapitel A Reproducibility Study for Visual MRSI Data Analytics

Agus, M., et al.: An interactive 3D medical visualization system based on a light field display. Vis. Comput. 25(9), 883–893 (2009). https://doi.org/10.1007/s00371-009-0311-yCrossRef

Agus, M., Gobbetti, E., Guitián, J.A.I., Marton, F., Pintore, G.: GPU accelerated direct volume rendering on an interactive light field display. Comput. Graph. Forum 27(2), 231–240 (2008). https://doi.org/10.1111/j.1467-8659.2008.01120.xCrossRef

Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., Gould, S.: Dynamic image networks for action recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3034–3042 (2016). https://doi.org/10.1109/CVPR.2016.331

Birklbauer, C., Bimber, O.: Light-field supported fast volume rendering. In: ACM SIGGRAPH 2012 Posters on - SIGGRAPH 2012, p. 1. ACM Press, Los Angeles, California (2012). https://doi.org/10.1145/2342896.2343040

Bruton, S., Ganter, D., Manzke, M.: Synthesising light field volumetric visualizations in real-time using a compressed volume representation. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP. pp. 96–105. SciTePress (2019). https://doi.org/10.5220/0007407200960105

Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv:1512.03012 [cs] (2015)

Drebin, R.A., Carpenter, L., Hanrahan, P.: Volume rendering. In: Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques, pp. 65–74. SIGGRAPH 1988, ACM, New York, NY, USA (1988). https://doi.org/10.1145/54852.378484

Engelmann, F., Kontogianni, T., Hermans, A., Leibe, B.: Exploring spatial context for 3D semantic segmentation of point clouds. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). pp. 716–724 (2017). https://doi.org/10.1109/ICCVW.2017.90

Favalora, G.E.: Volumetric 3D displays and application infrastructure. Computer 38(8), 37–44 (2005). https://doi.org/10.1109/MC.2005.276CrossRef

10.

Fernando, B., Gavves, E.M., Oramas, J., Ghodrati, A., Tuytelaars, T.: Rank pooling for action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 773–787 (2017). https://doi.org/10.1109/TPAMI.2016.2558148CrossRef

11.

Fishman, E.K., Ney, D.R., Heath, D.G., Corl, F.M., Horton, K.M., Johnson, P.T.: Volume rendering versus maximum intensity projection in CT angiography: what works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North America, Inc 26(3), 905–922 (2006). https://doi.org/10.1148/rg.263055186CrossRef

12.

Hadwiger, M., Kratz, A., Sigg, C., Bühler, K.: GPU-accelerated deep shadow maps for direct volume rendering. In: Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, pp. 49–52. GH 2006, ACM, New York, NY, USA (2006). https://doi.org/10.1145/1283900.1283908

13.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

14.

Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35(6), 193:1–193:10 (2016). https://doi.org/10.1145/2980179.2980251CrossRef

15.

Kühnapfel, U., Çakmak, H.K., Maaß, H.: Endoscopic surgery training using virtual reality and deformable tissue simulation. Comput. Graph. 24(5), 671–682 (2000). https://doi.org/10.1016/S0097-8493(00)00070-4CrossRef

16.

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 [cs] (2014)

17.

Klokov, R., Lempitsky, V.: Escape from cells: deep kd-networks for the recognition of 3D point cloud models. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 863–872 (2017). https://doi.org/10.1109/ICCV.2017.99

18.

Kniss, J., Kindlmann, G., Hansen, C.: Multidimensional transfer functions for interactive volume rendering. IEEE Trans. Vis. Comput. Graph. 8(3), 270–285 (2002). https://doi.org/10.1109/TVCG.2002.1021579CrossRef

19.

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., New York (2012)

20.

Lacroute, P., Levoy, M.: Fast volume rendering using a shear-warp factorization of the viewing transformation. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, pp. 451–458. SIGGRAPH 1994, ACM, New York, NY, USA (1994). https://doi.org/10.1145/192161.192283

21.

Lanman, D., Luebke, D.: Near-eye light field displays. In: ACM SIGGRAPH 2013 Emerging Technologies, p. 11:1. SIGGRAPH 2013, ACM, New York, NY, USA (2013). https://doi.org/10.1145/2503368.2503379

22.

Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31–42. SIGGRAPH 1996, ACM, New York, NY, USA (1996). https://doi.org/10.1145/237170.237199

23.

Li, Y., Pirk, S., Su, H., Qi, C.R., Guibas, L.J.: FPNN: field probing neural networks for 3D data. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 307–315. Curran Associates, Inc., New York (2016)

24.

Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 [cs] (2013)

25.

Liu, T.Y.: Learning to rank for information retrieval. Found. Trends Inf. Retr. 3(3), 225–331 (2009). https://doi.org/10.1561/1500000016CrossRef

26.

Liu, Z., Yeh, R.A., Tang, X., Liu, Y., Agarwala, A.: Video frame synthesis using deep voxel flow. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4473–4481 (2017). https://doi.org/10.1109/ICCV.2017.478

27.

Ljung, P., Krüger, J., Groller, E., Hadwiger, M., Hansen, C.D., Ynnerman, A.: State of the art in transfer functions for direct volume rendering. Comput. Graph. Forum 35(3), 669–691 (2016). https://doi.org/10.1111/cgf.12934CrossRef

28.

Mora, B., Maciejewski, R., Chen, M., Ebert, D.S.: Visualization and computer graphics on isotropically emissive volumetric displays. IEEE Trans. Vis. Comput. Graph. 15(2), 221–234 (2009). https://doi.org/10.1109/TVCG.2008.99CrossRef

29.

Mueller, K., Yagel, R.: Fast perspective volume rendering with splatting by utilizing a ray-driven approach. In: Proceedings of Seventh Annual IEEE Visualization 1996, pp. 65–72 (1996). https://doi.org/10.1109/VISUAL.1996.567608

30.

Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 261–270 (2017). https://doi.org/10.1109/ICCV.2017.37

31.

Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 702–711 (2017). https://doi.org/10.1109/CVPR.2017.82

32.

Philips, S., Hlawitschka, M., Scheuermann, G.: Slice-based visualization of brain fiber bundles - a lic-based approach. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, pp. 281–288. SciTePress (2018). https://doi.org/10.5220/0006619402810288

33.

Qi, C.R., Su, H., Kaichun, M., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16

34.

Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3D data. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5648–5656 (2016). https://doi.org/10.1109/CVPR.2016.609

35.

Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5099–5108. Curran Associates, Inc., New York (2017)

36.

Riegler, G., Ulusoy, A.O., Geiger, A.: OctNet: Learning deep 3D representations at high resolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6620–6629 (2017). https://doi.org/10.1109/CVPR.2017.701

37.

Salama, C.R.: GPU-based monte-carlo volume raycasting. In: 15th Pacific Conference on Computer Graphics and Applications (PG 2007). pp. 411–414 (2007). https://doi.org/10.1109/PG.2007.27

38.

Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004). https://doi.org/10.1023/B:STCO.0000035301.49549.88MathSciNetCrossRef

39.

Srinivasan, P.P., Wang, T., Sreelal, A., Ramamoorthi, R., Ng, R.: Learning to synthesize a 4D RGBD light field from a single image. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2262–2270 (2017). https://doi.org/10.1109/ICCV.2017.246

40.

Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 945–953. ICCV 2015, IEEE Computer Society, Washington, DC, USA (2015). https://doi.org/10.1109/ICCV.2015.114

41.

Sunden, E., et al.: Inviwo - an extensible, multi-purpose visualization framework. In: 2015 IEEE Scientific Visualization Conference (SciVis), pp. 163–164 (2015). https://doi.org/10.1109/SciVis.2015.7429514

42.

Tuzel, O., Liu, M.-Y., Taguchi, Y., Raghunathan, A.: Learning to rank 3D features. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 520–535. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_34CrossRef

43.

Wang, P., Li, W., Gao, Z., Zhang, Y., Tang, C., Ogunbona, P.: Scene flow to action map: a new representation for RGB-D based action recognition with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 416–425 (2017). https://doi.org/10.1109/CVPR.2017.52

44.

Wang, P.S., Liu, Y., Guo, Y.X., Sun, C.Y., Tong, X.: O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073608CrossRef

45.

Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R.: Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. 31(4), 80:1–80:11 (2012). https://doi.org/10.1145/2185520.2185576CrossRef

46.

Xie, J., Dai, G., Zhu, F., Wong, E.K., Fang, Y.: Deepshape: deep-learned shape descriptor for 3D shape retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1335–1345 (2017). https://doi.org/10.1109/TPAMI.2016.2596722CrossRef

47.

Zhang, Y., Dong, Z., Ma, K.: Real-time volume rendering in dynamic lighting environments using precomputed photon mapping. IEEE Trans. Vis. Comput. Graph. 19(8), 1317–1330 (2013). https://doi.org/10.1109/TVCG.2013.17CrossRef

48.

Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. 37(4), 65:1–65:12 (2018). https://doi.org/10.1145/3197517.3201323CrossRef

49.

Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18CrossRef

Titel: Fast Approximate Light Field Volume Rendering: Using Volume Data to Improve Light Field Synthesis via Convolutional Neural Networks
verfasst von: Seán Bruton
David Ganter
Michael Manzke
Verlag: Springer International Publishing
Buch: Computer Vision, Imaging and Computer Graphics Theory and Applications
Print ISBN: 978-3-030-41589-1

Electronic ISBN: 978-3-030-41590-7

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-41590-7_14

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"