Abstract
This paper describes a method for scene reconstruction of complex, detailed environments from 3D light fields. Densely sampled light fields in the order of 109 light rays allow us to capture the real world in unparalleled detail, but efficiently processing this amount of data to generate an equally detailed reconstruction represents a significant challenge to existing algorithms. We propose an algorithm that leverages coherence in massive light fields by breaking with a number of established practices in image-based reconstruction. Our algorithm first computes reliable depth estimates specifically around object boundaries instead of interior regions, by operating on individual light rays instead of image patches. More homogeneous interior regions are then processed in a fine-to-coarse procedure rather than the standard coarse-to-fine approaches. At no point in our method is any form of global optimization performed. This allows our algorithm to retain precise object contours while still ensuring smooth reconstructions in less detailed areas. While the core reconstruction method handles general unstructured input, we also introduce a sparse representation and a propagation scheme for reliable depth estimates which make our algorithm particularly effective for 3D input, enabling fast and memory efficient processing of "Gigaray light fields" on a standard GPU. We show dense 3D reconstructions of highly detailed scenes, enabling applications such as automatic segmentation and image-based rendering, and provide an extensive evaluation and comparison to existing image-based reconstruction techniques.
Supplemental Material
Available for Download
Supplemental material.
- Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE PAMI 14, 2. Google ScholarDigital Library
- Ayvaci, A., Raptis, M., and Soatto, S. 2012. Sparse occlusion detection with optical flow. IJCV 97, 3. Google ScholarDigital Library
- Basha, T., Avidan, S., Hornung, A., and Matusik, W. 2012. Structure and motion from scene registration. In CVPR.Google Scholar
- Beeler, T., Bickel, B., Beardsley, P. A., Sumner, B., and Gross, M. H. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4. Google ScholarDigital Library
- Bishop, T. E., and Favaro, P. 2010. Full-resolution depth map estimation from an aliased plenoptic light field. In ACCV. Google ScholarDigital Library
- Bishop, T., Zanetti, S., and Favaro, P. 2009. Light field superresolution. In ICCP.Google Scholar
- Bleyer, M., Rother, C., Kohli, P., Scharstein, D., and Sinha, S. 2011. Object stereo --- joint stereo matching and object segmentation. In CVPR. Google ScholarDigital Library
- Bolles, R. C., Baker, H. H., and Marimont, D. H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. IJCV 1, 1.Google ScholarCross Ref
- Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured lumigraph rendering. In SIGGRAPH. Google ScholarDigital Library
- Čech, J., and Šára, R. 2007. Efficient sampling of disparity space for fast and accurate matching. In CVPR.Google Scholar
- Chai, J., Chan, S.-C., Shum, H.-Y., and Tong, X. 2000. Plenoptic sampling. In SIGGRAPH. Google ScholarDigital Library
- Chen, W.-C., Bouguet, J.-Y., Chu, M. H., and Grzeszczuk, R. 2002. Light field mapping: Efficient representation and hardware rendering of surface light fields. In SIGGRAPH. Google ScholarDigital Library
- Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE PAMI 24, 5. Google ScholarDigital Library
- Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. CVIU 97, 1. Google ScholarDigital Library
- Davis, A., Levoy, M., and Durand, F. 2012. Unstructured light fields. Comput. Graph. Forum 31, 2. Google ScholarDigital Library
- Duda, R., Hart, P., and Stork, D. 1995. Pattern Classification and Scene Analysis, 2nd ed. Google ScholarDigital Library
- Fitzgibbon, A., Wexler, Y., and Zisserman, A. 2005. Image-based rendering using image-based priors. IJCV 63, 2. Google ScholarDigital Library
- Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multi-view stereopsis. IEEE PAMI 32, 8. Google ScholarDigital Library
- Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. 2010. Towards Internet-scale multi-view stereo. In CVPR.Google Scholar
- Fusiello, A., Trucco, E., and Verri, A. 2000. A compact algorithm for rectification of stereo pairs. Mach. Vis. Appl. 12, 1. Geiger, A., Roser, M., and Urtasun, R. 2010. Efficient large-scale stereo matching. In ACCV. Google ScholarDigital Library
- Georgiev, T., and Lumsdaine, A. 2010. Reducing plenoptic camera artifacts. Comp. Graph. Forum 29, 6.Google ScholarCross Ref
- Goldlücke, B., and Magnor, M. 2003. Joint 3D-reconstruction and background separation in multiple views using graph cuts. In CVPR. Google ScholarDigital Library
- Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. In SIGGRAPH. Google ScholarDigital Library
- Hirschmüller, H. 2005. Accurate and efficient stereo processing by semi-global matching and mutual information. In CVPR.Google Scholar
- Humayun, A., Mac Aodha, O., and Brostow, G. 2011. Learning to find occlusion regions. In CVPR. Google ScholarDigital Library
- Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In SIGGRAPH. Google ScholarDigital Library
- Kang, S. B., and Szeliski, R. 2004. Extracting view-dependent depth maps from a collection of images. IJCV 58, 2. Google ScholarDigital Library
- Kolmogorov, V., and Zabih, R. 2001. Computing visual correspondence with occlusions via graph cuts. In ICCV.Google Scholar
- Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH. Google ScholarDigital Library
- Liang, C.-K., Lin, T.-H., Wong, B.-Y., Liu, C., and Chen, H. H. 2008. Programmable aperture photography: multiplexed light field acquisition. ACM Trans. Graph. 27, 3. Google ScholarDigital Library
- Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Comp. Sci. Techn. Rep. CSTR 2.Google Scholar
- Rav-Acha, A., Shor, Y., and Peleg, S. 2004. Mosaicing with parallax using time warping. In IVR. Google ScholarDigital Library
- Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. 2011. Fast cost-volume filtering for visual correspondence and beyond. In CVPR. Google ScholarDigital Library
- Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47, 1--3. Google ScholarDigital Library
- Schechner, Y. Y., and Kiryati, N. 2000. Depth from defocus vs. stereo: How different really are they? IJCV 39, 2. Google ScholarDigital Library
- Seitz, S. M., and Dyer, C. R. 1999. Photorealistic scene reconstruction by voxel coloring. IJCV 35, 2. Google ScholarDigital Library
- Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR. Google ScholarDigital Library
- Snavely, N., Seitz, S. M., and Szeliski, R. 2008. Modeling the world from Internet photo collections. IJCV 80, 2. Google ScholarDigital Library
- Stich, T., Tevs, A., and Magnor, M. A. 2006. Global depth from epipolar volumes--a general framework for reconstructing non-lambertian surfaces. In 3DPVT. Google ScholarDigital Library
- Sun, X., Mei, X., Jiao, S., Zhou, M., and Wang, H. 2011. Stereo matching with reliable disparity propagation. In 3DIMPVT. Google ScholarDigital Library
- Sylwan, S. 2010. The application of vision algorithms to visual effects production. In ACCV. Google ScholarDigital Library
- Szeliski, R., and Scharstein, D. 2002. Symmetric sub-pixel stereo matching. In ECCV. Google ScholarDigital Library
- Vaish, V., Levoy, M., Szeliski, R., Zitnick, C., and Kang, S. 2006. Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures. In CVPR. Google ScholarDigital Library
- Veeraraghavan, A., Raskar, R., Agrawal, A. K., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3. Google ScholarDigital Library
- Vu, H.-H., Keriven, R., Labatut, P., and Pons, J.-P. 2009. Towards high-resolution large-scale multi-view stereo. In CVPR.Google Scholar
- Wanner, S., and Goldlücke, B. 2012. Globally consistent depth labeling of 4D light fields. In CVPR.Google Scholar
- Wanner, S., Fehr, J., and Jaehne, B. 2011. Generating EPI representations of 4D light fields with a single lens focused plenoptic camera. In IISVC. Google ScholarDigital Library
- Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antúnez, E. R., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3. Google ScholarDigital Library
- Wood, D. N., Azuma, D. I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D. H., and Stuetzle, W. 2000. Surface light fields for 3D photography. In SIGGRAPH. Google ScholarDigital Library
- Yu, Y., Ferencz, A., and Malik, J. 2001. Extracting objects from range and radiance images. IEEE TVCG 7, 4. Google ScholarDigital Library
- Zhang, C., and Chen, T. 2004. A self-reconfigurable camera array. In EGSR. Google ScholarDigital Library
- Zhu, Z., Xu, G., and Lin, X. 1999. Panoramic EPI generation and analysis of video from a moving platform with vibration. In CVPR.Google Scholar
- Ziegler, R., Bucheli, S., Ahrenberg, L., Magnor, M. A., and Gross, M. H. 2007. A bidirectional light field - hologram transform. Comput. Graph. Forum 26, 3.Google ScholarCross Ref
- Zitnick, C. L., and Kang, S. B. 2007. Stereo for image-based rendering using image over-segmentation. IJCV 75, 1. Google ScholarDigital Library
- Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3. Google ScholarDigital Library
Index Terms
- Scene reconstruction from high spatio-angular resolution light fields
Recommendations
Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphicsWe present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Interactive deformation of light fields
I3D '05: Proceedings of the 2005 symposium on Interactive 3D graphics and gamesWe present a software pipeline that enables an animator to deform light fields. The pipeline can be used to deform complex objects, such as furry toys, while maintaining photo-realistic quality. Our pipeline consists of three stages. First, we split the ...
Frequency analysis and sheared filtering for shadow light fields of complex occluders
Monte Carlo ray tracing of soft shadows produced by area lighting and intricate geometries, such as the shadows through plant leaves or arrays of blockers, is a critical challenge. The final image often has relatively smooth shadow patterns, since it ...
Comments