Sudipta N. Sinha
Michael Goesele
Richard Szeliski
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present a system for image-based modeling and rendering of real-world scenes containing reflective and glossy surfaces. Previous approaches to image-based rendering assume that the scene can be approximated by 3D proxies that enable view interpolation using traditional back-to-front or z-buffer compositing. In this work, we show how these can be generalized to multiple layers that are combined in an additive fashion to model the reflection and transmission of light that occurs at specular surfaces such as glass and glossy materials. To simplify the analysis and rendering stages, we model the world using piecewise-planar layers combined using both additive and opaque mixing of light. We also introduce novel techniques for estimating multiple depths in the scene and separating the reflection and transmission components into different layers. We then use our system to model and render a variety of real-world scenes with reflections.
Supplemental Material
Available for Download
zip
a100-sinha.zip (376.7 MB)
Supplemental material.
- Beery, E., and Yeredor, A. 2008. Blind separation of superimposed shifted images using parameterized joint diagonalization. IEEE Transactions on Image Processing 17, 3 (March), 340--353. Google Scholar
Digital Library
- Bergen, J. R., Burt, P. J., Hingorani, R., and Peleg, S. 1992. A three-frame algorithm for estimating two-component image motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9 (September), 886--896. Google ScholarDigital Library
- Bhat, D. N., and Nayar, S. K. 1998. Stereo and specular reflection. International Journal of Computer Vision 26, 2 (February), 91--106. Google ScholarDigital Library
- Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering, 327--338. Google ScholarDigital Library
- Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (November), 1222--1239. Google ScholarDigital Library
- Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured Lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH '01, 425--432. Google ScholarDigital Library
- Carceroni, R. L., and Kutulakos, K. N. 2002. Multi-view scene capture by surfel sampling: From video streams to nonrigid 3d motion, shape and reflectance. International Journal of Computer Vision 49, 2/3, 175--214. Google ScholarDigital Library
- Chen, S., and Williams, L. 1993. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, SIGGRAPH '93, 279--288. Google ScholarDigital Library
- Cohen-Steiner, D., Alliez, P., and Desbrun, M. 2004. Variational shape approximation. ACM Trans. Graph. 23 (August), 905--914. Google ScholarDigital Library
- Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. Computer Vision and Image Understanding 97, 1 (January), 51--85. Google ScholarDigital Library
- Darrell, T., and Pentland, A. 1995. Cooperative robust estimation using layers of support. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 5 (May), 474--487. Google ScholarDigital Library
- Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In ACM SIGGRAPH 1996 Conference Proceedings, 11--20. Google ScholarDigital Library
- Diamant, Y., and Schechner, Y. Y. 2008. Overcoming visual reverberations. In Computer Vision and Pattern Recognition (CVPR'08), 1--8.Google Scholar
- Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. 2009. Manhattan-world stereo. In Computer Vision and Pattern Recognition (CVPR 2009), 1422--1429.Google Scholar
- Gallup, D., Frahm, J.-M., and Pollefeys, M. 2010. Piece-wise planar and non-planar stereo for urban scene reconstruction. In Computer Vision and Pattern Recognition (CVPR'10), 1418--1425.Google Scholar
- Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH '96, 43--54. Google ScholarDigital Library
- Hirschmüller, H. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (February), 328--341. Google ScholarDigital Library
- Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions. International Journal of Computer Vision 12, 1 (January), 5--16. Google ScholarDigital Library
- Ju, S. X., Black, M. J., and Jepson, A. D. 1996. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In Computer Vision and Pattern Recognition (CVPR'96), 307--314. Google ScholarDigital Library
- Levin, A., Zomet, A., and Weiss, Y. 2002. Learning to perceive transparency from the statistics of natural scenes. In (NIPS), MIT Press, 1247--1254.Google Scholar
- Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH '96, 31--42. Google ScholarDigital Library
- Nocedal, J., and Wright, S. J. 2006. Numerical Optimization, second ed. Springer, New York.Google Scholar
- Popescu, V., Mei, C., Dauble, J., and Sacks, E. 2006. Reflected-scene impostors for realistic reflections at interactive rates. Computer Graphics Forum 25, 3 (Sept.), 313--322.Google ScholarCross Ref
- Schechner, Y. Y., Shamir, J., and Kiryati, N. 1999. Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface. In International Conference on Computer Vision (ICCV'99), 814--819. Google ScholarDigital Library
- Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In ACM SIGGRAPH 1998 Conference Proceedings, 231--242. Google ScholarDigital Library
- Shizawa, M., and Mase, K. 1991. A unified computational theory of motion transparency and motion boundaries based on eigenenergy analysis. In Computer Vision and Pattern Recognition (CVPR), 289--295.Google Scholar
- Sinha, S. N., Steedly, D., and Szeliski, R. 2009. Piece-wise planar stereo for image-based rendering. In International Conference on Computer Vision (ICCV 2009), 1881--1888.Google Scholar
- Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics (Proc. SIGGRAPH 2006) 25, 3 (August), 835--846. Google ScholarDigital Library
- Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Computer Vision and Pattern Recognition (CVPR'2000), vol. 1, 246--253.Google Scholar
- Szeliski, R. 2010. Computer Vision: Algorithms and Applications. Springer, New York. Google Scholar
- Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 2 (February), 290--301. Google ScholarDigital Library
- Weingarten, J. W., Gruener, G., and Dorf, A. 2004. Probabilistic plane fitting in 3D and an application to robotic mapping. In International Conference on Robotics and Automation (ICRA), 927--932.Google Scholar
- Zebedin, L., Bauer, J., Karner, K. F., and Bischof, H. 2008. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In European Conference on Computer Vision (ECCV'08), 873--886. Google ScholarDigital Library
- Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics (Proc. SIGGRAPH 2004) 23, 3 (August), 600--608. Google ScholarDigital Library
Recommendations
Sample-Based Cameras for Feed Forward Reflection Rendering
This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected ...
An introduction to image-based rendering
Integrated image and graphics technologiesIn this chapter, we review the techniques for image-based rendering. Unlike traditional 3D computer graphics in which 3D geometry of the scene is known, image-based rendering (IBR) techniques render novel views directly from input images. IBR techniques ...
Real-time multiply recursive reflections and refractions using hybrid rendering
We present a new method for real-time rendering of multiple recursions of reflections and refractions. The method uses the strengths of real-time ray tracing for objects close to the camera, by storing them in a per-frame constructed bounding volume ...
Comments