Abstract
Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both.
Supplemental Material
Available for Download
Supplemental files.
- Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Transactions on Graphics (TOG) 32, 4, Article 110 (2013), 12 pages. Google ScholarDigital Library
- Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics (TOG) 34, 4 (2015), 110:1--13. Google ScholarDigital Library
- Neil Alldrin, Todd Zickler, and David Kriegman. 2008. Photometric stereo with non-parametric and spatially-varying reflectance. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2008. 1--8.Google ScholarCross Ref
- Michael Ashikhmin, Simon Premoze, and Peter Shirley. 2000. A microfacet-based BRDF generator. In Proc. ACM SIGGRAPH 2000. 65--74. Google ScholarDigital Library
- Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, and Min H. Kim. 2018. Simultaneous Acquisition of Polarimetric SVBRDF and Normals. ACM Transactions on Graphics (TOG) 37, 6 (2018). Google ScholarDigital Library
- Mahdi M Bagher, John Snyder, and Derek Nowrouzezahrai. 2016. A Non-Parametric Factor Microfacet Model for Isotropic BRDFs. ACM Transactions on Graphics (TOG) 35, 5 (2016), 159. Google ScholarDigital Library
- Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance scanning: estimating shading frame and BRDF with generalized linear light sources. ACM Transactions on Graphics (TOG) 33, 4 (2014), 117:1--11. Google ScholarDigital Library
- Paolo Cignoni, Claudio Rocchini, and Roberto Scopigno. 1998. Metro: Measuring error on simplified surfaces. In Computer Graphics Forum, Vol. 17. Wiley Online Library, 167--174.Google Scholar
- Robert L. Cook and Kenneth E. Torrance. 1982. A Reflectance Model for Computer Graphics. ACM Transactions on Graphics (TOG) 1, 1 (1982), 7--24. Google ScholarDigital Library
- Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. ACM SIGGRAPH '97. 369--378. Google ScholarDigital Library
- Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold bootstrapping for SVBRDF capture. ACM Transactions on Graphics (TOG) 29, 4 (2010), 98. Google ScholarDigital Library
- Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear Light Source Reflectometry. ACM Transactions on Graphics (TOG) 22, 3 (2003), 749--758. Google ScholarDigital Library
- Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2010. Circularly Polarized Spherical Illumination Reflectometry. ACM Transactions on Graphics (TOG) (Dec. 2010). Google ScholarDigital Library
- Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011), 129. Google ScholarDigital Library
- Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical modeling and acquisition of layered facial reflectance. ACM Transactions on Graphics (TOG) 27, 5 (2008), 139. Google ScholarDigital Library
- Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec, and Abhijeet Ghosh. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum 32 (2013), 335--344.Google ScholarCross Ref
- Darya Guarnera, Giuseppe Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35 (2016), 625--650.Google ScholarCross Ref
- Carlos Hernandez, George Vogiatzis, and Roberto Cipolla. 2008. Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 3 (2008), 548--554. Google ScholarDigital Library
- Tomoaki Higo, Yasuyuki Matsushita, Neel Joshi, and Katsushi Ikeuchi. 2009. A handheld photometric stereo camera for 3-d modeling. In Computer Vision, 2009 IEEE 12th International Conference on. IEEE, 1234--1241.Google ScholarCross Ref
- Michael Holroyd, Jason Lawrence, Greg Humphreys, and Todd Zickler. 2008. A photometric approach for estimating normals and tangents. ACM Transactions on Graphics (TOG) 27, 5 (2008), 133. Google ScholarDigital Library
- Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics (TOG) 29, 4 (2010), 99. Google ScholarDigital Library
- Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5362--5370.Google ScholarCross Ref
- Olaf Kahler, Victor Adrian Prisacariu, Carl Yuheng Ren, Xin Sun, Philip Torr, and David Murray. 2015. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Transactions on Visualization and Computer Graphics 21, 11 (2015), 1241--1250. Google ScholarDigital Library
- Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013), 29. Google ScholarDigital Library
- Kalin Kolev, Petri Tanskanen, Pablo Speciale, and Marc Pollefeys. 2014. Turning Mobile Phones into 3D Scanners. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
- Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics (TOG) 25, 3 (2006), 735--745. Google ScholarDigital Library
- Joo Ho Lee, Adrian Jarabo, Daniel S. Jeon, Diego Gutierrez, and Min H. Kim. 2018. Practical Multiple Scattering for Rough Surfaces. ACM Transactions on Graphics (TOG) 37, 6 (2018).Google ScholarDigital Library
- Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2001. Image-based reconstruction of spatially varying materials. In Rendering Techniques 2001. Springer, 103--114. Google ScholarDigital Library
- Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Transactions on Graphics (TOG) 22, 2 (2003), 234--257. Google ScholarDigital Library
- Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J Gortler, and Leonard McMillan. 2000. Image-based Visual Hulls. In Proc. ACM SIGGRAPH '00. 369--374. Google ScholarDigital Library
- Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on Graphics 22, 3 (July 2003), 759--769. Google ScholarDigital Library
- NAG. 2015. The NAG Library, Numerical Algorithms Group. http://www.nag.com/.Google Scholar
- Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbic, Hao Li, Abhijeet Ghosh, and Paul E Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (TOG) 34, 4 (2015), 109--1. Google ScholarDigital Library
- Giljoo Nam and Min H. Kim. 2014. Multispectral Photometric Stereo for Acquiring High-Fidelity Surface Normals. IEEE Computer Graphics and Applications 34, 6 (2014), 57--68.Google ScholarCross Ref
- Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H. Kim. 2016. Simultaneous Acquisition of Microscale Reflectance and Normals. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2016) 35, 6 (2016). Google ScholarDigital Library
- Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. 2005. Efficiently combining positions and normals for precise 3D geometry. ACM Trans. Graph 24, 3 (2005), 536--543. Google ScholarDigital Library
- Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proc. Int. Sym. Mixed and augmented reality (ISMAR). 127--136. Google ScholarDigital Library
- Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics 34, 6 (2015), 186. Google ScholarDigital Library
- Peter Ondruska, Pushmeet Kohli, and Shahram Izadi. 2015. Mobilefusion: Real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE transactions on visualization and computer graphics 21, 11 (2015), 1251--1258. Google ScholarDigital Library
- Geoffrey Oxholm and Ko Nishino. 2014. Multiview Shape and Reflectance from Natural Illumination. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2163--2170. Google ScholarDigital Library
- Jaesik Park, Sudipta Sinha, Yasuyuki Matsushita, Yu Wing Tai, and In So Kweon. 2016. Robust Multiview Photometric Stereo using Planar Mesh Parameterization. IEEE Transactions on Pattern Analysis and Machine Intelligence (2016).Google Scholar
- Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket reflectometry. ACM Transactions on Graphics (TOG) 30, 4 (2011), 45:1--10. Google ScholarDigital Library
- Jérémy Riviere, Pieter Peers, and Abhijeet Ghosh. 2015. Mobile Surface Reflectometry. Computer Graphics Forum (2015). Google ScholarDigital Library
- Jeremy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 206. Google ScholarDigital Library
- Szymon Rusinkiewicz. 1998. A New Change of Variables for Efficient BRDF Representation. In Rendering Techniques '98 (Proceedings of Eurographics Rendering Workshop '98), G. Drettakis and N. Max (Eds.). Springer Wien, 11--22.Google ScholarCross Ref
- Johannes L. Schönberger. 2016. COLMAP. https://colmap.github.io.Google Scholar
- Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision. Springer, 501--518.Google ScholarCross Ref
- Thomas Schöps, Torsten Sattler, Christian Häne, and Marc Pollefeys. 2015. 3d modeling on the go: Interactive 3d reconstruction of large-scale scenes on mobile devices. In 3D Vision (3DV), 2015 International Conference on. IEEE, 291--299. Google ScholarDigital Library
- Christopher Schwartz, Ralf Sarlette, Michael Weinmann, and Reinhard Klein. 2013. Dome ii: A parallelized btf acquisition system. In Proceedings of the Eurographics 2013 Workshop on Material Appearance Modeling: Issues and Acquisition. Eurographics Association, 25--31. Google ScholarDigital Library
- Niranjan Thanikachalam, Loïc Baboulaz, Damien Firmenich, Sabine Süsstrunk, and Martin Vetterli. 2017. Handheld Reflectance Acquisition of Paintings. IEEE Transactions on Computational Imaging 3, 4 (2017), 580--591.Google ScholarCross Ref
- Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Transactions on Graphics (TOG) 32, 4 (2013), 109:1--12. Google ScholarDigital Library
- Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proc. Eurographics. 195--206. Google ScholarDigital Library
- Jiaping Wang, Shuang Zhao, Xin Tong, John Snyder, and Baining Guo. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Transactions on Graphics (TOG) 27, 3 (2008), 41:1--10. Google ScholarDigital Library
- Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition (Course Notes). In SIGGRAPH Asia 2015 Courses. ACM, Article 1, 71 pages. Google ScholarDigital Library
- Tim Weyrich, Jason Lawrence, Hendrik P.A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Foundations and Trends in Computer Graphics and Vision 4, 2 (2009), 75--191. Google ScholarDigital Library
- Jae Hyun Won, Man Hee Lee, and In Kyu Park. 2012. Active 3D shape acquisition using smartphones. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE, 29--34.Google ScholarCross Ref
- Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE transactions on visualization and computer graphics 22, 8 (2016), 2012--2023. Google ScholarDigital Library
- Hongzhi Wu and Kun Zhou. 2015. Appfusion: Interactive appearance acquisition using a kinect sensor. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 289--298. Google ScholarDigital Library
- Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-varying Surface Reflectance Under Unknown Illumination. ACM Transactions on Graphics (TOG) 35, 6, Article 187 (2016), 12 pages. Google ScholarDigital Library
- Z. Y. Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 11 (Nov. 2000), 1330--1334. Google ScholarDigital Library
- Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-possible SVBRDF acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 189. Google ScholarDigital Library
- Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view Photometric Stereo with Spatially Varying Isotropic Materials. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2013. 1482--1489. Google ScholarDigital Library
- Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2015. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics (TOG) 34, 4 (2015), 96. Google ScholarDigital Library
Index Terms
- Practical SVBRDF acquisition of 3D objects with unstructured flash photography
Recommendations
Simultaneous acquisition of polarimetric SVBRDF and normals
Capturing appearance often requires dense sampling in light-view space, which is often achieved in specialized, expensive hardware setups. With the aim of realizing a compact acquisition setup without multiple angular samples of light and view, we ...
Glare aware photography: 4D ray sampling for reducing glare effects of camera lenses
Glare arises due to multiple scattering of light inside the camera's body and lens optics and reduces image contrast. While previous approaches have analyzed glare in 2D image space, we show that glare is inherently a 4D ray-space phenomenon. By ...
Hand-held 3D light field photography and applications
We propose a method to acquire 3D light fields using a hand-held camera, and describe several computational photography applications facilitated by our approach. As our input we take an image sequence from a camera translating along an approximately ...
Comments