Chenglei Wu
Michael Zollhöfer
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-varying incident lighting, which is then used for geometry refinement. This leads to dramatically enhanced depth maps at 30Hz. Our algorithm makes few scene assumptions, handling arbitrary scene objects even under motion. To enable this type of real-time depth map enhancement, we contribute a new highly parallel algorithm that reformulates the inverse rendering optimization problem in prior work, allowing us to estimate lighting and shape in a temporally coherent way at video frame-rates. Our optimization problem is minimized using a new regular grid Gauss-Newton solver implemented fully on the GPU. We demonstrate results showing enhanced depth maps, which are comparable to offline methods but are computed orders of magnitude faster, as well as baseline comparisons with online filtering-based methods. We conclude with applications of our higher quality depth maps for improved real-time surface reconstruction and performance capture.
Supplemental Material
Available for Download
zip
a200.zip (201.3 MB)
Supplemental material.
- Ahmed, A. H., and Farag, A. A. 2007. Shape from shading under various imaging conditions. In Proc. CVPR, 1--8.Google Scholar
- Aodha, O. M., Campbell, N. D. F., Nair, A., and Brostow, G. J. 2012. Patch based synthesis for single depth image superresolution. In Proc. ECCV, 71--84. Google ScholarDigital Library
- Barron, J. T., and Malik, J. 2013. Intrinsic scene properties from a single rgb-d image. In Proc. CVPR, IEEE, 17--24. Google ScholarDigital Library
- Barron, J. T., and Malik, J. 2013. Shape, illumination, and reflectance from shading. Tech. rep., EECS, UC Berkeley, May.Google Scholar
- Beder, C., Bartczak, B., and Koch, R. 2007. A combined approach for estimating patchlets from PMD depth images and stereo intensity images. In Proc. DAGM, 11--20. Google ScholarDigital Library
- Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. Proc. SIGGRAPH 29, 3. Google ScholarDigital Library
- Beeler, T., Bradley, D., Zimmer, H., and Gross, M. 2012. Improved reconstruction of deforming surfaces by cancelling ambient occlusion. In Proc. ECCV, 30--43. Google ScholarDigital Library
- Bermano, A., Bradley, D., Zund, T. B. F., Nowrouzezahrai, D., Baran, I., Sorkine-hornung, O., Pfister, H., Sumner, R., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Transactions on Graphics 33. to appear. Google ScholarDigital Library
- Besl, P. J., and McKay, N. D. 1992. Method for registration of 3-d shapes. In Robotics-DL tentative, International Society for Optics and Photonics, 586--606.Google Scholar
- Böhme, M., Haker, M., Martinetz, T., and Barth, E. 2008. Shading constraint improves accuracy of time-of-flight measurements. In Proc. CVPR Workshop.Google Scholar
- Chan, D., Buisman, H., Theobalt, C., and Thrun, S. 2008. A noise-aware filter for real-time depth upsampling. In ECCV Workshop on multi-camera & multi-modal sensor fusion.Google Scholar
- Cui, Y., Schuon, S., Thrun, S., Stricker, D., and Theobalt, C. 2013. Algorithms for 3d shape scanning with a depth camera. IEEE Trans. PAMI 35, 5, 1039--1050. Google ScholarDigital Library
- Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. SIGGRAPH, 189--198. Google ScholarDigital Library
- Debevec, P. 2012. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia Technical Briefs.Google Scholar
- Diebel, J., and Thrun, S. 2006. An application of Markov Random Fields to range sensing. In Proc. NIPS, 291--298.Google Scholar
- Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In Proc. CVPR.Google Scholar
- Fanello, S., Keskin, C., Izadi, S., Kohli, P., et al. 2014. Learning to be a depth camera for close-range human capture and interaction. ACM Trans. Graph. 33, 4. Google ScholarDigital Library
- Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graph. 30. Google ScholarDigital Library
- Gudmundsson, S. A., Aanaes, H., and Larsen, R. 2008. Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. Int. J. Intell. Syst. Technol. Appl. 5, 425--433. Google ScholarDigital Library
- Han, Y., Lee, J.-Y., and Kweon, I. S. 2013. High quality shape from a single rgb-d image under uncalibrated natural illumination. In Proc. ICCV.} Google ScholarDigital Library
- Hernández, C., Vogiatzis, G., and Cipolla, R. 2008. Multiview photometric stereo. IEEE PAMI 30, 3, 548--554. Google ScholarDigital Library
- Horn, B. K. 1974. Determining lightness from an image. Computer graphics and image processing 3, 4, 277--299.Google Scholar
- Horn, B. K. 1975. Obtaining shape from shading information. The psychology of computer vision, 115--155.Google Scholar
- Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., and Fitzgibbon, A. 2011. Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559--568. Google ScholarDigital Library
- Khan, N., Tran, L., and Tappen, M. 2009. Training many-parameter shape-from-shading models using a surface database. In Proc. ICCV Workshop.Google Scholar
- Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3. Google ScholarDigital Library
- Lindner, M., Kolb, A., and Hartmann, K. 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. In Proc. ISSCS, 121--124.Google Scholar
- Mulligan, J., and Brolly, X. 2004. Surface determination by photometric ranging. In Proc. CVPR Workshop. Google ScholarDigital Library
- Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3D geometry. Proc. SIGGRAPH 24, 3. Google ScholarDigital Library
- Newcombe, R. A., Izadi, S., et al. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), IEEE international symposium on, 127--136. Google ScholarDigital Library
- Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG) 32, 6, 169. Google ScholarDigital Library
- Park, J., Kim, H., Tai, Y.-W., Brown, M. S., and Kweon, I.-S. 2011. High quality depth map upsampling for 3d-tof cameras. In ICCV, IEEE, 1623--1630. Google ScholarDigital Library
- Prados, E., and Faugeras, O. 2005. Shape from shading: a well-posed problem? In Proc. CVPR. Google ScholarDigital Library
- Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, 117--128. Google ScholarDigital Library
- Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. Computer Graphics Forum (Proceedings of Eurographics) 31, 2 (May). Google ScholarDigital Library
- Tunwattanapong, B., Fyffe, G., Graham, P., Busch, J., Yu, X., Ghosh, A., and Debevec, P. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Transactions on Graphics (TOG) 32, 4, 109. Google ScholarDigital Library
- Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6. Google ScholarDigital Library
- Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. Computer Graphics Forum 32, 1, 16--26.Google ScholarCross Ref
- Wei, G.-Q., and Hirzinger, G. 1996. Learning shape from shading by a multilayer network. IEEE Trans. Neural Networks. Google ScholarDigital Library
- Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV. Google ScholarDigital Library
- Wu, C., Stoll, C., Valgaerts, L., and Theobalt, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM Transactions on Graphics (TOG) 32, 6, 161. Google ScholarDigital Library
- Yang, Q., Yang, R., Davis, J., and Nistr, D. 2007. Spatial-depth super resolution for range images. In Proc. CVPR, IEEE.Google Scholar
- Yu, L.-F., Yeung, S.-K., Tai, Y.-W., and Lin, S. 2013. Shading-based shape refinement of rgb-d images. In Proc. CVPR. Google ScholarDigital Library
- Zhang, Z., Tsa, P.-S., Cryer, J. E., and Shah, M. 1999. Shape from shading: A survey. IEEE PAMI 21, 8, 690--706. Google ScholarDigital Library
- Zhu, J., Wang, L., Yang, R., and Davis, J. 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In Proc. CVPR.Google Scholar
- Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., and Stamminger, M. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG (Proc. SIGGRAPH) 33, 4. Google ScholarDigital Library
- Zollhöfer, M., Thies, J., Colaianni, M., Stamminger, M., and Greiner, G. 2014. Interactive model-based reconstruction of the human head using an rgb-d sensor. Computer Animation and Virtual Worlds 25, 3-4, 213--222.Google ScholarDigital Library
Index Terms
- Real-time shading-based refinement for consumer depth cameras
Recommendations
Shading-based refinement on volumetric signed distance functions
We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. As the depth data of these sensors is noisy, truncated signed distance fields are typically used to ...
Precomputed illuminance composition for real-time global illumination
I3D '16: Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and GamesIn this paper we present a new real-time approach for indirect global illumination under dynamic lighting conditions. We use surfels to gather a sampling of the local illumination and propagate the light through the scene using a hierarchy and a set of ...
Real-Time Illumination of Foliage Using Depth Maps
ICCS '08: Proceedings of the 8th international conference on Computational Science, Part IIThis article presents a new method for foliage illumination which takes into account direct, indirect illumination and self-shadowing. Both indirect illumination and self-shadowing are approximated by means of a novel technique using depth maps. In ...
Comments