Joel Carranza
Abstract
In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint.The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.
Supplemental Material
- ALLEN, B., CURLESS, B., AND POPOVIC, Z. 2002. Articulated body deformations from range scan data. In Proceedings of ACM SIGGRAPH 02, 612--619.]] Google Scholar
Digital Library
- BOROVIKOV, E., AND DAVIS, L. 2000. A dristibuted system for real-time volume reconstruction. In Proceedings of Intl. Workshop on Computer Architectures for Machine Perception, 183ff.]] Google ScholarDigital Library
- BOTTINO, A., AND LAURENTINI, A. 2001. A silhouette based technique for the reconstruction of human movement. CVIU 83, 79--95.]] Google ScholarDigital Library
- BREGLER, C., AND MALIK, J. 1998. Tracking people with twists and exponential maps. In Proc. of CVPR 98, 8--15.]] Google ScholarDigital Library
- BUEHLER, C., BOSSE, M., MCMILLAN, L., GORTLER, S. J., AND COHEN, M. F. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 01, ACM Press, S. Spencer, Ed., 425--432.]] Google ScholarDigital Library
- CHEUNG, K., KANADE, T., BOUGUET, J.-Y., AND HOLLER, M. 2000. A real time system for robust 3D voxel reconstruction of human motions In Proc. of CVPR, vol. 2, 714--720.]]Google Scholar
- CURLESS, B., AND SEITZ, S. 2000. 3D photography Course Notes. ACM SIGGRAPH 00.]]Google Scholar
- DANA, K., VAN GINNEKEN, B., NAYAR, S., AND KOENDERINK, J. 1999 Reflectance and texture of real-world surfaces. ACM Transactions on Graphics 18, 1, 1--34.]] Google ScholarDigital Library
- DEBEVEC, P., TAYLOR, C., MALIK, J., LEVIN, G., G. BORSHUKOV, AND YU, Y. 1998. Image-based modeling and rendering of architecture with interactive photogrammetry and view-dependent texture mapping. Proc. IEEE International Symposium on Circuits and Systems (ISCAS'98), Monterey, USA 5 (June), 514--517.]]Google ScholarCross Ref
- DELAMARRE, Q., AND FAUGERAS, O. 1999. 3D articulated models and multi-view tracking with silhouettes. In Proc. of ICCV 99, 716--721.]] Google ScholarDigital Library
- GAVRILA, D., AND DAVIS, L. 1996. 3D model-based tracking of humans in action: A multi-view approach. In Proc. of CVPR 96, 73--80.]] Google ScholarDigital Library
- GAVRILA, D. 1999. The visual analysis of human movement. CVIU 73, 1 (January), 82--98.]] Google ScholarDigital Library
- GRAMMALIDIS, N., GOUSSIS, G., TROUFAKOS, G., AND STRINTZIS, M. 2001. Estimating body animation parameters from depth images using analysis by synthesis. In Proc. of Second International Workshop on Digital and Computational Video (DCV'01), 93ff.]] Google ScholarDigital Library
- JAIN, R., KASTURI, R., AND SCHUNCK, B. 1995. Machine Vision. McGraw-Hill.]] Google ScholarDigital Library
- KILGARD, M. J., 2002. Nvidia opengl extension specifications. http://developer.nvidia.com/docs/IO/3260/ATT/nv30specs.pdf.]]Google Scholar
- KOENEN, R., 2002. Mpeg-4 overview. http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm.]]Google Scholar
- LAURENTINI, A. 1994. The visual hull concept for silhouette-based image understanding. Pattern Analysis and Machine Intelligence 16, 2 (February), 150--162.]] Google ScholarDigital Library
- LENSCH, H., HEIDRICH, W., AND SEIDEL, H. P. 2001. A silhouette-based algorithm for texture registration and stitching. Graphical Models 64(3), 245--262.]] Google ScholarDigital Library
- LEUNG, M., AND YANG, Y. 1995. First sight: A human body outline labeling system. PAMI 17(4), 359--379.]] Google ScholarDigital Library
- LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In Proceedings of ACM SIGGRAPH 96, vol. 30, 31--42.]] Google ScholarDigital Library
- LUCK, J., AND SMALL, D. 2002. Real-time markerless motion tracking using linked kinematic chains. In Proc. of CVPRIP02.]]Google Scholar
- MARTINEZ, G. 1995. 3D motion estimation of articulated objects for object-based analysis-synthesis coding (OBASC). In VLBV 95.]]Google Scholar
- MATSUYAMA, T., AND TAKAI, T. 2002. Generation, visualization, and editing of 3D video. In Proc. of 1st International Symposium on 3D Data Processing Visualization and Transmission (3DPVT'02), 234ff.]]Google ScholarCross Ref
- MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S., AND MCMILLAN, L. 2000. Image-based visual hulls. In Proceedings of ACM SIGGRAPH 00, 369--374.]] Google ScholarDigital Library
- MATUSIK, W., BUEHLER, C., AND MCMILLAN, L. 2001. Polyhedral visual hulls for real-time rendering. In Proceedings of 12th Eurographics Workshop on Rendering, 116--126.]] Google ScholarDigital Library
- MENACHE, A. 1995. Understanding Motion Capture for Computer Animation and Video Games. Morgan Kaufmann.]] Google ScholarDigital Library
- MIKIĆ, I., TRIVERDI, M., HUNTER, E., AND COSMAN, P. 2001. Articulated body posture estimation from multicamera voxel data. In Proc. of CVPR.]]Google Scholar
- MOEZZI, S., TAI, L.-C., AND GERARD, P. 1997. Virtual view generation for 3D digital video. IEEE MultiMedia 4, 1 (Jan.--Mar.), 18--26.]] Google ScholarDigital Library
- MULLIGAN, J., AND DANIILIDIS, K. 2000. View-independent scene acquisition for telepresence. In Proceedings of the International Symposium on Augmented Reality, 105--108.]]Google Scholar
- NARAYANAN, P., RANDER, P., AND KANADE, T. 1998. Constructing virtual worlds using dense stereo. In Proc. of ICCV 98, 3--10.]] Google ScholarDigital Library
- PLAENKERS, R., AND FUA, P. 2001. Tracking and modeling people in video sequences. CVIU 81, 3 (March), 285--302.]] Google ScholarDigital Library
- PRESS, W., TEUKOLSKY, S., VETTERLING, W., AND FLANNERY, B. 1992. Numerical Recipes. Cambridge University Press.]]Google Scholar
- RASKAR, R., AND LOW, K.-L. 2002. Blending multiple views. In Proceedings of Pacific Graphics 2002, 145--153.]] Google ScholarDigital Library
- ROHR, K. 1993. Incremental recognition of pedestrians from image sequences. In Proc. of CVPR 93, 8--13.]]Google Scholar
- SILAGHI, M.-C., PLAENKERS, R., BOULIC, R., FUA, P., AND THALMANN, D. 1998. Local and global skeleton fitting techniques for optical motion capture. In Modeling and Motion Capture Techniques for Virtual Environments, Springer, no. 1537 in LNAI, No1537, 26--40.]] Google ScholarDigital Library
- TERZOPOULOS, D., CARLBOM, I., FREEMAN, W., KLINKER, G., LORENSEN, W., SZELISKI, R., AND WATERS, K. 1995. Computer vision for computer graphics. In ACM SIGGRAPH 95 Course Notes, vol. 25.]]Google Scholar
- THEOBALT, C., MAGNOR, M., SCHUELER, P., AND SEIDEL, H.-P. 2002. Combining 2D feature tracking and volume reconstruction for online video-based human motion capture. In Proceedings of Pacific Graphics 2002, 96--103.]] Google ScholarDigital Library
- TSAI, R. 1986. An efficient and accurate camera calibration technique for 3D machine vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'86), 364--374.]]Google Scholar
- VEDULA, S., BAKER, S., AND KANADE, T. 2002. Spatio-temporal view interpolation. In Proceedings of the 13th ACM Eurographics Workshop on Rendering, 65--75.]] Google ScholarDigital Library
- WREN, C., AZARBAYEJANI, A., DARRELL, T., AND PENTLAND, A. 1997. Pfinder: Real-time tracking of the human body. PAMI 19, 7, 780--785.]] Google ScholarDigital Library
- WUERMLIN, S., LAMBORAY, E., STAADT, O., AND GROSS, M. 2002. 3d video recorder. In Proceedings of Pacific Graphics 2002, IEEE Computer Society Press, 325--334.]] Google ScholarDigital Library
- YONEMOTO, S., ARITA, D., AND TANIGUCHI, R. 2000. Real-time human motion analysis and IK-based human figure control. In Proceedings of IEEE Workshop on Human Motion, 149--154.]] Google ScholarDigital Library
Index Terms
- Free-viewpoint video of human actors
Recommendations
Free-viewpoint video of human actors
SIGGRAPH '03: ACM SIGGRAPH 2003 PapersIn free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its ...
Free-viewpoint video rendering for mobile devices
MIRAGE '13: Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and ApplicationsFree-viewpoint video renderers (FVVR) allow a user to view captured video footage from any position and direction. Despite the obvious appeal of such systems, they have yet to make a major impact on digital entertainment. Current FVVR implementations ...
Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We ...
Comments