Bernd Bickel
Mario Botsch
Roland Angst
Wojciech Matusik
Hanspeter Pfister
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then augment a traditional marker-based facial motion-capture system by two synchronized video cameras to track expression wrinkles. The resulting model consists of high-resolution geometry, motion-capture data, and expression wrinkles in 2D parametric form. This combination represents the facial shape and its salient features at multiple scales. During motion synthesis the motion-capture data deforms the high-resolution geometry using a linear shell-based mesh-deformation method. The wrinkle geometry is added to the facial base mesh using nonlinear energy optimization. We present the results of our approach for performance replay as well as for wrinkle editing.
Supplemental Material
- Bando, Y., Kuratate, T., and Nishita, T. 2002. A simple method for modeling wrinkles on human skin. In Proc. of Pacific Conference on Computer Graphics and Applications. Google Scholar
Digital Library
- Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3, 641--650.Google ScholarCross Ref
- Borshukov, G., and Lewis, J. 2003. Realistic human face rendering for "The Matrix Reloaded". In ACM SIGGRAPH 03 Sketches & Applications. Google ScholarDigital Library
- Borshukov, G., Piponi, D., Larsen, O., Lewis, J., and Tempelaar-Lietz, C. 2003. Universal capture - Image-based facial animation for "The Matrix Reloaded". In ACM SIGGRAPH 03 Sketches & Applications. Google ScholarDigital Library
- Botsch, M., and Kobbelt, L. 2004. An intuitive framework for real-time freeform modeling. ACM Transactions on Graphics 23, 3, 630--634. Google ScholarDigital Library
- Botsch, M., and Kobbelt, L. 2005. Real-time shape editing using radial basis functions. Computer Graphics Forum 24, 3, 611--621.Google ScholarCross Ref
- Botsch, M., and Sorkine, O. 2007. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics (TVCG), to appear. Google ScholarDigital Library
- Botsch, M., Bommes, D., and Kobbelt, L. 2005. Efficient linear system solvers for geometry processing. In 11th IMA conference on the Mathematics of Surfaces, 62--83. Google ScholarDigital Library
- Bridson, R., Marino, S., and Fedkiw, R. 2003. Simulation of clothing with folds and wrinkles. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA), 28--36. Google ScholarDigital Library
- Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., and Evans, T. R. 2001. Reconstruction and representation of 3D objects with radial basis functions. In Proc. of ACM SIGGRAPH 01, 67--76. Google ScholarDigital Library
- Celniker, G., and Gossard, D. 1991. Deformable curve and surface finite-elements for free-form shape design. In Proc. of ACM SIGGRAPH 91, 257--266. Google ScholarDigital Library
- Cortes, C., and Vapnik, V. 1995. Support-vector networks. Machine Learning 20, 3, 273--297. Google ScholarDigital Library
- DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 231--238. Google ScholarDigital Library
- Desbrun, M., Meyer, M., Schröder, P., and Barr, A. H. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proc. of ACM SIGGRAPH 99, 317--324. Google ScholarDigital Library
- Duchon, J. 1977. Spline minimizing rotation-invariant semi-norms in Sobolev spaces. In Constructive Theory of Functions of Several Variables, W. Schempp and K. Zeller, Eds., no. 571 in Lecture Notes in Mathematics. Springer, 85--100.Google Scholar
- Essa, I. A., and Pentland, A. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 19, 7, 757--763. Google ScholarDigital Library
- Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: Using input from video. In Proc. of Computer Animation 96, 68--79. Google ScholarDigital Library
- Grinspun, E., Hirani, A. N., Desbrun, M., and Schröder, P. 2003. Discrete shells. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA), 62--67. Google ScholarDigital Library
- Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proc. of ACM SIGGRAPH 98, 55--66. Google ScholarDigital Library
- Horn, B. K. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America 4, 4, 629--642.Google ScholarCross Ref
- Hyneman, W., Itokazu, H., Williams, L., and Zhao, X. 2005. Human face project. In ACM SIGGRAPH 05 Course Notes. Google ScholarDigital Library
- Igarashi, T., Nishino, K., and Nayar, S. 2005. The appearance of human skin. Tech. Rep. CUCS-024-05, Department of Computer Science, Columbia University.Google Scholar
- Jones, A., Gardner, A., Bolas, M., McDowall, I., and Debevec, P. 2006. Performance geometry capture for spatially varying relighting. In 3rd European Conference on Visual Media Production (CVMP 2006).Google Scholar
- Kobbelt, L., Vorsatz, J., and Seidel, H.-P. 1999. Multiresolution hierarchies on unstructured triangle meshes. Comput. Geom. Theory Appl. 14, 1--3, 5--24. Google ScholarDigital Library
- Koch, R. M., Gross, M. H., Carls, F. R., Von Büren, D. F., Fankhauser, G., and Parish, Y. 1996. Simulating facial surgery using finite element methods. In Proc. of ACM SIGGRAPH 96, 421--428. Google ScholarDigital Library
- Lanir, Y. 1987. Skin mechanics. In Handbook of Bioengineering, R. Skalak and S. Chien, Eds. McGraw-Hill, 11.1--11.25.Google Scholar
- Li, H., Roivainen, P., and Forchheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 15, 6, 545--555. Google ScholarDigital Library
- Magnenat-Thalmann, N., Kalra, P., Lévêque, J. L., Bazin, R., Batisse, D., and Queleux, B. 2002. A computational skin model: fold and wrinkle formation. IEEE Trans. on Information Technology in Biomedicine 6, 4, 317--323. Google ScholarDigital Library
- Meyer, M., Desbrun, M., Schröder, P., and Barr, A. H. 2003. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and Mathematics III, H.-C. Hege and K. Polthier, Eds. Springer-Verlag, Heidelberg, 35--57.Google Scholar
- Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Transactions on Graphics 24, 3, 536--543. Google ScholarDigital Library
- Noh, J.-Y, and Neumann, U. 1999. A survey of facial modeling and animation techniques. Tech. Rep. USC-TR-99-705, Univeristy of Southern Californina.Google Scholar
- Noh, J.-Y., and Neumann, U. 2001. Expression cloning. In Proc. of SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 277--288. Google ScholarDigital Library
- Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision (ICCV), 143--150.Google Scholar
- Shi, L., Yu, Y., Bell, N., and Feng, W.-W. 2006. A fast multigrid algorithm for mesh deformation. ACM Transactions on Graphics 25, 3, 1108--1117. Google ScholarDigital Library
- Sifakis, E., Neverov, I., and Fedkiw, R. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3, 417--425. Google ScholarDigital Library
- Svoboda, T., Martinec, D., and Pajdla, T. 2005. A convenient multicamera self-calibration for virtual environments. Presence: Teleoper. Virtual Environ. 14, 4, 407--422. Google ScholarDigital Library
- Terzopoulus, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 14, 569--579. Google ScholarDigital Library
- Venkataraman, K., Lodha, S., and Raghavan, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computers & Graphics 29, 5, 756--770. Google ScholarDigital Library
- Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3, 426--433. Google ScholarDigital Library
- Wang, Y., Huang, X., Lee, C.-S., Zhang, S., LI, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. Computer Graphics Forum 23, 3, 677--686.Google ScholarCross Ref
- Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3, 756--764. Google ScholarDigital Library
- Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., TU, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics 25, 3, 1013--1024. Google ScholarDigital Library
- Williams, L. 1990. Performance-driven facial animation. In Proc. of ACM SIGGRAPH 90, vol. 24, 235--242. Google ScholarDigital Library
- Wu, Y., Kalra, P., and Magnenat-Thalmann, N. 1996. Simulation of static and dynamic wrinkles of skin. In Proc. of Computer Animation, 90--97. Google ScholarDigital Library
- Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.Google Scholar
- Zhang, Y., and Sim, T. 2005. Realistic and efficient wrinkle simulation using an anatomy-based face model with adaptive refinement. In Computer Graphics International 2005, 3--10. Google ScholarDigital Library
- Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3, 548--558. Google ScholarDigital Library
Index Terms
- Multi-scale capture of facial geometry and motion
Recommendations
Multi-scale capture of facial geometry and motion
SIGGRAPH '07: ACM SIGGRAPH 2007 papersWe present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then ...
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
SIGGRAPH '11: ACM SIGGRAPH 2011 papersThis paper introduces a new approach for acquiring high-fidelity 3D facial performances with realistic dynamic wrinkles and fine-scale facial details. Our approach leverages state-of-the-art motion capture technology and advanced 3D scanning technology ...
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
This paper introduces a new approach for acquiring high-fidelity 3D facial performances with realistic dynamic wrinkles and fine-scale facial details. Our approach leverages state-of-the-art motion capture technology and advanced 3D scanning technology ...
Comments