Fei Yang
Abstract
We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending using existing compositing tools results in semantically unnatural composites, since expression is a global effect and the local component in one expression is often incompatible with the shape and other components of the face in another expression. To solve this problem we present Expression Flow, a 2D flow field which can warp the target face globally in a natural way, so that the warped face is compatible with the new facial component to be copied over. To do this, starting with the two input face photos, we jointly construct a pair of 3D face shapes with the same identity but different expressions. The expression flow is computed by projecting the difference between the two 3D shapes back to 2D. It describes how to warp the target face photo to match the expression of the reference photo. User studies suggest that our system is able to generate face composites with much higher fidelity than existing methods.
Supplemental Material
Available for Download
- Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. In Proceedings of ACM SIGGRAPH, vol. 23, 294--302. Google Scholar
Digital Library
- Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. In Proceedings of ACM SIGGRAPH, 39:1--39:8. Google ScholarDigital Library
- Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM SIGGRAPH, 187--194. Google Scholar
- Blanz, V., Basso, C., Poggio, T., and Vetter. 2003. Rean-imating faces in images and video. Computer Graphics Forum 22, 3.Google ScholarCross Ref
- Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging faces in images. Computer Graphics Forum 23, 3, 669--676.Google ScholarCross Ref
- Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence 23. Google Scholar
- Cootes, T., Taylor, C., Cooper, D., and Graham, J. 1995. Active shape models: Their training and application. Computer Vision and Image Understanding 61, 1, 38--59. Google ScholarDigital Library
- Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Trans. Pattern Analysis and Machine Intelligence 23, 681--685. Google ScholarDigital Library
- Decarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 99--127. Google ScholarCross Ref
- Dovgard, R., and Basri, R. 2004. Statistical symmetric shape from shading for 3d structure recovery of faces. In Proceedings of ECCV, 99--113.Google Scholar
- Faigin, G. 1991. The Artist's Complete Guide to Facial Expression. Watson-Guptill Publications Inc., New York.Google Scholar
- Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. In Proceedings of ACM SIGGRAPH, vol. 28, 67:1--67:9. Google ScholarDigital Library
- Fattal, R. 2009. Edge-avoiding wavelets and their applications. In Proceedings of ACM SIGGRAPH, vol. 28. Google ScholarDigital Library
- Felzenszwalb, P. F., and Huttenlocher, D. P. 2005. Pictorial structures for object recognition. International Journal of Computer Vision 61, 55--79. Google ScholarDigital Library
- Joshi, N., Matusik, W., Adelson, E. H., and Kriegman, D. J. 2010. Personal photo enhancement using example images. ACM Trans. Graphics 29, 12:1--12:15. Google ScholarDigital Library
- Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. In Proceedings of ACM SIGGRAPH, 38:1--38:9. Google ScholarDigital Library
- Liang, L., Xiao, R., Wen, F., and Sun, J. 2008. Face alignment via component-based discriminative search. In Proceedings of ECCV. Google Scholar
- Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of ACM SIGGRAPH, 271--276. Google Scholar
- Liu, C., Shum, H.-Y., and Freeman, W. T. 2007. Face hallucination: Theory and practice. International Journal of Computer Vision 75, 115--134. Google ScholarDigital Library
- Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 1981 DARPA Image Understanding Workshop, 121--130. Google ScholarDigital Library
- Milborrow, S., and Nicolls, F. 2008. Locating facial features with an extended active shape model. In Proceedings of ECCV, 504--513. Google Scholar
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In Proceedings of ACM SIGGRAPH, 313--318. Google Scholar
- Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of ACM SIGGRAPH, 75--84. Google Scholar
- Romdhani, S., and Vetter, T. 2003. Efficient, robust and accurate fitting of a 3d morphable model. In Proceedings of ICCV, 59--66. Google ScholarDigital Library
- Saragih, J., Lucey, S., and Cohn, J. 2009. Face alignment through subspace constrained mean-shifts. In Proceedings of the 12th ICCV, 1034--1041.Google Scholar
- Shlizerman, I. K., and Basri, R. 2011. 3d face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Analysis and Machine Intelligence 33, 394--405. Google ScholarDigital Library
- Shlizerman, I. K., Sankar, A., Shechtman, E., and Seitz, S. M. 2010. Being john malkovich. In Proceedings of ECCV, 341--353. Google ScholarDigital Library
- Singular Inversions Inc. 2009. Facegen modeller manual. In www.facegen.com.Google Scholar
- Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. In Proceedings of ACM SIGGRAPH, vol. 29. Google ScholarDigital Library
- Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In Proceedings of ACM SIGGRAPH, vol. 24, 426--433. Google ScholarDigital Library
- Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. In Proceedings of EuroGraphics, 677--686.Google Scholar
- Williams, L. 1990. Performance-driven facial animation. In Proceedings of ACM SIGGRAPH, vol. 24, 235--242. Google ScholarCross Ref
- Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In Proceedings of ACM SIGGRAPH, 548--558. Google Scholar
Index Terms
- Expression flow for 3D-aware face component transfer
Recommendations
Expression flow for 3D-aware face component transfer
SIGGRAPH '11: ACM SIGGRAPH 2011 papersWe address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending ...
Generating 3D views of facial expressions from frontal face video based on topographic analysis
MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on MultimediaIn this paper, we report our newly developed 3D face modeling system with arbitrary expressions in a high level of detail using the topographic analysis and mesh instantiation process. Given a sequence of images of facial expressions at frontal views, ...
A bi-modal face recognition framework integrating facial expression with facial appearance
Among many biometric characteristics, the facial biometric is considered to be the least intrusive technology that can be deployed in the real-world visual surveillance environment. However, in facial biometric, little research attention has been paid ...
Comments