Pascal Bérard
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Facial scanning has become ubiquitous in digital media, but so far most efforts have focused on reconstructing the skin. Eye reconstruction, on the other hand, has received only little attention, and the current state-of-the-art method is cumbersome for the actor, time-consuming, and requires carefully setup and calibrated hardware. These constraints currently make eye capture impractical for general use. We present the first approach for high-quality lightweight eye capture, which leverages a database of pre-captured eyes to guide the reconstruction of new eyes from much less constrained inputs, such as traditional single-shot face scanners or even a single photo from the internet. This is accomplished with a new parametric model of the eye built from the database, and a novel image-based model fitting algorithm. Our method provides both automatic reconstructions of real eyes, as well as artistic control over the parameters to generate user-specific eyes.
Supplemental Material
Available for Download
zip
a117-berard-supp.zip (165 MB)
Supplemental files.
- Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 587--594. Google Scholar
Digital Library
- Amberg, B., Romdhani, S., and Vetter, T. 2007. Optimal step nonrigid icp algorithms for surface registration. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on, IEEE, 1-8.Google Scholar
- Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. Scape: shape completion and animation of people. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 408-416. Google ScholarDigital Library
- Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
- Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C, Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 75. Google ScholarDigital Library
- Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, 223:1--223:12. Google ScholarDigital Library
- Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. of the 26th annual conference on Computer graphics and interactive techniques, 187--194. Google ScholarDigital Library
- Boykov, Y., and Kolmogorov, V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 9, 1124--1137. Google ScholarDigital Library
- Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 41. Google ScholarDigital Library
- Brown, B. J., and Rusinkiewicz, S. 2004. Non-rigid range-scan alignment using thin-plate splines. In 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004. Proceedings. 2nd International Symposium on, IEEE, 759--765. Google ScholarDigital Library
- Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33, 4, 43. Google ScholarDigital Library
- Chen, K., Johan, H., and Mueller-Wittig, W. 2013. Simple and efficient example-based texture synthesis using tiling and deformation. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, 145--152. Google ScholarDigital Library
- Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, 341--346. Google ScholarDigital Library
- Flickr. 2006. Mohammed Alnaser - Mr. Falcon. Creative Commons - Attribution 2.0. https://www.flickr.com/photos/69er/324313066.Google Scholar
- Francois G. Gautron P. Breton G. and Bouatouch K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815--827. Google ScholarDigital Library
- Funkhouser T. Kazhdan M. Shilane P. Min P. Kiefer W. Tal A. Rusinkiewicz S. and Dobkin D. 2004. Modeling by example. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 652--663. Google ScholarDigital Library
- Fyffe G. Hawkins T. Watts C. Ma W.-C. and Debevec P. 2011. Comprehensive facial performance capture. In Eurographics.Google Scholar
- Fyffe, G., Jones, A., Alexander, O., Ichikari, R., and Debevec, P. 2014. Driving high-resolution facial scans with video performance capture. ACM Trans. Graphics 34, 1, 8:1--8:14. Google ScholarDigital Library
- Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 32, 6, 158. Google ScholarDigital Library
- Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129. Google ScholarDigital Library
- Guo, B., Liang, L., Liu, C., Shum, H.-Y., and Xu, Y. 2001. Real-time texture synthesis by patch-based sampling.Google Scholar
- Haehnel, D., Thrun, S., and Burgard, W. 2003. An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In IJCAI, vol. 3, 915--920. Google ScholarDigital Library
- Ikemoto, L., Gelfand, N., and Levoy, M. 2003. A hierarchical method for aligning warped meshes. In 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings. Fourth International Conference on, IEEE, 434--441.Google Scholar
- Jacob, M., and Unser, M. 2004. Design of steerable filters for feature detection using canny-like criteria. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 8, 1007--1019. Google ScholarDigital Library
- Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. 2004. Shape matching and anisotropy. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 623--629. Google ScholarDigital Library
- Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. In ACM Transactions on Graphics (ToG), vol. 22, ACM, 277--286. Google ScholarDigital Library
- Lefohn, A., Budge, B., Shirley, P., Caruso, R., and Reinhard, E. 2003. An ocularist's approach to human iris synthesis. IEEE CG&A 23, 6, 70--75. Google ScholarDigital Library
- Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. In Computer graphics forum, vol. 27, Wiley Online Library, 1421--1430. Google ScholarDigital Library
- Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 117--122. Google ScholarDigital Library
- Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Proc. Rendering Techniques, 183--194. Google ScholarDigital Library
- Mohammed, U., Prince, S. J., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Transactions on Graphics (TOG) 28, 3, 57. Google ScholarDigital Library
- Palubicki, W., Horel, K., Longay, S., Runions, A., Lane, B., Měch, R., and Prusinkiewicz, P. 2009. Self-organizing tree models for image synthesis. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 58. Google ScholarDigital Library
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 313--318. Google ScholarDigital Library
- Praun, E., Finkelstein, A., and Hoppe, H. 2000. Lapped textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 465--470. Google ScholarDigital Library
- Ramanarayanan, G., and Bala, K. 2007. Constrained texture synthesis via energy minimization. IEEE TVCG 13, 1. Google ScholarDigital Library
- Rozenberg, G., and Salomaa, A. 1976. The mathematical theory of L systems. Springer. Google ScholarDigital Library
- Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69--91.Google Scholar
- Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205--212. Google ScholarDigital Library
- Shen, H.-L., and Cai, Q.-Y. 2009. Simple and efficient method for specularity removal in an image. Applied optics 48, 14, 2711--2719.Google Scholar
- Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, ACM, 175--184. Google ScholarDigital Library
- Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In Computer Vision--ECCV 2014. Springer, 796--812.Google Scholar
- Tai, Y.-W., Liu, S., Brown, M. S., and Lin, S. 2010. Super resolution using edge prior and single image detail synthesis. In CVPR.Google Scholar
- Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187. Google ScholarDigital Library
- Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 426--433. Google ScholarDigital Library
- Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In Eurographics 2009, State of the Art Report, EG-STAR, Eurographics Association, 93--117.Google Scholar
- Wikimedia Commons. 1485. Sandro Botticelli - The Birth of the Venus. Public Domain. https://commons.wikimedia.org/wiki/File:Venus_botticelli_detail.jpg.Google Scholar
- Wikimedia Commons. 1887. Vincent Van Gogh - Self-Portrait. Public Domain. https://commons.wikimedia.org/wiki/File:VanGogh_1887_Selbstbildnis.jpg.Google Scholar
- Wikimedia Commons. 2006. Blue Eye Image. GNU Free Documentation License Version 1.2. https://commons.wikimedia.org/wiki/File:Blueye.JPG.Google Scholar
Index Terms
- Lightweight eye capture using a parametric model
Recommendations
High-quality capture of eyes
Even though the human eye is one of the central features of individual appearance, its shape has so far been mostly approximated in our community with gross simplifications. In this paper we demonstrate that there is a lot of individuality to every eye, ...
Dual-State Parametric Eye Tracking
FG '00: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000Most eye trackers work well for open eyes. However, blinking is a physiological necessity for humans. Moreover, for applications such as facial expression analysis and driver awareness systems, we need to do more than tracking the locations of the ...
Eye&Head: Synergetic Eye and Head Movement for Gaze Pointing and Selection
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and TechnologyEye gaze involves the coordination of eye and head movement to acquire gaze targets, but existing approaches to gaze pointing are based on eye-tracking in abstraction from head motion. We propose to leverage the synergetic movement of eye and head, and ...
Comments