Robust 3D face modeling and tracking from RGB-D images

We address the issue of 3D face modeling and tracking from RGB-D images. Existing methods usually fit a deformable model to an RGB-D image using iterative closest point algorithm. Due to the noise and occlusion of the depth image, these methods are not robust enough. To solve this issue, we propose a method for robust 3D face modeling and tracking. For an input RGB-D face image, our method first estimates the initial head pose of a person using random forests. Then, a generic bilinear face model is fitted to the RGB-D image using iterative closest point algorithm. To improve the accuracy and robustness of face modeling, an optimal weight for each face vertex is integrated into the fitting procedure. The distances between facial landmarks are also used to better estimate facial expressions. Finally, the head pose, the identity, and expression parameters of the bilinear face model are jointly optimized. Experiments show that our method can generate accurate 3D face models from an RGB-D image or image sequence. The method can also robustly track the face even if the person is with large head rotations and various facial expressions.