3D Landmarking in Multiexpression Face Analysis: A Preliminary Study on Eyebrows and Mouth

Abstract

The application of three-dimensional (3D) facial analysis and landmarking algorithms in the field of maxillofacial surgery and other medical applications, such as diagnosis of diseases by facial anomalies and dysmorphism, has gained a lot of attention. In a previous work, we used a geometric approach to automatically extract some 3D facial key points, called landmarks, working in the differential geometry domain, through the coefficients of fundamental forms, principal curvatures, mean and Gaussian curvatures, derivatives, shape and curvedness indexes, and tangent map. In this article we describe the extension of our previous landmarking algorithm, which is now able to extract eyebrows and mouth landmarks using both old and new meshes. The algorithm has been tested on our face database and on the public Bosphorus 3D database. We chose to work on the mouth and eyebrows as a separate study because of the role that these parts play in facial expressions. In fact, since the mouth is the part of the face that moves the most and affects mainly facial expressions, extracting mouth landmarks from various facial poses means that the newly developed algorithm is pose-independent.

No Level Assigned

This journal requires that authors assign a level of evidence to each submission to which Evidence-Based Medicine rankings are applicable. This excludes Review Articles, Book Reviews, and manuscripts that concern Basic Science, Animal Studies, Cadaver Studies, and Experimental Studies. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors http://www.springer.com/00266.

Appendix

The first and second fundamental forms are used to measure distance on surfaces and are defined by

$$\begin{gathered} E{\text{d}}u^{2} + 2F{\text{d}}u{\text{d}}v + G{\text{d}}v^{2} , \hfill \\ e{\text{d}}u^{2} + 2f{\text{d}}u{\text{d}}v + g{\text{d}}v^{2} , \hfill \\ \end{gathered}$$

respectively, where E, F, G, e, f, and g are their coefficients. Curvatures are used to measure how a regular surface x bends in R ³. If D is the differential and N is the normal plane of a surface, then the determinant of DN is the product of the principal curvatures, (−k ₁) (−k ₂) = k ₁ k ₂, and the trace of DN is the negative of the sum of principal curvatures, −(k ₁ + k ₂). In point P, the determinant of DN _P is the Gaussian curvature K of x at P. The negative of half of the trace of DN is called the mean curvature H of x at P. In terms of the principal curvatures, K and H can be written

$$\begin{gathered} K = k_{1} k_{2} , \hfill \\ H = \frac{{k_{1} + k_{2} }}{2}. \hfill \\ \end{gathered}$$

Some definitions of these descriptors are given. These are the forms implemented in the algorithm:

$$\begin{gathered} E = 1 + h_{x}^{2} , \hfill \\ F = h_{x} h_{y} , \hfill \\ G = 1 + h_{y}^{2} , \hfill \\ e = \frac{{h_{xx} }}{{\sqrt {1 + h_{x}^{2} + h_{y}^{2} } }}, \hfill \\ f = \frac{{ - h_{xy} }}{{\sqrt {1 + h_{x}^{2} + h_{y}^{2} } }}, \hfill \\ g = \frac{{ - h_{yy} }}{{\sqrt {1 + h_{x}^{2} + h_{y}^{2} } }}, \hfill \\ K = \frac{{h_{xx} h_{yy} - h_{xy}^{2} }}{{\left( {1 + h_{x}^{2} + h_{y}^{2} } \right)^{2} }}, \hfill \\ H = \frac{{\left( {1 + h_{x}^{2} } \right)h_{yy} - 2h_{x} h_{y} h_{xy} + \left( {1 + h_{y}^{2} } \right)h_{xx} }}{{\left( {1 + h_{x}^{2} + h_{y}^{2} } \right)^{3/2} }}, \hfill \\ k_{1} = H + \sqrt {H^{2} - K} , \hfill \\ k_{2} = H - \sqrt {H^{2} - K} , \hfill \\ \end{gathered}$$

where h is a differentiable function z = h(x,y). It is, therefore, convenient to have at hand formulas for the relevant concepts in this case. To obtain such formulas, let us parameterize the surface by

$$x\left( {u,v} \right) = \left( {u,v,h\left( {u,v} \right)} \right),\quad\left( {u,v} \right) \in U,$$

where u = x and v = y.

The most used descriptors are the shape and curvedness indexes S and C, introduced by Koenderink and Van Doorn [14]:

$$\begin{gathered} S = - \frac{2}{\pi }\arctan \frac{{k_{1} + k_{2} }}{{k_{1} - k_{2} }},\,S \in \left[ { - 1,1} \right],\,k_{1} \ge k_{2} , \hfill \\ C = \sqrt {\frac{{k_{1}^{2} + k_{2}^{2} }}{2}} . \hfill \\ \end{gathered}$$

For the role they play in the work, a little digression about their significance is needed. Their meaning is shown in Figs. 19, 20, 21 and in Table 1.

Fig. 19

Illustration of the Shape Index scale divided into seven categories. Different subintervals of its range [–1, 1] correspond to seven geometric surfaces

Fig. 20

Curvedness Index scale; range is (−∞, ∞)

Fig. 21

Indexes (S, C) are viewed as polar coordinates in the (k ₁, k ₂) plane, with planar points mapped to the origin. The effects on surface structure from variations in the Curvedness Index (radial coordinate) and Shape Index (angular coordinate) parameters of curvature, and the relationship of these components to the principal curvatures k ₁ and k ₂. The degree of curvature increases radially from the center

Table 1 Topographic classes