Rectified Wing Loss for Efficient and Robust Facial Landmark Localisation with Convolutional Neural Networks

Facial landmark localisation, also known as face alignment, aims to automatically localise a set of pre-defined 2D key points for a given facial image. A facial landmark usually has a specific semantic meaning, e.g. nose tip or eye centre, which provides rich geometric information for other face analysis tasks such as face recognition (Taigman et al. 2014; Masi et al. 2016a; Liu et al. 2017a; Yang et al. 2017b; Wu and Ji 2019; Deng et al. 2019), emotion estimation (Fabian Benitez-Quiroz et al. 2016; Walecki et al. 2016; Li et al. 2017; Zeng et al. 2009) and 3D face reconstruction (Kittler et al. 2016; Roth et al. 2016; Koppen et al. 2018; Deng et al. 2018; Feng et al. 2018a).

Thanks to the successive developments in this area of research during the past decades, we are able to achieve accurate facial landmark localisation in constrained scenarios even using traditional approaches such as active shape model (Cootes et al. 1995) and active appearance model (Cootes et al. 2001). The existing challenge is to perform efficient and robust landmark localisation of unconstrained faces that are impacted by a variety of appearance variations, e.g. in pose, expression, illumination, image blur and occlusion. To address this challenge, Cascaded Shape Regression (CSR) has been widely used. The key idea of CSR is to form a strong regressor by cascading a set of weak regressors (Doll et al. 2010; Xiong and Torre 2013). CSR-based facial landmark localisation approaches have proved to be very successful, delivering promising performance in terms of both accuracy and efficiency (Cao et al. 2014; Feng et al. 2015b; Ren et al. 2016; Wu et al. 2017a; Feng et al. 2017a; Jourabloo and Liu 2017). However, the capability of CSR is practically saturated due to its shallow structure. After cascading more than four or five weak regressors, the performance of CSR is hard to improve further (Sun et al. 2015; Feng et al. 2015a). More recently, deep neural networks have been put forward as a more powerful alternative in a wide range of computer vision and pattern recognition tasks, including facial landmark localisation (Sun et al. 2013; Zhang et al. 2016b, a; Lv et al. 2017; Yang et al. 2017a; Wu et al. 2017b; Ranjan et al. 2017).

To perform robust facial landmark localisation with deep neural networks, different network architectures have been explored, including Convolutional Neural Networks (CNN) (Sun et al. 2013; Feng et al. 2019), auto-encoder (Zhang et al. 2014; Weng et al. 2016), deep belief networks (Luo et al. 2012) and recurrent neural networks (Trigeorgis et al. 2016; Xiao et al. 2016). In general, deep-learning-based facial landmark localisation approaches can be divided into two main categories: regression-based (Trigeorgis et al. 2016; Lv et al. 2017; Feng et al. 2019) and heatmap-based (Yang et al. 2017a; Deng et al. 2019b; Bulat and Tzimiropoulos 2017a, b; Wu et al. 2018). For regression-based methods, a network directly outputs a vector consisting of the 2D coordinates of all the landmarks. In contrast, a heatmap-based method outputs multiple heatmaps, each corresponding to a single facial landmark. The intensity value of a pixel in a heatmap indicates the probability that its location is the predicted position of the corresponding landmark. Despite the success of heatmap-based approaches in landmark localisation, they are computationally expensive. Such a method cannot meet the requirements for the deployment in real-time facial analysis systems. In this paper, we focus on regression-based facial landmark localisation due to the fast inference speed.

One crucial aspect of regression-based facial landmark localisation with CNNs is to define a loss function leading to a better-learnt representation from underlying data. However, this aspect of the design seems to be scarcely investigated by the community. To the best of our knowledge, most existing regression-based facial landmark localisation approaches with deep neural networks are based on the L2 loss function (Lv et al. 2017; Dong et al. 2018b; Zeng et al. 2018). However, it is well known that the L2 loss function is sensitive to outliers, which has been noted in connection with the bounding box regression problem in face detection (Girshick 2015). Rashid et. al.  Rashid et al. (2017) also noticed this issue and used the smooth L1 loss function instead of L2. Additionally, outliers are not the only subset of points which deserve special consideration. We argue that the behaviour of the loss function at points exhibiting small-medium errors is just as crucial to finding a good solution to the facial landmarking problem. Based on more detailed analysis, we propose a new loss function, namely Rectified Wing (RWing) loss, for robust facial landmark localisation with CNNs. The main contributions of our work include:The rest of this paper is organised as follows. Section 2 presents a brief review of the related literature. The regression-based facial landmarking problem with CNNs is formulated in Sect. 3. The properties of common loss functions (L1, smooth L1 and L2) are discussed in Sect. 4 which also motivate the introduction of the novel RWing loss function in Sect. 5. The pose-based data balancing strategy is the subject of Sect. 6. The coarse-to-fine localisation framework is presented in Sect. 7. The advocated approach is validated experimentally in Sect. 8 and the paper is drawn to conclusion in Sect. 9.

In the last section, we mentioned some traditional facial landmark localisation algorithms, e.g. active shape model (Cootes et al. 1995), active appearance model (Cootes et al. 2001), constrained local model (Cristinacce and Cootes 2006) and cascaded shape regression (Doll et al. 2010; Xiong and Torre 2013). As the current mainstream of the area is to use deep neural networks, this section focuses on deep-learning-based methods. For traditional facial landmark localisation approaches, a reader is referred to comprehensive surveys (Wu and Ji 2019; Wang et al. 2018; Gao et al. 2010).

Network Architecture Most existing deep-learning-based facial landmark localisation approaches use regression networks. For such a landmarking task, the most straightforward way is to use a CNN model with a regression output layer (Sun et al. 2013; Rashid et al. 2017; Feng et al. 2019). The input for a regression CNN is usually an image patch enclosing the whole face region and the output is a vector consisting of the 2D coordinates of facial landmarks. Figure 1 depicts an example of CNN-based facial landmark localisation with the whole face region as an input. Instead of the whole face image, shape- or landmark-related local patches have also been widely used in deep-learning-based facial landmark localisation (Trigeorgis et al. 2016; Sun et al. 2013). To use local patches, one can apply CNN-based feature extraction to the neighbourhoods of all the landmarks and concatenate the extracted local features for landmark prediction or update (Trigeorgis et al. 2016). The advantage of the use of an image with a whole face region, in which the only input of the network is a cropped face image, is that it does not require the initialisation of facial landmarks. In contrast, to use local patches, a system usually requires initial estimates of facial landmarks for any given image. This can be achieved by either using the mean facial landmarks (Trigeorgis et al. 2016) or the output of a network coarsely landmarking the whole face image (Sun et al. 2013; Lv et al. 2017; Xiao et al. 2017).

Besides regression-based facial landmark localisation methods, recently heatmap-based variants have been proposed for the task and shown to deliver promising results, e.g. fully convolutional network  (Liang et al. 2015) and the hourglass network (Newell et al. 2016; Yang et al. 2017a; Deng et al. 2019b; Bulat and Tzimiropoulos 2017a, b; Wu et al. 2018). To reduce false alarms of a generated 2D sparse heatmap,  Wu et al. (2018b) proposed a distance-aware softmax function that facilitates the training of their dual-path network. Wu et al. (2018) proposed to create a boundary heatmap mask using hourglass network for feature map fusion and showed its beneficial impact on the landmark localisation accuracy.

As noted in the last section, heatmap-based facial landmark localisation approaches are computationally expensive, which becomes an obstacle for the deployment of a network in real-time facial analysis systems. In this paper, we focus on efficient regression-based methods with CNNs. Thanks to the extensive studies of different deep neural network architectures and their use in unconstrained facial landmark localisation, the development of regression-based systems has recently been greatly stimulated. However, the current research lacks a systematic analysis of the effect of different loss functions on the solution. In this paper, we close this gap and design a new loss function for regression-based facial landmark localisation with CNNs.

Dealing with Pose Variations Extreme pose variations give rise to many difficulties in unconstrained facial landmark localisation. To mitigate this issue, different strategies have been explored. The first opts for multi-view models. There is a long history of the use of multi-view models in landmark localisation, from the earlier studies (Romdhani et al. 1999; Cootes et al. 2002) to recent work on cascaded-shape-regression-based (Xiong and Torre 2015; Zhu et al. 2016a; Feng et al. 2017b) and deep-learning-based approaches (Deng et al. 2019b). For example, we proposed to train multi-view cascaded shape regression models using a fuzzy membership weighting strategy, which, interestingly, outperformed even some deep-learning-based approaches (Feng et al. 2017b). The second strategy, which has become very popular in recent years, is to use 3D face models (Zhu et al. 2016b; Jourabloo and Liu 2016; Bhagavatula et al. 2017; Liu et al. 2017b; Jourabloo et al. 2017; Xiao et al. 2017). By recovering the 3D shape and estimating the pose of a given 2D face image, the issue of extreme pose variations can be alleviated to a great extent. 3D face models have also been widely used to synthesise additional 2D face images with extreme pose variations for the training of a pose-invariant system (Masi et al. 2016b; Feng et al. 2015a; Zhu et al. 2016b). Last, multi-task learning has been adopted to address the difficulties posed by image degradation, including pose variation. For example, face attribute estimation, pose estimation or 3D face reconstruction can jointly be trained with facial landmark localisation (Zhang et al. 2016b; Xu and Kakadiaris 2017; Ranjan et al. 2017). The collaboration of different tasks in a multi-task learning framework can boost the performance of individual sub-tasks.

In contrast to these approaches, we treat the pose challenge as a training data imbalance problem and advocate a pose-based data balancing strategy to address this issue.

Cascaded Networks Motivated by the well known benefits of coarse-to-fine cascaded shape regression, multiple networks can be stacked to boost the performance further. To this end, shape- or landmark-related features should be used to satisfy the training of multiple networks in cascade. However, a CNN using a global face image as input cannot meet this requirement. To address this issue, one solution is to use local CNN features. This idea is advocated, for example, by Trigeorgis et. al.  Trigeorgis et al. (2016) who use CNN for local feature extraction and a recurrent neural network for landmark localisation in an end-to-end training fashion. As an alternative, one can train a network based on the global image patch for rough facial landmark localisation. Then, for each landmark or a composition of multiple landmarks in a specific region of the face, a new network is trained to perform fine-grained landmark prediction (Sun et al. 2013; Dong and Wu 2015; Weng et al. 2016; Yu et al. 2016; Xu and Kakadiaris 2017; Lv et al. 2017).

In this paper, we advocate a coarse-to-fine localisation framework. The first coarse network is very simple. It performs coarse facial landmark localisation at a very high speed. The aim of the first network is to mitigate the difficulties posed by inaccurate face detection and in-plane head rotations. The second CNN performs fine-grained landmark localisation by applying rigid transformation to an input image with the facial landmarks estimated by the first CNN. More importantly, we analyse the advantages of using two small networks compared to a single large-capacity network, in terms of both accuracy and speed.

As depicted in Fig. 1, the task of regression-based facial landmark localisation using CNNs is to find a nonlinear mapping function:that outputs a shape vector \(\mathbf {s} \in \mathbb {R}^{2L}\) for a given input colour image \(\mathcal {I} \in \mathbb {R}^{H \times W \times 3}\). The input image is usually cropped from a bounding box output by a face detector. The shape vector is in the form of:where L is the number of pre-defined 2D facial landmarks and \((x_l, y_l)\) are the coordinates of the lth landmark. To obtain this mapping, first, we have to define a multi-layer neural network with randomly initialised parameters. In fact, a deep neural network is a compositional function:consisting of M sub-functions, in which each sub-function (\(\phi \)) stands for a specific layer in the network.

Given a set of labelled training samples \(\Omega = \{\mathcal {I}_i, \mathbf {s}_i\}_{i=1}^{N}\), the target of CNN training is to find a \(\Phi \) that minimises:where loss() is a pre-defined loss function that measures the difference between a predicted shape vector and its ground truth value. In this case, the CNN is used as a regression model learned in a supervised manner. To optimise the above objective function, a variety of optimisation methods, such as Stochastic Gradient Descent (SGD),  Zeiler (2012) and  Kingma and Ba (2014), can be used. In this paper, we use SGD with momentum for network training. Note that we also tested other optimisation approaches, but none of them resulted in higher accuracy than SGD.

In this section, we systematically analyse the impact of different loss functions as well as network architectures on regression-based facial landmark localisation. To the best of our knowledge, this is the first work in the area performing such a systematic analysis using different loss functions and CNN architectures.

We compare three different loss functions, including L2, L1 and smooth L1, using four different plain CNN architectures. The configurations of these plain CNN networks are shown in Table 1. In the rest of this paper, we use the term ‘CNN-5/6/7/8’ for these CNN models. Note that we do not use any fancy techniques, such as residual connection or intermediate supervision, in these plain CNN architectures so as not to cloud the comparison across different loss functions and network architectures with additional factors. We evaluate the performance of other network architectures such as MobileNets (Howard et al. 2017; Sandler et al. 2018), VGG (Parkhi et al. 2015) and ResNet (He et al. 2017) in Sect. 8.2.1.

The input of a plain CNN architecture is a colour image and the output is a vector of 2L real numbers consisting of the coordinates of L 2D facial landmarks. Each plain CNN has multiple convolutional layers with \(3\times 3\) kernels, a fully connected layer and an output layer. After each convolutional and fully connected layer, a standard ReLU layer is used for nonlinear activation. A Max pooling layer following each ReLU layer is used to downsize the feature map to half of the size. As an example, Fig. 2 depicts the detailed architecture of our CNN-6 network.

Given a training image \(\mathcal {I}\) and a network \(\Phi \), we can predict the facial landmarks as a vector \(\mathbf {s}' = \Phi (\mathcal {I})\). The loss function is defined as:where \(\mathbf {s}\) is the ground-truth shape vector of the facial landmarks and \(s_i\) is its ith element. For f(x) in the above equation, the L2 loss is defined as:and the L1 loss is defined as:For the smooth L1 loss, f(x) is piecewise-defined as:which is quadratic for small values of |x| and linear for large values (Girshick 2015). More specifically, smooth L1 uses \(f_{L2}(x)\) for \(x\in (-1,1)\) and shifted \(f_{L1}(x)\) elsewhere. Figure 3 depicts the plots of these three loss functions. It should be noted that the smooth L1 loss is a special case of the Huber loss (Huber 1964). The loss function that has widely been used in facial landmark localisation is L2. However, L2 loss is sensitive to outliers.

To perform empirical analysis, we use the AFLW dataset with the AFLW-Full protocol (Zhu et al. 2016a).¹ This protocol consists of 20,000 training and 4386 test images. Each image has 19 manually annotated facial landmarks. We train the plain CNN networks on AFLW using three different loss functions. In addition, we compare the results obtained by these CNN networks with five state-of-the-art baseline algorithms (Feng et al. 2017b; Lv et al. 2017; Dong et al. 2018b, a; Wu et al. 2018b). The first baseline method is a multi-view cascaded shape regression model, namely Dynamic Attention Controlled Cascaded Shape Regression (DAC-CSR) (Feng et al. 2017b). The other four baseline approaches are all deep-learning-based, including the Two-stage Re-initialisation Deep Regression Network (TR-DRN) (Lv et al. 2017), Supervision-by-Registration (SBR) (Dong et al. 2018b), Style Aggregated Network (SAN) (Dong et al. 2018a) and the Globally Optimised Dual Pathway neural network (GoDP) (Wu et al. 2018b). A comparison with more state-of-the-art algorithms on the AFLW dataset is reported in Sect. 8.

The results are reported in Table  2. The L2 loss function, which has been widely used for facial landmark localisation, obtains competitive results as compared with the baseline methods. Surprisingly, by simply switching the loss function from L2 to L1 or smooth L1, the landmarking error can be significantly reduced. CNN-7 outperforms DAC-CSR, TR-DRN, CPM+SBR and SAN in terms of accuracy and performs equally well as the GoDP approach. The combination of CNN-8 with L1 or smooth L1 beats all the baseline approaches. The NME of CNN-8 using the L1 loss function is \(1.72\times 10^{-2}\), which is around \(7\%\) lower than GoDP that has the NME of \(1.84\times 10^{-2}\).

Another conclusion is that, a deeper network with higher resolution input images usually performs better in accuracy. This finding has also been validated in many other CNN-based computer vision and pattern recognition tasks, e.g. in VGG (Parkhi et al. 2015) and ResNet (He et al. 2017). To boost the performance in accuracy, more powerful network architectures are suggested, such as our coarse-to-fine landmark localisation framework presented in Sect. 7, VGG and ResNet. We will report the results of these advanced network architectures in Sect. 8.2.1. But the use of deeper and wider neural networks increases the computational complexity dramatically. For example, the model parameter and model size increase around four times by upgrading each plain network to a higher level, e.g. from CNN-6 to CNN-7, as shown in Table 1. Accordingly, the FLoating Point Operations (FLOPs) increase around five times. In the next section, we propose a new loss function that brings further performance boosting for lightweight networks.

As analysed in the last section, the design of a proper loss function is crucial for regression-based facial landmark localisation with CNNs. However, predominantly the L2 loss has been used in existing deep-neural-network-based facial landmarking systems, in spite of the findings supporting the use of the L1 and smooth L1 loss functions (Girshick 2015; Rashid et al. 2017). Inspired by our analysis, we propose a new loss function, namely Rectified Wing (RWing) loss, to further improve the accuracy of a CNN-based facial landmark localisation system.

We first compare the results obtained on the AFLW dataset using four plain CNN architectures and three different loss functions (L2, L1 and smooth L1) in Fig. 4 by plotting the Cumulative Error Distribution (CED) curves. On one hand, we can see that all the loss functions analysed in the last section perform well for large errors, regardless of the choice of the CNN architecture. This indicates that the training of a neural network should pay more attention to the samples with small or medium range errors. On the other hand, it is very hard to achieve very small errors even for large-capacity networks, e.g. CNN-7 and CNN-8. The main reason stems from the residual noise in the ground truth labelling of the training data. These inaccuracies suggest that we should ignore very small errors in CNN training. To satisfy these two observations, we propose the RWing loss for CNN-based facial landmark localisation.

In order to motivate the new loss function, we provide an intuitive analysis of the properties of the classical loss functions, as shown in Fig. 3. We also plot their corresponding influence functions (derivatives) in Fig. 5. As shown in the figure, the magnitude of the gradients of the L1 and L2 loss functions is 1 and |x| respectively, and the magnitude of the corresponding optimal step sizes should be |x| and 1. Finding the minimum in either case is straightforward. However, the situation becomes more complicated when we try to optimise simultaneously the location of multiple points, as in our problem of facial landmark localisation formulated in Eq. (5). In both cases the update towards the solution will be dominated by larger errors. In the case of L1, the magnitude of the gradient is the same for all the points, but the step size is disproportionately influenced by larger errors. For L2, the step size is the same but the gradient will be dominated by large errors. Thus in both cases it is hard to correct relatively small displacements.

The influence of small errors can be enhanced by an alternative loss function, such as \(\ln x\). Its gradient, given by 1 / x, increases as we approach zero error. The magnitude of the optimal step size is \(x^2\). When compounding the contributions from multiple points, the gradient will be dominated by small errors, but the step size by larger errors. This restores the balance between the influence of errors of different sizes. However, to prevent making large update steps in a potentially wrong direction, it is important not to overcompensate the influence of small localisation errors. This can be achieved by opting for a logarithm function with a positive offset. In addition, to eliminate the effects posed by noise, we rectify the loss function for very small values.

The above intuitive argument points to a loss function which for very small errors should have the value of zero, for small medium range errors behave as a logarithm function with an offset, and for larger errors as L1. Such a loss function can be piecewise defined as:where the non-negative parameter r sets the range of rectified region to \((-r, r)\) for very small values. For small medium range values with the absolute value in [r, w), we use a modified logarithm function, where \(\epsilon \) limits the curvature of the nonlinear region and \( C = w - w\ln ({1 + (w -r)/\epsilon })\) is a constant that smoothly links the piecewise-defined linear and nonlinear parts. Note that we should not set \(\epsilon \) to a very small value because this would make the training of a network very unstable and cause the exploding gradient problem for small errors. In fact, the nonlinear part of our RWing loss function just simply takes a part of the curve of \(\ln (x)\) and scales it along both the X-axis and Y-axis. Also, we apply translation along the Y-axis to allow \(RWing(\pm r) = 0\) and to impose continuity on the loss function at \(\pm w\). Figure 6 depicts our RWing loss using different parameter settings.

We compare our RWing loss with other loss functions in Table 2 and Fig. 4. According to the figure, our RWing loss outperforms L2, L1 and smooth L1 in terms of accuracy for all the plain networks, i.e. CNN-5/6/7/8. Although the improvement for CNN-8 in Fig. 4 seems not obvious, the actual NME is reduced from \(1.72 \times 10^{-2}\) of L1 to \(1.63 \times 10^{-2}\), which is around \(6\%\) lower than the best result obtained in the last section for CNN-8 (Table 2), and \(11\%\) lower than the best baseline approach, i.e. GoDP (Wu et al. 2018b). Additionally, by virtue of the proposed RWing loss, smaller networks are able to perform equally well or even better than larger networks. For example, CNN-6 is four times smaller in size as compared with CNN-7. But the NME of CNN-6 powered by our RWing loss is \(1.77\times 10^{-2}\), which is smaller than the NMEs of CNN-7 trained with L2 loss (\(2.35\times 10^{-2}\)), L1 loss (\(1.85\times 10^{-2}\)) and smooth L1 loss (\(1.85\times 10^{-2}\)). This validates the effectiveness of the proposed RWing loss for the training of lightweight CNNs.

As the facial landmarks of a training image are labelled by human annotators, they will be subject to individual biases of different annotators. Moreover, if we ask the same annotator to label the facial landmarks of the same image twice, the results will be slightly different, even for some landmarks with a clear semantic meaning such as eye corner or mouth corner. Note that the Wing loss function without rectification will have the highest gradient when a training sample has a very small error that might be caused by annotation noise. As aforementioned, for network training, a sample with very small errors should be ignored in back propagation. This observation motivates the idea of rectifying the Wing loss for very small errors. To validate the effectiveness of the rectification measure, we compared the performance of RWing loss and the Wing loss function without rectification. The results are reported in Table 2. According to the table, both Wing loss functions (with and without rectification) outperform L2, L1 and smooth L1 in accuracy, regardless of the network architecture. However, our rectified Wing loss has a slight performance edge over the Wing loss function without rectification. This confirms the merit of the proposed RWing loss function for regression-based facial landmark localisation.

Extreme pose variations are very challenging for robust facial landmark localisation. To mitigate this problem, we propose a simple but very effective Pose-based Data Balancing (PDB) strategy. We argue that the difficulty for accurately localising faces with large poses is mainly due to data imbalance, which is a well-known problem in many computer vision applications (Shrivastava et al. 2016). For example, given a training dataset, most samples in it are likely to be near-frontal faces. The network trained on such a dataset is dominated by frontal faces. By over-fitting to the frontal pose it cannot adapt well to faces with large poses. In fact, the difficulty of training and testing on merely frontal faces should be similar to that on profile faces. This is the main reason why a view-based face analysis algorithm usually works well for pose-varying faces. As an evidence, even the classical view-based active appearance model can localise faces with large poses very well (up to \(90^{\circ }\) in yaw) Cootes et al. (2000).

To perform PDB, we first align all the training shapes to a reference shape using Procrustes Analysis. Then we apply Principal Component Analysis (PCA) to the aligned shapes and project them to the one dimensional space defined by the shape eigenvector (pose space) controlling pose variations. To be more specific, for a training dataset \(\{\mathbf {s}_i\}_{i=1}^N\) with N samples, where \(\mathbf {s}_i \in \mathbb {R}^{2L}\) is the ith training shape vector consisting of the 2D coordinates of all the L landmarks, the use of Procrustes Analysis aligns all the training shapes to a reference shape, i.e. the mean shape, using rigid transformations. Then we can approximate any training shape or a new shape, \(\mathbf {s}\), using a statistical linear shape model:where \(\bar{\mathbf {s}} = \frac{1}{N}\sum _{i=1}^{N} \mathbf {s}_i\) is the mean shape over all the training samples, \(\mathbf {s}_j^*\) is the jth eigenvector obtained by applying PCA to all the aligned training shapes and \(p_j\) is the coefficient of the jth shape eigenvector. Among those shape eigenvectors, we can find an eigenvector, usually the first one, that controls the pose variation. We denote this eigenvector as \(\hat{\mathbf {s}}\). Then we can obtain the pose coefficient of each training sample \(\mathbf {s}_i\) as:We plot the distribution of the pose coefficients of all the AFLW training samples in Fig. 7. According to the figure, we can see that the AFLW dataset is not well-balanced in terms of pose variation.

With the pose coefficients of all the training samples, we first categorise the training dataset into K subsets. Then we balance the training data by duplicating the samples falling into the subsets of lower cardinality. To be more specific, we denote the number of training samples of the kth subset as \(B_k\) and the maximum size of the K subsets as \(B^*\). To balance the whole training dataset in terms of pose variation, we add more training samples to the kth subset by randomly sampling \(B^*-B_k\) samples from the original kth subset. Then all the subsets have the size of \(B^*\) and the total number of training samples is increased from \(\sum _{k=1}^{K}B_k\) to \(KB^*\). It should be noted that we perform pose-based data balancing before network training by randomly duplicating some training samples of each subset of lower occupancy. Additionally, we modify each duplicated training image online with random image rotation, bounding box perturbation and other data augmentation approaches, as introduced in Sect. 8.1. After pose-based data balancing, the training samples of each mini-batch is randomly sampled from the balanced training dataset for network training. As the samples with different poses have the same probability to be sampled for a mini-batch, the network training is pose balanced.

We compare the performance of the four plain CNN architectures on the AFLW dataset in Table 3, using four different loss functions as well as the proposed PDB strategy. Note that, for a fair comparison, we also apply data augmentation to the training samples when we train a network without PDB. From the table, we can see that PDB improves the performance of all different CNN architectures in accuracy, in spite of the choice of loss functions.

Besides out-of-plane head rotations, the accuracy of a facial landmark localisation algorithm can be degraded by other factors, such as in-plane rotations and inaccurate bounding boxes output from a poor face detector. To address these issues, we can stack or cascade multiple networks to form a coarse-to-fine structure. In fact, this technique has been widely used in the community. For example, Huang et. al.   Huang et al. (2015) proposed to use a global network to obtain coarse facial landmarks for transforming a face to the canonical view and then applies multiple networks trained on different facial parts for landmark refinement. Similarly, both  Yang et al. (2017a) and Deng et al. (2019b) proposed to train a network that predicts a small number of facial landmarks (5 or 19) to transform the face to a canonical view. Because the first network can be trained on a large-scale dataset, such as CelebA (Liu et al. 2015) and UMDFacs (Bansal et al. 2017), it performs well for unconstrained faces with in-plane head rotation, scale and translation. With the normalised faces from the first stage, the performance of subsequent networks trained on a small dataset with all the facial landmarks is boosted. However, there are two outstanding issues in the use of a multi-stage network. First, one should question its effectiveness. Does a multi-stage network perform better than a single large-capacity network that has more parameters? The second important issue is whether stacking multiple networks would slow down the speed of the network. In other words, how can a multi-stage network be used in the most efficient way?

In this section, we answer these two questions using a coarse-to-fine network as depicted in Fig. 8. Given a fixed neural network architecture, the network trained on a dataset exhibiting wide diversity usually has a better generalisation capacity but achieves lower accuracy. In contrast, the network trained on a dataset with less diversity usually performs better for the cohorts involved in the training but is not able to generalise well for untrained cohorts. To achieve good performance in terms of both generalisation capability and accuracy, we need a large-capacity model and a large-scale dataset with a large number of labelled training samples. However, the collection of such a face dataset with manually annotated facial landmarks is very expensive and tedious. An alternative is to train multiple stacked networks, e.g. the proposed coarse-to-fine localisation network.

The coarse network is trained on a dataset with very heavy data augmentation by randomly rotating an original training image between \([-180^{\circ },180^{\circ }]\) and perturbing the bounding box with \(20\%\) of the original bounding box size. Such a trained network is able to perform well for large in-plane head rotations as well as low-quality face bounding boxes. For the second network training, we feed each heavily augmented training sample to the first trained network and obtain its facial landmarks. Then two anchor points (blue points in Fig. 8) are defined using these landmarks to perform rigid transformation. For AFLW, the mean of four inner eye and eyebrow corners is used as the first anchor point and the landmark on the chin is used as the second anchor point. After that, we inject a light data augmentation by randomly rotating the image between \([-10^{\circ }, 10^{\circ }]\) and perturbing the bounding box with \(10\%\) of the bounding box size. Then, the second network is trained using a dataset with less in-plane rotations and high-quality face bounding boxes hence it is able to perform better in terms of accuracy. The joint use of these two networks in a coarse-to-fine fashion is instrumental in enhancing the generalisation capacity as well as accuracy.

We compare the four single-stage CNN plain networks with our coarse-to-fine networks in Table 4, in terms of NME. We can see that the use of our coarse-to-fine framework improves the accuracy of the original plain network at the expense of doubling the network inference time. The speed of each network is reported in Table 6. In addition, the use of two small networks performs better than a single large-capacity network. For example, the model sizes of CNN-6 and CNN-7 are 40MB and 160MB respectively (Table 1). The size of CNN-7 is four times that of CNN-6. When we stack two CNN-6 networks, the size of CNN-7 is still twice that of CNN-6/6. However, the accuracy obtained by the coarse-to-fine CNN-6/6 is better than the single CNN-7 network. The same conclusion can be drawn from the comparison between CNN-7/7 and CNN-8. Moreover, we do not, in fact, need a large-capacity network for the first stage because we only use it to perform coarse facial landmark localisation. We can use a lightweight network, e.g. CNN-6, for the first stage and then cascade a large-capacity network for landmark refinement. According to Table 4, CNN-6/7 and CNN-6/8 perform as well as CNN-7/7 and CNN-8/8.

In this section, we first introduce the implementation details and experimental settings of the proposed method. Second, we conduct an ablation study of its different components. Last, we compare our method with the state-of-the-art algorithms on four well-known benchmarking datasets, i.e. the Caltech Occluded Faces in the Wild (COFW) dataset (Burgos-Artizzu et al. 2013), the Annotated Facial Landmarks in the Wild (AFLW) dataset (Koestinger et al. 2011), the Wider Facial Landmarks in-the-wild (WFLW) dataset (Wu et al. 2018) and the 300 faces in-the-Wild (300W) dataset (Sagonas et al. 2016).

In this paper, we analysed different loss functions that can be used for the task of regression-based facial landmark localisation. We found that L1 and smooth L1 loss functions perform much better in accuracy than the L2 loss function. Motivated by our analysis of these loss functions, we proposed a new, RWing loss performance measure. The key idea of the RWing loss criterion is to increase the contribution of the samples with small and medium size errors to the training of the regression network. To prove the effectiveness of the proposed RWing loss function, extensive experiments have been conducted using several CNN network architectures. As shown in our experiments, by equipping a lightweight CNN network with the proposed RWing loss, it is able to achieve as good performance as large-capacity networks. Furthermore, a Pose-based Data Balancing (PDB) strategy and a coarse-to-fine landmark localisation framework were advocated to improve the accuracy of CNN-based facial landmark localisation further. We found that the proposed PDB strategy and coarse-to-fine framework can effectively deal with the difficulties posed by large-scale head rotations and small sample size problems, respectively. By evaluating our algorithm on multiple well-known benchmarking datasets, we demonstrated the merits of the proposed approach.