Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order

Highlights

• A CNN based approach for facial expression recognition.

• A set of pre-processing steps allowing for a simpler CNN architecture.

• A study of the impact of each pre-processing step in the accuracy.

• A study for lowering the impact of the sample presentation order during training.

• High facial expression recognition accuracy (96.76%) with real time evaluation.

Abstract

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine learning methods, since people can vary significantly in the way they show their expressions. Even images of the same person in the same facial expression can vary in brightness, background and pose, and these variations are emphasized if considering different subjects (because of variations in shape, ethnicity among others). Although facial expression recognition is very studied in the literature, few works perform fair evaluation avoiding mixing subjects while training and testing the proposed algorithms. Hence, facial expression recognition is still a challenging problem in computer vision. In this work, we propose a simple solution for facial expression recognition that uses a combination of Convolutional Neural Network and specific image pre-processing steps. Convolutional Neural Networks achieve better accuracy with big data. However, there are no publicly available datasets with sufficient data for facial expression recognition with deep architectures. Therefore, to tackle the problem, we apply some pre-processing techniques to extract only expression specific features from a face image and explore the presentation order of the samples during training. The experiments employed to evaluate our technique were carried out using three largely used public databases (CK+, JAFFE and BU-3DFE). A study of the impact of each image pre-processing operation in the accuracy rate is presented. The proposed method: achieves competitive results when compared with other facial expression recognition methods – 96.76% of accuracy in the CK+ database – it is fast to train, and it allows for real time facial expression recognition with standard computers.

Introduction

Facial expression is one of the most important features of human emotion recognition [1]. It was introduced as a research field by Darwin in his book “The Expression of the Emotions in Man and Animals' [2]. According to Li and Jain [3], it can be defined as the facial changes in response to a person's internal emotional state, intentions, or social communication. Nowadays, automated facial expression recognition has a large variety of applications, such as data-driven animation, neuromarketing, interactive games, sociable robotics and many other human–computer interaction systems.

Expression recognition is a task that humans perform daily and effortlessly [3], but it is not yet easily performed by computers, despite recent methods have presented with accuracies larger than 95% in some conditions (frontal face, controlled environments, and high-resolution images). Many works in the literature do not perform a consistent evaluation methodology (e.g. without subject overlap in training and testing) and therefore present a misleading high-accuracy, but do not represent most of the face expression recognition problems real scenarios. On the other hand, low accuracy has been reported on databases with uncontrolled environments and in cross-database evaluations. Trying to cope with these limitations, several research works have tried to make computers reach the same accuracy of humans, and some examples of these works are highlighted below. This problem is still a challenge for computers because it is very hard to separate the expressions’ feature space, i.e. facial features from one subject in two different expressions may be very close in the feature space, while facial features from two subjects with the same expression may be very far from each other. In addition, some expressions like “sad” and “fear”, for example, are, in some cases, very similar.

Fig. 1 shows three subjects with a happy expression. As it can be seen in the figure, the images vary a lot from each other not only in the way that the subjects show their expression, but also in lighting, brightness, pose and background. This figure also exemplifies another challenge related to the facial expression recognition that is the uncontrolled training–testing scenarios (training images can be very different in terms of environmental conditions and subject ethnicity from the testing images). One approach to evaluate the facial expression recognition under these scenarios is to train the method with one database and to test it with another (possibly from different ethnic groups). We present results following this approach.

Facial expression recognition systems can be divided into two main categories: those that work with static images [7], [8], [9], [10], [11], [12], [13] and those that work with dynamic image sequences [14], [15], [16], [17]. Static-based methods do not use temporal information, i.e. the feature vector comprises information about the current input image only. Sequence based methods, in the other hand, use temporal information of images to recognize the expression captured from one or more frames. Automated systems for facial expression recognition receive the expected input (static image or image sequence) and typically give as output one of six basic expressions (anger, sad, surprise, happy, disgust and fear, for example); some systems also recognize the neutral expression. This work will focus on methods based on static images and it will consider the six and seven expressions sets (six basic plus neutral), for controlled and uncontrolled scenarios.

As described by Li and Jain [3], automatic facial expression analysis comprises three steps: face acquisition, facial data extraction and representation, and facial expression recognition. Face acquisition can be split in two major steps: face detection [18], [19], [20], [21] and head pose estimation [22], [23], [24]. After the face acquisition, the facial changes caused by facial expressions need to be extracted. These changes are usually extracted using geometric feature-based methods [25], [26], [27], [21] or appearance-based methods [8], [9], [10], [11], [13], [25], [28]. The extracted features are often represented in vectors, referred as feature vectors. Geometric feature-based methods work with shape and location of facial components like mouth, eyes, nose and eyebrows. The feature vector that represents the face geometry is composed of facial components or facial feature points. Appearance-based methods work with feature vectors extracted from the whole face, or from specific regions; these feature vectors are acquired using image filters applied to the whole face image [3].

Once feature vectors related to the facial expression are available, expression recognition can be performed. According to Liu et al. [7], expression recognition systems basically use a three-stage training procedure: feature learning, feature selection and classifier construction, in this order. The feature learning stage is responsible for the extraction of all features related to the facial expression. The feature selection selects the best features to represent the facial expression. They should minimize the intra-class variation of expressions while maximizing the inter-class variation [8]. Minimizing the intra-class variation of expressions is a problem because images of different individuals with the same expression are far from each other in the pixel's space. Maximizing the inter-class variation is also difficult because images of the same person in different expressions may be very close to one another in the pixel's space [29]. At the end of the whole process, a classifier (or a set of classifiers, with one for each expression) is used to infer the facial expression, given the selected features.

One of the techniques that has been successfully applied to the facial expression recognition problem was the deep multi-layer neural network [30], [31], [32], [14], [10], [11], [13]. This technique comprises the three steps of facial expression recognition (learning and selection of features and classification) in one single step. In the last decade, neural network researches were motivated to find a way to train deep multi-layer neural networks (i.e. networks with more than one or two hidden layers) in order to increase their accuracy [33], [34]. According to Bengio [35], until 2006, many new attempts have shown little success. Although somewhat old, the Convolutional Neural Networks (CNNs) proposed in 1998 by Lecun et al. [36] has shown to be very effective in learning features with a high level of abstraction when using deeper architectures (i.e. with a lot of layers) and new training techniques. In general, this type of hierarchical network has alternating types of layers, including convolutional layers, sub-sampling layers and fully connected layers. Convolutional layers are characterized by the kernel's size and the number of generated maps. The kernel is shifted over the valid region of the input image generating one map. Sub-sampling layers are used to increase the position invariance of the kernels by reducing the map size [37]. The main types of sub-sampling layers are maximum-pooling and average pooling [37]. Fully connected layers on CNN's are similar to the ones in general neural networks, its neurons are fully connected with the previous layer (generally: convolution layer, sub-sampling layer or even a fully connected layer). The learning procedure of CNNs consists of finding the best synapses’ weights. Supervised learning can be performed using a gradient descent method, like the one proposed by Lecun et al. [36]. One of the main advantages of CNN, is that the models' input is a raw image rather than a set of hand-coded features.

Besides the methods using deep architecture, there are many others in the literature, but some aspects of the evaluation of these methods still deserve attention. For example, validation methods could be improved in [10], [30], [38], [39], [40], [41], [42] in order to consider situations where the subject in the test set is not in the training set (i.e. test without subject overlap), accuracy is somewhat low in [38], [1], [43], [44], and the recognition time in [7], [43], [16] could be improved so as to perform real time evaluations.

Trying to cope with some of these limitations while keeping a simple solution, in this paper, we present a deep learning approach combining standard methods, like image normalizations, synthetic training-samples generation (for example, real images with artificial rotations, translation and scaling) and Convolutional Neural Network, into a simple solution that is able to achieve a very high accuracy rate of 96.76% in the CK+ database for 6 expressions, which is the state-of -the-art. The training time is significantly smaller if compared with other methods in the literature and the whole facial expression recognition system can operate in real time in standard computers. We have examined the performance of our system using the Extensive Cohn–Kanade (CK+) database [4], the Japanese Female Facial Expression (JAFFE) database [5] and the Binghamton University 3D Facial Expression (BU-3DFE) database [6], achieving a better accuracy in the CK+ database, which contains more samples (important for deep learning techniques) than the JAFFE and BU-3DFE databases. In addition, we have performed an extensive validation, with cross-database tests (i.e. training the method using one database and evaluating its accuracy using another one). In summary, the main contributions of this work are:

• an efficient method for facial expression recognition that operates in real time;

• a study of the effects of image pre-processing operations in the facial expression recognition problem;

• a set of pre-processing operations for face normalization (spatial and intensity) in order to decrease the need of controlled environments and to cope with the lack of data;

• a study to handle the variability in the accuracy caused by the presentation order of the samples during training; and

• a study of the performance of the proposed system with different cultures and environments (cross-database evaluation).

This work extends the one presented in the 28th SIBGRAPI (Conference on Graphics, Patterns and Images) [45] as follows:

• it presents a deeper literature review which were used to enlarge the comparisons of the results;

• it presents the results using a new implementation based on a different framework (from ConvNet, a Matlab based implementation [46], to Caffe, a C++ based implementation [47]), which consequently reduced the total training time by almost a factor of four;

• it presents results showing a reduced recognition time, which is now real time;

• it presents results showing better accuracy due to longer training and small changes (see below) in the method;

• it includes improved experimental methodology, as for example, using training, validation and test sets, instead of training and test sets only; and

• it presents a more complete evaluation including cross-database tests.

The changes in the method that allowed better accuracy were as follows:

• The synthetic samples have a slight different generation process, allowing larger variation among them (now synthetic samples can be the original image rotated, scaled or translated, instead of only rotated);

• We increased the number of synthetic samples, from 30 to 70 (motivated by the previous item); and

• The logistic regression loss function was replaced by a SoftmaxWithLoss function (described in Section 3).

The remainder of this paper is organized as follows: the next section presents the most recent related work, while Section 3 describes the proposed approach. In Section 4, the experiments we have performed to evaluate our system are presented and compared with several recent facial expression recognition methods. Finally, we conclude in Section 5.