Semi-supervised facial expression recognition using reduced spatial features and Deep Belief Networks

Abstract

A semi-supervised emotion recognition algorithm using reduced features as well as a novel feature selection approach is proposed. The proposed algorithm consists of a cascaded structure where first a feature extraction is applied to the facial images, followed by a feature reduction. A semi-supervised training with all the available labeled and unlabeled data is applied to a Deep Belief Network (DBN). Feature selection is performed to eliminate those features that do not provide information, using a reconstruction error-based ranking. Results show that HOG features of mouth provide the best performance. The performance evaluation has been done between the semi-supervised approach using DBN and other supervised strategies such as Support Vector Machine (SVM) and Convolutional Neural Network (CNN). The results show that the semi-supervised approach has improved efficiency using the information contained in both labeled and unlabeled data. Different databases were used to validate the experiments and the application of Linear Discriminant Analysis (LDA) on the HOG features of mouth gave the highest recognition rate.

Introduction

Amongst the various modes of emotion recognition (ER), the facial expression is one of the conveying forms used for the display of emotions. ER can be applied in various fields like medicine, marketing, entertainment. For example, a medical robot can be designed to continuously monitoring their emotional state [1], [2], or a diagnostic suggestion system for therapists [3]. In Human-Computer Interaction, a system endowed with emotional intelligence can be used to create effective communication with users [4]. In emergency situations, as part of the corresponding situational awareness, real-time decisions can be made from the behavioral patterns of the subjects.

The development of a facial ER system is challenging since the images of the same person with the same facial expression can vary with the lighting conditions, background, and occlusions [5], which precludes homogeneity. Certain emotions have only subtle distinctions which make them harder to analyze and describe. The state-of-the-art approaches in facial ER used feature-based methods [6], [7] and template-based methods [8]. The first ones focus appearance and geometric modelled feature extraction. Template-based methods were less reliable because they are limited to only frontal faces and the accuracy rate changed with variations in pose, scale, and shape. Feature extraction is mostly based on Histogram of Oriented Gradients (HOG) [9] and Local Binary Patterns (LBP) [10]. HOG descriptors were used to encode facial components since it projected the appearance of gradient orientation in an image. Other works use Discrete Wavelet Transform (DWT) for feature extraction and Neural Networks for classification [11]. Dimensionality reduction (DR) techniques in ER include principal components analysis (PCA) [12] and linear discriminant analysis (LDA) [13]. Recently, PCA based facial feature projection has also been used for age progression application [14]. These methods cannot be used to find the nonlinear structure of the data. To overcome this limitation various nonlinear DR algorithms such as kernel PCA [15], locally linear embedding (LLE) [16], isometric feature mapping (Isomap) [17] and T-distributed Stochastic Neighbor Embedding (t-SNE) [18] have been proposed. The sparse representation-based methods for classification (SRC) are also widely used since 2009 [19]. SRC is most effective when there is high separability between the subspaces [20], [21], [22]. But its main disadvantage over classical subspace learning algorithms is that the classification criterion of SRC fails and leads to misclassification when the samples are highly correlated. Deep Neural Networks [23] have gained popularity in the recent years as a choice for supervised learning. The major drawback of supervised learning comes from the fact that most of the data available in general is unlabeled., something particularly evident in the case of human face images. In 2004, Hinton and co-workers Hinton et al. [24] proposed the idea of the Restricted Boltzmann Machines (RBM) and its generalization to Deep Belief Networks (DBN), where unsupervised techniques can model the probabilistic distribution of the data and cluster it [25].

In this paper, a semi-supervised DBN is used to include unlabeled and labeled data to improve the accuracy of the classifier. Semi-supervised learning is comparable to human learning, which involves a small amount of labeled data along with greater amounts of unlabelled observations [26]. To make use of unlabeled data, DBNs are applied to learn the model [27] and the obtained discriminative model is fitted to a labeled dataset by performing Backpropagation (BP).

The major contributions in this paper are two. First, we propose to use semi-supervised learning during feature selection to determine the features that are more explanatory of the human emotions in the available data. The proposed DBN has an input layer taking in the dimensionality-reduced feature vectors corresponding to Histogram Oriented Gradients (HOG) of mouth, HOG of the eye, Wavelet Transform of mouth and Wavelet transform of the eye. Reconstruction error and validation accuracy were used to find the most significant feature vector. Second, a semisupervised learning process is proposed that uses non labelled data to train a DBN. After convergence, we proceed to fine tune the structure with BP and the available labelled data. The data is previously processed by a dimensionality reduction method. The most efficient linear method was LDA and amongst the nonlinear approaches, the best one was Isomap.The proposed semi-supervised framework was evaluated on the CK+, MMI and RAFD databases. The results show that the presented approach have a performance similar or better than those of the state of the art (SVM and CNN) with the additional benefits of using significantly less labelled data and a dramatically reduced training and test computational burdens.