Facial expression recognition through multi-level features extraction and fusion

Recent studies have shown that deep learning has presented great potential in facial expression recognition (FER) tasks and attracted more and more researchers’ attention. Many existing methods have achieved good results on facial expression images in the laboratory environment. However, there are still great challenges for FER in the wild environment where facial expression images are more complex and diverse than those in the laboratory. In this paper, we propose a new method for FER from the perspective of multi-level features extraction and fusion. Different from the existing feature extraction network where only a single convolution kernel scale is present, we propose a feature extraction module with different convolutional kernel scales, which extracts multi-level features as the output of the whole feature extraction network. Further, we do not directly use these multi-level features but propose a feature fusion module with global and local attention to adaptively fuse these different level features in pairs with a top-down way and construct a new facial expression feature. To relieve the overfitting effect caused by data imbalance, we employ label smoothing and L2 regularization strategies to further guide our model forward in a better direction. Through extensive experiments, we demonstrate our method achieves accuracies of 88.08% on RAFDB, 88.11% on FERPlus and 59.38% on AffectNet, respectively, which are very competitive performances. Moreover, our multi-level feature fusion approach enables traditional convolutional backbone networks to improve performance by 0.64–1.13% on FER tasks.