A novel driver emotion recognition system based on deep ensemble classification

In computer vision and artificial intelligence, the facial expression is a significant and promising field of research and one of the primary processing methods for intentions expressed through nonverbal means. When interacting with other people, it is impossible to avoid experiencing emotions [1]. They may or may not be visible to the naked eye. Therefore, trained professionals can detect and recognize any indication [2] before or after it is expressed if they have the appropriate tools at their disposal, regardless of when it occurs [3]. R-CNN and deep learning classifier techniques are used for emotion recognition. Only a few of the topics covered in this study are medical [4, 5], human–machine interfaces [6], urban sound perception [7], and animation [8]. Several fields, including the diagnosis of autism spectrum disorder in children [9] and security [10, 11], are seeing an increase in emotion recognition technology. To recognize emotions, various features such as EEG [9], facial expressions (FE) [5, 12, 13], text [14], and speech [15, 16] are used.

Moreover, due to various factors, including their ease of recognition, FE features are one of the most well-known methods of human recognition. The following are some advantages: (1) they are noticeable and visible; (2) they allow for the quick and easy collection of large face datasets using facial expression features; and (3) they contain a large number of features for emotion recognition [17, 18]. Through deep learning, specifically CNN-based learnable image features [15], it is also possible to compute, learn, and extract good facial expressions [19, 20]. Experts predict that FE will become increasingly significant due to advancements in artificial intelligence technology and the rising demand for applications in the era of big data. To be effective in complex environments, such as those characterized by occlusion, multiple views, and multiple objectives, facial emotion recognition (FER) solutions must be proposed in novel and innovative ways. The most relevant data collected under the most favorable conditions at the time of collection is highly desirable when attempting to accurately classify facial expressions to train a FE classifier [21]. The “golden rule” is a term used to describe this. Traditionally, a DFER system will first preprocess the image that will be used as an input to accomplish this. Face detection is a preprocessing step included in most peer-reviewed papers and is described in detail here. The nose and mouth are the most frequently used facial expression cues, even though numerous human face regions can be used to cue facial expressions. The cheeks, forehead, and eyes are just a few other parts of the face that can cue different types [22] of FE. According to a recent study, a small amount of data is collected from the ears and hairs when detecting FE [23].

Accordingly, since the mouth and eyes can detect more FE than the rest, the computer vision deep learning model should place the most significant emphasis on these parts of the face and ignore the others. A CNN framework for DFER is proposed in this manuscript [24], which is based on the findings of this study and incorporates some of the observations made above. Attentional mechanisms, in particular, are employed to [25] draw attention to the essential features of the face. Attentional convolutional networks can achieve extremely high accuracy rates even when using only a few layers (i.e., no more than 50).

Numerous techniques for extracting features have been utilized in the literature for recognizing emotions in drivers, but these techniques restrict the recognition and extraction of emotions. In addition, most emotion recognition systems rely on handcrafted features (such as grayscale statistics, RGB histogram, RGB statistical, and geometrical features) and traditional machine learning classifiers (e.g., SVM, K-NN, and Naive Bayes). The majority of work on feature extraction in recognition systems employs standard methods such as LPB, HOG, and GLCM. However, handcrafted features are considered less robust due to their non-invariant nature, and they extract too many features, which can negatively affect model training and validation. Some invariant features, such as SURF, SIFT, and ORB, operate on low-level features such as edges and corners. Nevertheless, due to the vast number of images in a dataset, these descriptors may not achieve high accuracy. Image data have various forms of noise, blurriness, oversharpening, and unbalanced contrast, which may impair model training, resulting in low accuracy.

This paper develops a deep learning model based on the improved Faster R-CNN and deep ensemble classifier for emotion recognition, along with the ability to observe the driver’s emotions while driving a vehicle. The Faster R-CNN model has been improved for detecting the driver’s face region, and the learning block of the Faster R-CNN has been replaced with an improved CNN block (InceptionV3), which improves the accuracy and efficiency of face detection. The proposed approach can work in environments where outdated and other approaches are deficient with full potential. The main contributions of this paper are as follows:

The paper is organized as follows: the first section elaborates and explains the introduction of the entire manuscript. The next section explains and highlights the related work with regard to current works from different authors’ perspectives. The third section introduces the methodology and the proposed framework of the research. The fourth section explains and illustrates the experimental results of the overall research. Finally, the fifth section concludes the manuscript.

The six primary emotions: the emotions of pleasure, fear, hate, sorrow, disgust, and surprise (except neutral) are identified in [26]. Ekman utilized this concept to create the facial action coding system (FACS) [27], which became the gold standard for emotion recognition research. Neutral was later added to most human recognition datasets as a seventh fundamental emotion. Figure 1 shows sample pictures of various emotions from the four benchmark datasets (FER-2013, JAFFE, CK+, and AffectNet datasets). The primary emotions are the happy face, angry face, disgusted face, fearful face, sad face, surprised face, and contemptuous face.

A two-step machine learning methodology was employed in initial studies on emotion recognition. In the first step, the image’s attributes are extracted, whereas, in the second step, classifiers are used for emotion detection. Gabor wavelets [28], Haar features [29], Texture features linear binary pattern (LBP) [30], and Edge Histogram Descriptors [31, 32] are some of the most frequently exploited manual features for the detection of FE. The classifier then identifies the image with the most appropriate sentiment. These techniques seem to be effective on more specialized datasets. However, posing significant limitations when applied to challenging datasets (with greater intra-class variation). To help the reader better understand some of the problems that images can provide, Fig. 1 of the first row included an image that only showed the reader’s eyes or the covered hand or portions of the face.

Numerous firms have made significant advancements in neural networks, deep learning, picture categorization, and vision challenges. In [33], Khorrami demonstrated that CNN could achieve a better accuracy level for emotion recognition. Moreover, zero-bias CNN’s Toronto Face Datasets (TFD) and Cohn–Kanade dataset (CK+) for attaining state-of-the-art results when applied to model human facial expressions. To construct a model for FE of stylized animation characters, the authors in [34] trained a network using deep learning and translated human images to animated faces. A FER neural network A network with a top layer of pooling, two convolution layers, and four initial layers or subnetworks has been proposed by Mollahosseini [35]. The authors in [36] integrate the removal and classification of features using a single recurrent network, highlighting the importance of input from both components. To achieve cutting-edge CK+ and JAFFE accuracy, the BDBN Network was utilized.

The authors in [37] implemented a deep CNN on noisy labeling of authentic images acquired through crowdsourcing. They deployed ten taggers to re-enact each image to acquire the required precision, with ten tags in their dataset and numerous costing functions for their Deep Convolutional neural network (DCNN). To improve the spontaneous recognition of facial experiences, authors in [38] used more discriminative neurons that outperformed Incremental Boosting CNN (IB-CNN). The authors of [39] An identity-aware CNN (IA-CNN) created that uses identity- and expression-sensitive contrast loss to reduce variation of expression-related information during identity learning. Similarly, they have developed a network architecture with a focal model called end-to-end network architecture [40]. To minimize uncertainty and avoid unclear face images (caused by labeling noise) from overfitting the deeper network, the authors in [41] devised a quick and efficient self-repair technique (SCN). SCN reduces uncertainty in two dimensions and ways: (1) using a self-attention mechanism to weight each sample of workouts in small batches with rank regularization; and (2) by carefully changing these samples in the lowest rank set. It was identified using an algorithm. In the real world, it is employed for occlusion changes and posture resistance [42]. They created a new network dubbed the regional attention network to adequately represent the significance of position variant FER and face areas in occlusion (RAN). Deep learning attention networks for the recognition of facial emotions [43], multi-attention networks for the recognition of facial expressions [44], and a new review on emotion recognition using facial appearance [45] are some of the related works for the recognition of FE. All the works mentioned [46] above have improved emotion recognition significantly over previous work. Still, none of these works contains a simple method for identifying essential face regions to detect emotions. This study [47] suggests a new framework based on a further attentiveness-coevolutionary neural network to focus on critical facial areas [48, 49].

The authors of [50] proposed ENCASE to combine expert features and DNN (Deep Neural Networks) for electrocardiogram (ECG) classification. They investigated specific applications and applied them to statistics, signal processing, and medicine. Then, they developed DNN for automatic deep feature extraction [51]. They also proposed a new algorithm to find the most representative wave (called the central wave) in long ECG recordings and extract the features of the central wave. Finally, they combined these features and subjected them to ensemble classification. The authors of [52, 53] proposed a new remote sensing scene classification approach using a set of robust and independently trained Deep Rule-Based (DRB) classifiers with different spatial information levels. Each DRB classifier is a comprehensive parallel set of transparent and human-interpretable zero-order fuzzy rules with a prototypical nature. The DRB classifier can be organized “from scratch” and built upon its structure. Using pre-trained neural networks as feature descriptors, the proposed DRB ensemble can show human-level performance through a parallel and transparent training process [54, 55]. Numerical examples on reference datasets show that the proposed method is more accurate when DRB classification creates fuzzy rules for human understanding [56‐58]. According to the aforementioned literature review, numerous researchers are working to improve the performance of current CNN models on real-time facial expression and DFE datasets, which contain both real-world and lab-trained images. As a result, a powerful model with deep learning fusion techniques is required to accurately classify driver emotions and extract key image features. As a result, in our research, the author recommends the CNN-based DenseNet model. The most recent performance analysis based on recent research on multi-feature fusion-based methods using deep learning techniques is shown in Table 1.

The proposed ensemble learning framework involves several levels of merging different classifiers trained on different feature sets. Figure 2 illustrates the entire process of the proposed framework. Specifically, as shown in the feature preparation layer in Fig. 2, various feature sets are prepared using various feature extraction techniques. Furthermore, the feature set obtained by applying a specific method is further processed by applying various feature selection methods to obtain various feature subsets. Three state-of-the-art CNN models, DenseNet, InceptionV3, and Resnet-50 are used to extract high-dimensional features from the train and validation set images. The extracted fused features are concatenated to create a fused features vector. An ensemble classifier is developed using three state-of-the-art deep learning classifiers, CNN, gate recurrent unit (GRU), and multi-layer perception (MLP). The final output is selected by a voting scheme and assigned a class label predicted by most of the classifiers. In a real-world scenario, ensemble learning can be implemented in a more flexible configuration than that described in Fig. 2. For example, multiple base classifiers trained on the same feature set can be combined into a primary integration, then combined with the remaining base classifiers to form a quadratic integration. In this case, a secondary integration can be created from each feature set, and then some, all, or a secondary integration can be combined to create a top-level or even a final integration. The proposed framework can be viewed as a philosophical strategy for structural thinking and can also be used to solve the problem of driver emotion recognition. For example, multiple base classifiers trained on the same feature set can be combined into a primary integration, then combined with the remaining base classifiers to form a quadratic integration. In this case, a secondary integration can be created from each feature set, and then some, all, or a secondary integration can be combined to create a top-level or even a final integration. Granular computing can be viewed as a philosophical strategy for structural thinking, but it can also be used to solve the problem of driver emotion recognition. Each integration can be conceptualized as a single model in an ensemble learning framework because it contains multiple classifiers. Images can be of different sizes, which led to the development of the concept of particle size. Image size assists in improving the proposed model performance as it changes the proportion corresponding to the model size. The length and width of the images and models can vary considerably. Each level of classifier fusion can be interpreted as a different level of granularity in the proposed input learning framework.

Three models are used in this stage: GRU, MLP, and CNN. MLP is frequently used to solve problems requiring supervised learning and research into computational neuroscience and parallel distributed processing. The GRU is a type of RNN that uses less memory than long short-term memory and considers more efficient. However, using datasets with longer sequences improves the accuracy of LSTM. GRU occasionally has advantages over LSTM that are greater. These models ran on the data from the task of feature fusion and feature concatenation. Every model also has a distinct architecture where various tasks are carried out. The next step is to predict the data in tabular form so that a confusion matrix can be effectively generated after these models have been fully processed. The final stage will involve performing cure graphs in a simulation environment based on the confusion matrix.

In this stage, three models are used: Gate Recurrent Unit, Multi-Layer Perception (MLP), and CNN. MLP is commonly used to solve problems that require supervised learning and research in computational neuroscience and parallel distributed processing. The GRU is a type of RNN that uses less memory than LSTM and is considered more efficient. However, using datasets with longer sequences improves the accuracy of LSTM. GRU occasionally has advantages over LSTM that are greater. These models ran on the data from the task of feature fusion and feature concatenation. Each model also has a distinct architecture where various tasks are carried out. The next step is to predict the data in tabular form so that a confusion matrix can be effectively generated after these models have been fully processed. The final stage will involve performing cure graphs in a simulation environment based on the confusion matrix.

Several benchmark datasets, including AffectNet, CK+, FER-2013, JAFFE, and custom-developed datasets, are used to train the proposed ensemble CNN classifier. It should be noted that we trained a separate model for each dataset used in this study. A separate validation and test set are used for model parameter tuning and performance evaluation. A confusion matrix is an important metric consisting of various measures such as true positive, false positive, false negative, and true negative. These measures are used for validating model performance using the accuracy, precision, recall, and f1-score metrics.

To proposed driver facial expression recognition (DFER) method has been effectively tested and proven on various standard datasets for the development of this section. In this study, we compared the current FER method with the state-of-the-art method, and the data collected as also encompassed the result of the qualitative and quantitative evaluation. The proposed system used two reference datasets. Every dataset is divided into two sections such as training, and validation sets randomly. The training sets will be 70% of the datasets and 30% will be for validation sets of each dataset. All simulations are performed under the simulation environment using the MATLAB R2021a platform contained in the proposal. All of this is done on a workstation shown in Table 2.

In this paper, a new framework based on CNN is proposed to recognize the emotional state of the driver. We believe that FE can be detected by focusing on specific facial regions. We have also conducted experimental research, utilizing four different expression datasets and a custom-made dataset for DFER, which yielded promising results. In addition, we employed a visualization technique to highlight the most crucial areas of face images to recognize various drivers' facial emotions. The model identifies the driver’s emotions using images of the driver’s face with the help of a deep learning algorithm. According to the proposed DFER model, a DFE state can be identified without additional effort from the driver. A custom CNN block replaces the Improved Faster R-CNN features learning block to improve the accuracy and efficiency of face detection. Transfer learning is implemented in DenseNet CNN’s model by substituting custom driver emotion datasets for ImageNet data. The CNN model consists of 201 layers and is used to recognize facial expressions. The CDD contains seven basic driver emotions, and the proposed face detection and DFER models are evaluated using the benchmark dataset. The effectiveness of the DenseNet model has been evaluated by achieving high accuracy. Multiple facial expression recognition datasets were utilized to evaluate the proposed model, including JAFFE, CK+, FER-2013, AffectNet, and custom-made datasets. The proposed model outperformed several advanced facial expression recognition models using a benchmark dataset. The DenseNet model is efficient and precise and can be implemented by various hardware devices to recognize the driver’s emotions and improve the performance of automatic driver assistance systems.

Putting aside what we have successfully achieved, several useful extensions can be addressed for further improvements. Without considering the influence of head pose variations, only the frontal faces of drivers are taken for training and implementation purposes. Therefore, further faces from several views can be considered from the images or videos which may help to improve the recognition accuracy. The deep learning techniques lack sufficient data to be the most effective they can. Therefore, it may be useful to pre-train a deep CNN on many other datasets before applying a fine-tuning process. A hybrid method can be developed in the future by combining geometric features and appearance-based features to improve the performance of the DER.