A descriptive human visual cognitive strategy using graph neural network for facial expression recognition

Humans perceive the world through his organs of perception. Under normal circumstances, people form the cognition of external environment information through the five organs of perception: sight, hearing, smell, touch, and taste. However, among the five organs of human cognition, vision is the main channel through which humans detect and receive information. Research shows that of the five senses, about 70% of information is detected through vision. Thus, the visual cognitive system is the most commonly used sensory organ in humans to contact external information. It is capable of rationally analysing, associating, interpreting, and comprehending external phenomena and distinguishing the contours of information in the visual process [9, 10].

Vision is not only a psychological and physical perception, but also the source of creativity. Experience comes from understanding and analyzing the environment. And all human behavior begins with cognition, so without cognition, there is nothing. In visual cognition, human beings have certain characteristics. In general, the visual image is first perceived as a unified whole and then in terms of parts. That is, humans first “see” the whole of a composition and then the parts of the whole composition. Therefore, vision is the most important way for humans to perceive the objective world.

Given the increasingly complex evolution of information visualization and understanding, Green et al. [11] proposed a design guide for visual interface development by discussing the human cognitive modeling framework to optimize human–computer interaction. Based on the recent biological findings, Hong et al. [12] proposed a framework to imitate the active and dynamic learning and recognition process of primate visual cortex. Scott et al. [13] proposed a general cognitive architecture that closely combines symbol, spatial, and visual representations to realize the free movement of the cognition between feature patterns. James et al. [14] investigated the spatial, dynamic, and molecular properties of neural activity in a range of within-system cognitive tasks and found that neural activity converges to a low-dimensional manifold. This research furthered the understanding of cognitive brain tissue. Hong et al. [15] improved the theoretical framework of visual cognition by imitating the finer structure and function of primate visual cortex to realize position and scale recognition. Hideho et al. [16] investigated the relationship between difficulties in visual cognition and the eye movements of the target, and proposed a gaze movement system to estimate human cognition, which is widely used in the cognitive process of motor targets.

Our method realizes two key processes of “local visual cognition” and “regional cooperative recognition” by using human visual cognitive strategies. The face is divided into six parts to learn the features. Finally, the six parts are considered as a whole to judge the facial expression. It simulates the human learning process from part to whole.

In computer science, the graph is a data structure composed of two parts: vertices and edges. The graph \(G\) can be described by the vertex set \(V\) and the contained edges \(E\).\(V\) is a non-empty finite set of vertices, and \(E\) is the set of edges between any two points in the vertex set \(V\).

In modern terms, the representation of a graph generally uses circles to represent vertices, and line segments to represent edges, and one edge connects two different vertices. If there are no directional edges, edges are called undirected edges, and the graph is called an undirected graph, as shown in Fig. 1a. If some edges have directions, these edges are called directed edges, and the graph is called a directed graph, as shown in Fig. 1b.

The size of the disordered nodes in the graph is variable, and each node has a different number of neighbour nodes. Therefore, GNN is used in social networks, knowledge graphs, recommender systems, and even in life sciences [17‐19]. For instance, Gori et al. [20] first proposed the concept of GNN and used it to process graph data structures. Dalibor et al. [21] extended the graph matching algorithm with a new method combining fuzzy logic and recurrent neural networks. This method was more adaptable to input noise than conventional neural networks based on nonfuzzy supervised learning. Shen et al. [22] proposed a novel similarity guided graph neural network to overcome the relationship information between different probe libraries. Marco et al. [23] combined the coding module of the recurrent neural network and the radial basis function network to estimate the unknown probability density function from unsupervised samples for graph classification and graph clustering, based on the graph structured data. Rusek et al. [24] proposed a novel GNN based network model that can understand the complex relationship between topology, routing and input traffic to accurately estimate the delay distribution and loss of each data packet in each source/target.

Our method completes facial expression recognition through GNN modelling of the face, and considers the relationship between different regions of the face through undirected graph, which improves the representation and generalization ability of the model and is interpretable.

Facial expressions are the result of one or more actions or states of the facial muscles. These states of movement fully express the person's emotions to the observer. Facial expressions are a form of nonverbal communication. It is the primary means of expressing social information between humans, and it also occurs in most other mammals and some other animal species. However, it is a difficult task to automatically recognize facial expressions from facial images [25‐28].

The traditional FER first uses mathematical methods [29] (FastICA method, Gabor wavelet extraction method and so on) to extract facial expression features, such as keypoint information of facial features, facial texture features, facial histograms and other features. Then, classifiers (Naive Bayes classifier, Support Vector Machines, etc.) are used to classify the extracted information features and complete FER. For example, Ira et al. [30] used a naive Bayes classifier in the traditional FER to convert the Gaussian distribution of facial expression features into a Cauchy distribution, and used Gaussian to improve the naive Bayes classifier, which performs FER by learning the dependencies between the different facial expression features. Shan et al. [31] proposed a novel low-computation discriminative feature space based on a simple local binary pattern to represent the expression state of a face image. With this method, the features could not only be extracted faster, but also the face information could be stored effectively in a low-dimensional space. With the perfection of deep learning theory in recent years, deep learning technology has shown strong extraction ability in processing European-style image data, which has also promoted the rapid development of face recognition.

Unlike the traditional methods of FER, deep learning methods have a strong ability to extract features from images, especially the Convolutional Neural Network (CNN). The CNN is a convolutional kernel obtained by learning the feature types of facial expressions on the image, which naturally takes precedence over the features and classifiers of traditional methods. In these deep learning FER methods, multiple convolutional layers are first used to extract a large amount of key information about the facial expression features, and then a fully concatenated layer is used to perform nonlinear classification to realize FER. For example, Ali et al. [32] proposed a deep neural network architecture to address the shortcomings of inadequate representation of facial expressions by traditional features. Zeng et al. [33] developed a deep sparse autoencoder, introduced high-dimensional features, and proposed a novel FER framework to automatically distinguish facial expressions with high precision.

In the study of facial feature classification, Kamil et al. [34] proposed an entropy-based feature selection method applied to the 3D distance of facial features to realize FER by using the 3D distance between 83 points of the face as facial features. Khan et al. [35] proposed a new measurement for copolymer feature selection based on singular value decomposition to find the most salient areas in facial expressions to achieve accurate recognition. For FER, most methods use the entire image as the extraction region, ignoring the extraction of key areas of facial expressions. To solve this problem, He et al. [36] proposed a facial expression recognition method based on local binary patterns to fuse the key facial expression features. This method divides the facial expression into a part of the eyes, a part of the eyebrows, a part of the nose, and a part of the mouth. Then the feature information of these parts is obtained and combined to form a new feature. Then support vector machine and NN are used to classify the combined information for facial expression recognition. Shojaeilangari et al. [37] proposed an extreme sparse learning method that combines the discrimination ability of an extreme learning machine with the reconstruction of a sparse representation, which can achieve accurate FER in noisy signal areas and natural environments.

To solve the difficult feature extraction problem, Li et al. [38] proposed a unified probability concept based on dynamic Bayesian networks to represent changes in facial expression at the same level to detect the state of facial expression. Mohan et al. [39] designed a new type of deep CNN to study the geometric features of human faces, such as edges, curves, and straight lines. Then, a fractional fusion technology was used for facial expression recognition. Lee et al. [40] proposed a multimodal recurrent attention network integrating multimodal face information. This network could not only learn spatiotemporal attention information to add feature sets, but also use depth and popularity sequences as a priori advice for color sequence. And the emotion regions are selected to receive attention to achieve accurate recognition. To solve the problem caused by personal attributes and achieve better FER performance, Meng et al. [41] proposed a novel identity recognition CNN that can not only ensure that the features learned by the network remain unchanged under expression changes, but also that the identities remain unchanged. Andre et al. [42] proposed a combination of CNN and specific image preprocessing steps to solve the problem of FER. The network used the preprocessing technology to extract a large amount of feature data, and then used the deep structure to explore the presentation order of the facial expression feature data for FER. Sun et al. [43] proposed a multi-channel deep neural network to extract the optical flow from the changes between facial images with high expression and neutral facial images as the time information of facial expression. Then it used the horizontal images of gray emotional faces as spatial information to achieve FER.

The facial expressions of the human face are generally divided into expressions such as anger, disgust, fear, happiness, sadness, surprise, and natural. However, the change of each expression is reflected in multiple areas of the human face, such as eyebrows, eyes, nose, and mouth. Different facial expressions have different forms of expression in each area of the face. Therefore, the existing deep learning FER methods first perform preprocessing operations after inputting a face image or video. In the preprocessing process, the depth detection algorithm is used to divide the facial features \(Q\) into eyebrows \({Q}_{1}\), eyes \({Q}_{2}\), nose \({Q}_{3}\), and mouth \({Q}_{4}\), and then extract the key feature points of each part for information fusion processing, as given in Eqs. (2) and (3):

In Eq. (2), \(\left(a,b\right)\) represents the coordinates of key feature points; \(m\) represents the total number of features in each part; \(k\) represents the type of facial expressions; \(t\) represents the number of divided regions of the face; \(i\) represents the current \(i\)-th part area; and \(j\) represents the \(j\)-th feature key point in the \(i\)-th part.

In Eq. (3), \(\Phi\) represents the feature fusion function and \(G\) is the result after feature fusion.

By using this method to train the deep learning model to achieve the effect of recognizing facial expressions, a schematic diagram of the existing deep learning framework for FER is shown in Fig. 3.

It can be seen from Fig. 3 that although the deep learning detection algorithm can accurately divide the various parts of the face \({Q}_{1},{Q}_{2},\cdots ,{Q}_{t}\), and extracts the expression feature information \(\left({a}_{ij},{b}_{ij}\right)\) between the various parts. But this method only works in each independent part, does not pay attention to the connotative connection between the parts of \({Q}_{1},{Q}_{2},\cdots ,{Q}_{t}\), and ignores the internal connection between blocks. In other words, this method does not fully combine the implicit relationships between features to further realize FER.

Moreover, the key information of facial expression features extracted by this method is not all credible, which may cause the generalization ability of existing deep learning FER algorithms to weaken as shown in Fig. 4.

In Fig. 4a, the green dots represent the coordinates of the key feature points extracted by the detection algorithm, and the red dots represent the true coordinates of the key feature points. Figure 4b shows the heat map of the key feature points extracted by the detection algorithm. From Fig. 4a, it is clear that some green dots are farther from the red dots, and the deviation is large. From Fig. 4b, it can also be seen that some green dots have larger heat maps. All this shows that the key feature points extracted by traditional deep learning detection algorithms are not necessarily credible. If the model is constructed in this way (selecting all the key feature points), the accuracy of FER will be greatly affected. Moreover, when the model is used for FER, its features and recognition results are not well interpretable.

Based on some of the above drawbacks. We use the region division mechanism, and use the two key processes of local visual cognition and regional collaborative recognition to link the features of different regions, find the connotation relationship between originally independent parts, and enhance the generalization ability of the model. At the same time, we reduce some unreliable feature points through local uniform selection in the process of local visual cognition, and improve the accuracy of the model.

Visual cognition refers to the process by which people obtain knowledge or information through the visual system. Visual cognition is the foundation of human thought. When the human visual system is stimulated by the external environment, it first focuses its attention on the external information and performs local cognition by observing the salient features or key feature information points of the local information; then it performs regional collaborative association with the salient features of the observed information to identify the current information content. The process described above is the process of human visual cognition.

Following the logical process of visual cognition in the human brain, this paper divides the process of FER into two main stages: “local visual cognition” and “regional collaborative recognition”:

In the case of image convolution, the sampling function \(P:{G}^{2}\times {G}^{2}\) can also be expressed by Eq. (11):\(w\):\({G}^{2}\to {R}^{C}\) represents the weight function. To realize the “regional cooperative recognition”, this paper redefines the weight function \(w\). First, the distance d between the key points of the feature is calculated, which can be obtained by Eq. (12):

In Eq. (12), \(\left(x,y\right)\) represents the coordinates of the feature key point; and \(t\) represents the selected \(t\)-th feature key point.

Through centralized processing, calculate the distance between each feature key point and other feature key points and then add them to get \({D}_{{(x}_{t},{y}_{t})}\). It can be expressed as formula (13):

The smaller the value of \({D}_{{(x}_{i},{y}_{i})}\), the closer the i-th cognitive key point is from the center. Therefore, starting from the key point with the smallest value of \({D}_{{(x}_{i},{y}_{i})}\), each cognitive feature key point is sorted in turn. Then determine a domain \(B\left({N}_{i}\right)\) for each selected cognitive feature key point \({N}_{i}\), where the mapping relationship can be expressed as \({f}_{i}:{N}_{i}\to B({N}_{i})\). Because there are Z labeled subsets in each field, we can get a mapping function \({f}_{i}:B({N}_{i})\to \left\{0,\cdots ,Z-1\right\}\), and then map the neighborhood node to its corresponding subset. The weight function \(w\left({N}_{i},{N}_{j}\right):B({N}_{i})\to {R}^{C}\) of the key points of cognitive features \({N}_{i}\) and \({N}_{j}\) can be obtained by Eq. (14):

The schematic diagram of the “regional collaborative recognition” process is shown in Fig. 6 as follows:

Based on the aforementioned human cognitive vision strategy, this paper constructs an undirected graph \(G=\left(V,E\right)\) for each face image through the two processes of “local visual recognition” and “regional collaborative recognition”. Each undirected graph has \(n\) cognitive nodes (\(V=N=\left\{\mathrm{1,2},\cdots ,n\right\}\)) and \(E(E\subseteq V\times V)\) edges. Generally, the representation of an undirected graph depends on the adjacency matrix \(W\in {R}^{\left|V\right|*\left|V\right|}\), and each element in \(W\) represents the weight of an edge, which is the connection strength between a pair of key feature nodes.

In order to address the shortcomings of the existing deep learning algorithms for FER, this work needs to adopt the key feature selection method based on the local visual recognition process and construct the edge of the regional collaborative recognition process to construct the undirected GNN, which is a more effective FER model. The model mainly implements two key processes (local visual recognition process and regional collaborative recognition process). First, the credibility between the key points of the facial features obtained by the detection algorithm and the corresponding key points of the actual features is calculated. Then, based on the credibility, the cognitive keypoint information of each face part (upper-left, upper-right, upper-middle-left, upper-middle-right, middle and lower) is selected, and the total distance of each cognitive keypoint to other cognitive keypoints for sorting is calculated; Finally, the weight function as the edge of the graph is constructed by the idea of 2D convolution operation to build the GNN model of visual cognitive strategy for FER. The specific process of FER algorithm based on the human GNN visual cognitive strategy is described in Algorithm 1.

Time complexity analysis: The algorithm proposed in this paper is divided into two key processes: “partial visual recognition process” and “regional collaborative recognition process”. In the process of local visual recognition, the facial expression area is first divided into \({Q}_{1},{Q}_{2},\dots ,{Q}_{i}\) parts, each part has \({n}_{1},{n}_{2},\dots ,{n}_{i}\) key feature points. The time complexity at this time is \(O(1)\). Then, the credibility of the key feature points in each area of ​​the face is calculated by Eq. (5). The time complexity at this time is \(O{(n}_{1}),O{(n}_{2}),\dots ,O{(n}_{i})\). Finally, the ranking function is used to rank the credibility of the key feature points in each region. According to the result of ranking the credibility of key feature points,\({N}_{1}, {N}_{2},\dots ,{N}_{i}\) key cognitive features are selected in each area by using Eq. (8). The time complexity at this time is \(O{(N}_{1}),O{(N}_{2}),\dots ,O{(N}_{i})\). Therefore, the time complexity of the local visual cognition process is \(max(O{(n}_{1}),O{(n}_{2}),\dots ,O{(n}_{i}))\). In the process of regional collaborative recognition, first the distance of each cognitive key point is calculated (the sum of the selected cognitive key points is \(O(N)\), and the time complexity at this time is \(O(\frac{N(N-1)}{2})\) (According to the nature of time complexity, \(O(\frac{N(N-1)}{2})\) is equivalent to \(O({N}^{2}))\). Then, each key point is sorted according to the center distance \(D\), and the time complexity at this time is \(O(1)\). Finally, by mapping all neighbor nodes to other subsets, the edge weight of each cognitive key can be obtained, and the time complexity at this time is \(O(N)\). Therefore, the time complexity of the regional collaborative recognition process is \(O({N}^{2})\). According to the analysis of the time complexity of the local visual recognition process and the regional collaborative recognition process, the time complexity of the algorithm proposed in this paper is \(O({N}^{2})\). Note: The time complexity of our algorithm does not consider the complexity of calling the function.

To verify the performance of the newly proposed model for FER, this paper uses the cross-validation method with the Helen dataset [45], the 300 W dataset, and the LFPW dataset [46].

The 300 W dataset and the LFPW dataset were released one after another at the 2013 International Computer Vision Conference. These two datasets are two types of data with marked key point information specifically used for automatic facial expression recognition. Most of the images were taken outdoors, while a small portion were taken indoors. The Helen dataset was proposed in 2012 and includes the changes in the background and facial expressions of the characters (e.g., changes in the poses of the subjects, changes in the background light, changes in the facial expressions, and so on). It is particularly important to note that before annotating the true location of the dataset, the data must be checked for anomalies in order to obtain high-quality annotated data results. Although the images in these three types of datasets all show human faces, there are no labels representing facial expressions. Therefore, this work mainly uses deep learning based FER algorithms and supplemented by a manual method to accurately assign these images with facial expressions to the seven types of tags/classes: angry, disgusted, fearful, happy, sad, surprised, and natural (inappropriate images are removed). The distribution of the number of images corresponding to the seven facial expressions in the above dataset is shown in Table 1.

Cross-validation is a method used to evaluate whether a trained model can be applied to another dataset with the same data structure. In this method, the original data is divided into two parts, one part is used as a training dataset to train the model, and the other part is used as a test data set to evaluate the model. In this work, first, each data set (Helen dataset, 300 W dataset and LFPW dataset) is divided into 2 sets of data samples and these 2 sets of data samples are used for training respectively. Then another set of data samples is used as a test set for testing. Finally, the average recognition accuracy of these 2 models is used as the average performance standard of the model.

The experimental environment used in this paper is based on Pytorch1.4 under the Ubuntu16.04 system. The hardware environment consists of an Intel(R) Xeon(R) Gold5218 CPU, a GeForce RTX 1080Ti (1 GPU) with 30 GB memory, and a 1 T mechanical hard disk.

To investigate the behavior of the proposed method of local uniform selection, we performed an ablation study. From the theoretical point of view, the coordinate information of the key feature points of the face, the visual behavior of the local uniform selection of main features, and the random selection of key features are analysed, as shown in Fig. 7.

In the upper image in Fig. 7a, it is obvious that the reliability of some key feature points detected by the deep learning algorithm is low, that is to say, many feature points are unnecessary and may even have a bad impact on the final judgment. As shown in the upper image in Fig. 7b, when the key feature points with high reliability are selected by using the random selection method, the selected key features will not be evenly distributed, and the state of the entire facial expression cannot be effectively described. That is to say, the feature information of a relatively important part may be missed, such as eyebrows. Therefore, it is necessary to uniformly select the key feature points by dividing the face regions (upper left, upper right, middle left, middle right, upper middle, and lower middle), as shown in the upper row of images in Fig. 7c. Compared with Fig. 7a, the key feature points selected in Fig. 7c are more reliable and the model is more concise. Compared with Fig. 7b, c can well describe the state of facial expression, and there is no omission of a key part. It can also be seen from the heat map of the key feature points in the second row of the figure that (c) has a more concise and efficient representation. In other words, the uniform selection method can not only describe the entire face information, but also improve the characterization and generalization ability of GNN.

The two methods of locally uniformly selecting key feature point information and randomly selecting key feature point information were quantitatively analyzed using the 300 W dataset, as shown in Fig. 8. From Fig. 8, it can be seen that the GNN model created by the random key feature point selection method cannot effectively describe the facial expression features globally, resulting in poor model recognition. The GNN model constructed by uniformly selecting key feature points can recognize the relevant features of the face from the entire domain and helps to accurately recognize facial expressions.

It can be seen from Table 2 that the detection accuracy of the random key feature point selection method on seven different expressions is lower than that of the GNN model constructed by uniformly selecting key feature points, which indicates that uniformly selecting key feature points more effectively describes the overall facial features and helps to accurately identify facial expressions.

To illustrate the feasibility of the model proposed in this article, the 300 W, Helen, and LFPW datasets are split into training and validation sets in a 7:3 ratio for verification. The heat map of the model for the 300 W, Helen, and LFPW validation sets is shown in Fig. 9.

The left side of Fig. 9 shows the heat map of the GNN model in the 300 W verification set, the middle of Fig. 9 shows the heat map of the GNN model in the Helen verification set, and the right side of Fig. 9 shows the heat map of the GNN model in the LFPW verification set. As can be seen from the three small images in Fig. 9, the main misclassification of facial expressions is the confusion between happy and natural expressions. In the 300 W, Helen, and LFPW verification sets, the percentages of happy expressions misclassified as natural expressions are 27.27%, 41.84%, and 32.87%, respectively, while the percentages of natural samples misclassified as happy expressions are 8.87%, 35.03%, 32.87%, and 25.87%. This high degree of confusion between happy and natural expressions is due not only to occlusion, postural changes, or other factors, but also to similar lip movements in the lower part of the face (mouth).

To further verify the effectiveness of the proposed GNN model based on visual cognitive strategies, this work compares the GNN model with traditional IACNN [47], Inception [32], VGG19 [48], Resnet18 [49], SAFC [50], and SCN [51] models on the 300 W dataset, Helen dataset, and LFPW dataset.

Automatic FER is a long-standing problem in computer vision. In this section, we verify the accuracy of the proposed GNN model in FER by testing several images with seven common facial expressions. The visualized result of FER based on the GNN model is shown in Fig. 13.

In Fig. 13, the upper row is a plot of visualization results from FER over a neural network constructed by uniformly selecting key feature points, and the lower row is a plot of visualization results from FER over a neural network constructed by randomly selecting key feature points. From the upper row of images in Fig. 13a, it can be seen intuitively that the lady's facial expressions are happy. However, it is not scientific to rely only on intuition. If you analyze and evaluate the model proposed in this article, the lady's facial expression can be considered happy with 95% probability. The model first learns and masters the information about the lady's facial expression in the image through the process of “local visual cognition”, and then uniformly selects key cognitive feature points with high credibility as nodes of the graph, such as the raised corners of the mouth in the lower area, the slightly curved eyebrows in the upper area, and the curved eyes in the middle and upper area. Finally, in combination with the “regional collaborative recognition” process, the edge information between the key cognitive features is extracted as the edge of the graph to build a GNN model that achieves the effect of FER.

However, in the lower row of images in Fig. 13a, the random selection method is used, which cannot accurately describe the entire facial expression, so the accuracy of the model is only 57%. In the upper row of images in Fig. 13b, the facial features of the man do not vary significantly. Therefore, the proposed model classifies the man's facial expression as natural with a probability of 92%. However, in the lower row of Fig. 13b, the key feature points are randomly selected, so the constructed GNN model cannot describe the facial expression state well. In this way, the accuracy of the natural facial expressions is only 63%. In the upper row of Fig. 13c, the GNN model constructed in this paper mainly extracts obvious feature information such as the wide-open eyes in the middle and upper regions and the slightly downturned corner of the mouth in the lower region of the girl in the image. Through the joint analysis, it can be concluded that the facial expression of the girl can be considered as fear with 96% probability. The recognition rate of the randomly selected GNN model is only 48%. In the upper picture of Fig. 13d, the mouth in the lower area of a man's face corresponds to an “o” shape, the eyes are widened in the middle-upper area, and the eyebrows in the upper area are raised. The model proposed in this paper uses these salient features to conclude with a 92% probability that the man is pleasantly surprised. However, the upper row of images in Fig. 13d uses a random selection that cannot describe the mouth in the lower part of the face. This will make the recognition rate of the GNN model only 49%. In the upper image of Fig. 13e, the enhanced model learns the feature information about the boy's facial features through the process of “local visual recognition”, such as the downturned corners of the mouth in the lower area, the frowning brows in the upper area, and the drooping in the middle and upper areas. According to the “regional coordination” assessment, the boy's sadness is 99%. However, in the lower row of images in Fig. 13e, the random selection cannot describe the whole facial expression, so the recognition rate of the GNN model is only 61%. Similarly, for the proposed model, the girls in the upper image of Fig. 13f have a disgust level of 98%. However, the recognition rate using random selection in the lower row of Fig. 13f is only 52%.

It is not difficult to see from the Fig. 13 that the extraction of feature points of each region is more accurate after using the region division mechanism. Combining the two key processes of “local visual cognition” and “regional collaborative recognition”, a GNN model is constructed by using the idea of human visual cognition from local to global. This model can more accurately express facial expression with less feature point information, reducing the network parameters and improving the accuracy of expression recognition.