Mixture Kernel Density Estimation and Remedied Correlation Matrix on the EEG-Based Copula Model for the Assessment of Visual Discomfort

The marginal density function of an arbitrary variate can be estimated by KDE with smooth kernel functions. Based on Scott’s research [24], the quality of KDE greatly depends on the smoothing parameters of the kernel functions. Assuming there are \(\textit{\textbf{N}}\) \(d-\)dimensional samples \(\textit{\textbf{X}}=[{\textit{\textbf{x}}}_1,{\textit{\textbf{x}}}_2,...,{\textit{\textbf{x}}}_N]\), the marginal density function of the variate \(\textit{\textbf{X}}\) can be smoothly estimated by:where \(K_{\textit{\textbf{H}}}(\cdot )\) is the non-negative kernel function. The \(\textit{\textbf{H}}\) is \(d\times d\) symmetric and positive definite matrix which determines the smoothness of the estimated marginal density function [25]. Once the d equals to 1, the matrix \(\textit{\textbf{H}}\) will degenerate to a scalar h.

As the number and the dimension of samples increase, the complexity and the cost of KDE can be relatively high. Therefore, KDE combined with binning strategy [26] or with K-means [27], reduced set density estimator [28], finite mixture model [29], etc., are proposed to reduce the scale of data to mitigate the mentioned problem. However, these methods or models ignore the fine-grained information hidden in the overall samples, affecting the estimated marginal density function. Based on the assumption that the overall data may be composed of several subcomponents [25, 27], the MKDE employing the KDE with the K-means and multiscale strategy is proposed to explore fine-grained information for better estimation. Concretely, the whole data are firstly clustered into several clusters by K-means, which is a classic unsupervised method in the field of clustering. After that, to dig out the fine-grained information, KDE with different smoothing parameters, hereinafter multiscale strategy, is used to estimate the marginal density function of each cluster. In terms of the theory of the KDE, the larger and smaller smoothing parameters separately focus on global data and regional data. Therefore, with the multiscale strategy, smaller smoothing parameters can explore fine-grained information. Meanwhile, the other larger smoothing parameters can compensate for the non-smoothness of estimated marginal density function caused by the smaller smoothing parameters. The concrete equations are as follows:where \(f^*(\cdot )\) represents the marginal density function of the \(\textit{\textbf{N}}\) \(1-\)dimensional samples data \(X=[x_1,x_2,...,x_N]\). The k is the kth cluster in C clusters. The coefficient \(\lambda _k\) represents the contribution of the marginal density function of the kth cluster to the marginal density function of the overall data, which is the ratio of the number of data in the kth cluster to the entire data. The \(\widehat{f_k}(\cdot )\) is the estimated marginal density function of the kth cluster.where j means the jth smoothing parameter among m smoothing parameters. The coefficient \(\alpha _j\) represents the contribution of the jth smoothing parameter to the estimated marginal density function of the kth cluster, which is the reciprocal of m smoothing parameters. The \(\widehat{f_j}(\cdot )\) is the marginal density function estimated by KDE with the jth smoothing parameter.where \(N_k\) represents the number of data in the kth cluster. And \(x_i\) is the ith data in the kth cluster. The \(h_j\) and \(K_{h_j}(\cdot )\) are the value of and the kernel function of the jth smoothing parameter.

The framework and procedure of MKDE are, respectively, shown in Fig. 2 and Alg. 1. As Fig. 2 shows, the overall EEG data are grouped into several clusters by K-means. The candidates for the number of clusters should be set in advance. The GridSearch algorithm is introduced to find the best result, a general exhaustive method for finding the optimal parameters. Although the smaller smoothing parameter can explore fine-grained information, too small parameters may be easily disturbed by noise. Thereby, the GridSearch algorithm is also used to find the optimal number of smoothing parameters. With different smoothing parameters, the estimated marginal density function of a single cluster may present some differences, as the dashed blue rectangle in Estimation with multiscale part shows. The final marginal density function of a single cluster is obtained by the weighted summation of the estimated marginal density function by different smoothing parameters, as the dashed blue rectangle in Estimation of each cluster part shows. The estimated marginal density function of the overall data is weighted mixed by the estimation results of clusters, as the final estimation with cluster part depicts.

There are several candidates for the copula density function, such as Gaussian copula, Student’s t copula, and Clayton copula [30‐32]. The Gaussian copula is selected in this paper for its wide applications on time series. Each variate in the Gaussian copula density function is assumed to follow the Gaussian distribution for simplifying the Gaussian copula density function. The Gaussian copula density function is expressed as:where \(c(\cdot )\) is the Gaussian copula density function. The parameters \(\mu\) and \(\Sigma\) are separately the mean and the correlation matrix of \(c(\cdot )\). The parameters \(\mu _i\) and \(\Sigma _i\) \((i=1,\;...,N)\) are the mean and the correlation matrix of the Gaussian distribution of variate \(x_i\).

Because this paper focuses on better extracting the time-domain EEG features by the EEG-based copula model, estimating the correlation matrix \(\Sigma\) is crucial. The locality-sensitive dictionary learning is introduced to mitigate the pseudo-correlation among EEG data caused by noise, distortion, etc., for better estimating the correlation matrix. Specifically, based on the theory of the dictionary learning, the original data can be decomposed into a dictionary matrix that learns the essence of the original data, and a sparse matrix (hereinafter remedied data) that learns the most important information of the original data [33, 34]. Because the remedied data can keep the most relevant information to original data, it is considered to contain less pseudo-correlation. In the conventional dictionary learning, only considering the constraints on coefficients, locality-sensitive dictionary learning has the advantage of exploring the local information of data for data construction [35‐39]. Therefore, locality-sensitive dictionary learning is used to obtain remedied data. The columns in the remedied data are considered as variables to estimate RCM. As equation (7) indicates, the marginal probability distribution of each variable \(x_i\) is required to be estimated. Assuming that each variable \(x_i\) follows a Gaussian distribution, the parameters \(\mu _i\) and \(\Sigma _i\) can be obtained from the sample mean and sample variance. The data sampled from the estimated Gaussian distribution are concatenated in the order of the columns of the remedied data. The RCM is finally estimated by correlation analysis of the concatenated data. The process of estimating the RCM is shown in Fig. 3.

Five kinds of simulated EEG data are generated to evaluate MKDE, namely clean data, data with signal to noise (SNR) of 5dB, data with SNR of 10dB, data with SNR of 15dB, data with SNR of 20dB. Simulated EEG data induced by auditory and visual stimuli are generated to evaluate RCM. All simulated EEG data are generated by the MNE-Python package.

The behaviour experiment aims to subjectively assess visual discomfort induced by the designed stereoscopic stimuli to assist in selecting reasonable stimuli that can be used in the EEG experiment. In terms of previous studies [7, 40], the symptom questionnaire is chosen to be the subjective questionnaire. The question in the subjective questionnaire is whether the stimulus that just appeared makes you visual discomfort. Participants need to select one from the five choices (no fatigue, mild fatigue, moderate fatigue, serious fatigue, and extreme fatigue) to rate the degree of visual discomfort according to their feelings. These five choices are, in turn, quantified as five levels (1, 2, 3, 4, 5). The rounded average level from all participants is regarded as a label of the stimulus. Referring to the studies of other researchers and our previous research [5, 40‐42], disparity images with disparities ranging from \(\pm 0.1^\circ\) to \(\pm 1^\circ\) are used as stimuli. The designed stimuli are presented by the LG D2343P display with \(1920\times 1080\) resolution. Seventeen participants aged from 23 to 27 with normal stereo vision were recruited from Beijing Normal University.

The EEG experiment is designed to record the EEG signals of participants when facing stimuli for objectively assessing visual discomfort. With the results of the behaviour experiment, ten disparity images are selected that can effectively induce visual discomfort or maintain visual comfort. Forty-three participants were recruited from Beijing Normal University, including 22 males and 21 females. Before the experiment, each participant was suggested to stay healthy, avoid staying up late and excessive stress. The EEG signals were recorded by the 128-channel EGI system. The sampling frequency is 500 Hz. Each participant has a chance to practice several times before the formal experiment starts. In Fig. 4, the ‘Cross’ image is presented to fix the participant’s vision. Once the stimulus disappears, the judgment image will be presented to encourage each participant to determine whether the stimulus just appeared in front of or behind the screen. The judgment part aims at keeping the participant’s attention on the current experiment. The repetition times are set to 150 to balance the trade-off between recording the EEG data as much as possible and preventing the long experimental time from inducing extra visual discomfort.

Because raw EEG data are easily disturbed by irrelevant components, such as eye movement and head movement, independent component analysis (ICA) is introduced to eliminate the first two or three components.

Inspired by researches [43, 44], the autoregressive model (AR model) is introduced to extract rough EEG features, one of the classic methods for roughly extracting time-domain EEG features. The coefficients of the AR model (the extracted rough EEG features) are determined by AIC. For each participant, since the dimension of rough EEG features is high ([number of channels \(\times\) number of AR coefficients]), the dimension reduction is required for reducing the computation cost. The Random Forest is adopted as it not only keeps valuable features but also shows the importance of these features [45]. Concretely, the importance of channels can be ranked by Random Forest following the Gini coefficient when channels of rough EEG features are considered as the nodes of Random Forest. Based on the results of all participants, the top 40 most important and frequent channels are kept. The reduced dimension of rough EEG features is [number of selected channels \(\times\) number of AR coefficients].

As Fig. 5 presents, the extracted EEG features by our proposed model are the linear concatenation of the smoothing parameters from MKDE and the RCM according to the order of selected channels. The smoothing parameters can be obtained by MKDE used to estimate the marginal density function of the overall data from an individual channel. The RCM is obtained by the remedied data from a specific stimulus. Based on the extracted EEG features, SVM with radius basis function (RBF) kernel is used to assess visual discomfort. The parameters C and gamma of SVM are determined by the GridSearch algorithm.

As Fig. 6(a) shows, with the SNR increases, simulated EEG data are gradually close to clean data. In Fig. 6(b), the grey, green and blue lines individually represent the real marginal density function, the estimated marginal density function by KDE and the estimated marginal density function by MKDE. The estimation results of the clean data or the data with high SNRs by MKDE are much closer to the real marginal density functions. However, the estimation results of the heavily contaminated data (as the second and third subplots in Fig. 6(a) presents) by MKDE deviate from the real marginal density functions.

The mean-squared error (MSE) is used to evaluate the differences between the real and the estimated marginal density functions for objectively testifying MKDE. In Fig.7, the grey and blue lines separately represent the estimation results by KDE and MKDE. As SNR increases, the MSE of the estimation by both methods is gradually stable. However, the MSE of the estimation by MKDE is smaller. For heavily contaminated data, the gap between the performance of MKDE and the performance of KDE is narrowed. But if the number of clusters is set to one, the performance of MKDE is the same as KDE.

The details of MKDE exploring the fine-grained information hidden in the overall data are shown in Fig. 8. Better than KDE with only one smoothing parameter, MKDE not only uses the best smoothing parameter to estimate the marginal density function but also employs several smoothing parameters to ensure that hidden information at different fine-grained can be explored. Fig. 8(c) shows that the marginal density function of the data from one cluster estimated by selected smoothing parameters present apparent differences, which further indicates that these parameters can mine the fine and different-grained information. The marginal density function of one cluster, showing in Fig. 8(d), is obtained by the weighted summation of the estimation with selected smoothing parameters. Repeating the process presented in Fig. 8(c) and Fig. 8(d), the marginal density function of each cluster can be estimated, as Fig. 8(e) shows. The final marginal density function of the overall data is estimated by a weighted mixture of the estimated results from each cluster.

Figure 9 shows the heatmap results of the correlation matrix parameters of the copula density functions obtained by original data and remedied data. The presented heatmap results are randomly selected from the heatmap results of all correlation matrices. In Fig. 9(a) and 9(b), the left and the right subfigures present correlation matrices obtained by the original data and the remedied data. According to Chen et al. [46], channels in the fronto-central region and occipital region are highly sensitive to hearing and vision, respectively. Hence, channel Fz is selected to be an example of further analysis because it is sensitive to the auditory stimulus. As the white rectangle in Fig. 9(a) shows, the correlation value between the channels FC1 and Fz, and channels C4 and Fz in RCM gets increased. Both correlation values between channels T7 and Fz, and channels P3 and Fz in RCM get decreased. The correlation relationship between channels O1 and Fz in RCM remains stable. All findings are consistent with the results of Chen et al. The channel Oz highly related to visual stimuli is taken as an example. The correlation value between channels PO3 and Oz in RCM presents mild increases, which follows the findings in the study of Chen et al. The correlation relationship between channels C5 and Oz, and channels C6 and Oz in RCM both remain relatively stable. The correlation value between channels F3 and Oz in RCM shows significant changes, which may be explained by the fact that the frontal region has a connection to the visual process [47]. Therefore, the differences between the correlation matrices indicate that the RCM contains less pseudo-correlation.

To find the EEG features extracted by our proposed model highly related to visual discomfort, various linear concatenations of the smoothing parameters from MKDE and the RCM are implemented. In Fig. 10, the horizontal coordinate presents different linear concatenations. The details of these extracted features are shown in Table 1. In Fig. 10, the top three highest accuracies of the SVM classifiers on different features are marked. The standard deviations of these three accuracies are also shown to verify the stability. The leftmost rectangle represents the accuracy of the SVM classifier based on feature K_O. The remaining rectangles present the accuracies of SVM on the features extracted by the proposed model. As Fig. 10 shows, the features extracted by the proposed model perform better. In other words, no matter how concatenated, these features have better discrimination to visual discomfort. The differences between the accuracies of the SVM classifiers on feature K_O and feature K_R indicate that RCM could greatly improve the discrimination of features to visual discomfort. As the differences in the remaining rectangles depict, although the number of smoothing parameters may contribute little to improving the accuracy of the classifier, it can keep the classifier stable. And the increased number of smoothing parameters may not always contribute to the accuracy of the SVM classifier. This may be explained by that the smoothing parameters corresponding to the cluster with the smallest data may be disturbed by redundant or irrelevant information. The SVM classifier based on feature K_R presents the highest accuracy among the three marked features, whereas the standard deviation is the largest. The feature 2C_M_R is considered the most promising feature to balance the trade-off between accuracy and stability, which has high discrimination to visual discomfort. The accuracy of the SVM classifier based on the feature 2C_M_R is 81.63%.

The receiver operating characteristic curve (ROC curve) to evaluate the extracted features further is shown in Fig. 11. The area under curve (AUC) value of feature K_O or features extracted by the proposed model is larger than 0.8. It means that features extracted by the EEG-based copula model have high discrimination to visual discomfort. Compared to the remaining lines, the discrimination of features represented by orange, yellow, and dark green lines for visual discomfort is more precise, consistent with the results shown in Fig. 10. Therefore, the proposed model can extract the time-domain EEG features closely related to visual discomfort.

To testify the performance of the feature 2C_M_R extracted by the proposed model, the multiscale entropy (the state-of-the-art method) and sample entropy to assess discomfort are introduced. The AR model is also adopted to extract time-domain EEG features to compare with the feature 2C_M_R. Although the AR model has not been introduced into assessing visual discomfort, it is widely used to extract time-domain EEG features in other tasks, such as evaluating mental load [43] and classifying seizure [44]. Additionally, the statistics are performed on the features extracted by the AR model (hereinafter rough EEG features), such as the mean of rough EEG features and the standard deviation of rough EEG features, to explore the practical usefulness of the feature 2C_M_R. The results of the comparisons are shown in Table 2. The first column shows the method to extract the features. The second column presents the concrete features. The last column indicates the mean and standard deviation of the accuracy of the SVM classifiers based on extracted different features. The rough EEG features are not sensitive to visual discomfort. And the statistical features of rough EEG features do not show better discrimination to visual discomfort. All features extracted by entropy measures are less sensitive to visual discomfort compared with statistical features of rough EEG features. The features extracted by the proposed model perform better in discriminating visual discomfort despite the larger standard deviation. However, the value of the standard deviation is still much smaller than the mean value. Thereby, the features extracted by the proposed model present acceptable stability.

For MKDE, three parameters are needed to tune: the number of clusters C, the number of smoothing parameters m, and the coefficient \(\alpha _j\). Given the candidates of C, the GridSearch algorithm can exhaust all the clustering results of K-means. The C is selected according to the smallest within-cluster sum of squares. Given ranges of the number of smoothing parameters m, the GridSearch algorithm can exhaust all the results of estimated marginal density function with smoothing parameters. The m is selected according to the top several smallest mean error between the samples from the estimated marginal density function and the real data. Two strategies are used to find the coefficient \(\alpha _j\). The first strategy is to set the coefficient \(\alpha _j\) equal the reciprocal of the number of smoothing parameters m. The second strategy is to determine the coefficient \(\alpha _j\) is determined by the importance of each smoothing parameter to the estimated marginal density function. The importance of the smoothing parameters is determined by the reverse order of the sequence of error ratio, which is obtained by the ratio of the error of the estimated marginal density function by each smoothing parameter to the summation of the errors of the estimated marginal density function by all smoothing parameters. The coefficient \(\alpha _j\) of each smoothing parameter equals to its importance value. According to a simulation experiment, there is no significant difference between the estimated marginal density functions based on the two strategies. To reduce the computation cost, the coefficient \(\alpha _j\) is set to be the reciprocal of the number of smoothing parameters m.

In terms of Table 2, the time-domain EEG features extracted by the proposed method perform better than the features extracted by other methods, especially by the multiscale entropy (state-of-the-art method) proposed by Peng et al. Unlike the stereoscopic stimulus to induce the visual discomfort in this paper, Peng et al. designed the 2D stimuli with different flicking frequencies to cause visual discomfort. The differences between the designed stimuli may lead to different brain activities, which could explain that the features extracted by the multiscale entropy are less sensitive to visual discomfort induced by stereoscopic displays than the features extracted by the proposed model.