Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization

The EEG data classified in the current study were recorded in a previous study, combined with simultaneous MEG recordings to compare both modalities for BCI use [46]. In brief, 19 participants (10 females) performed a covert attention task in multiple runs (10.7 runs per participant on average), each comprising 12 trials. In each trial, the participants covertly attended to one of 12 stimuli of their choice in an unpredictable order. The stimuli were presented as colored and numbered spherical objects at equidistant spaces over the circumference of an invisible circle with the fixation point located at its center. All the objects flashed five times in a pseudorandom order, leading to 60 flashes with a stimulus onset asynchrony (SOA) of 167 ms. The pseudo-random order ensured that the same object would not flash twice in a row, but with at least two other objects flashing in between. The EEG data were recorded at 29 standard electrode locations (Fp1, Fp2, Fz, F3, F4, F7, F8, FC1, FC2, Cz, C3, C4, T7, T8, CP1, CP3, Pz, P3, P4, P7, P8, PO3, PO4, PO7, PO8, Oz, O9, O10, Iz), sampled at 508.63 Hz, and referenced against the right mastoid. Vertical and horizontal electrooculograms (EOG) were also recorded to ensure fixation of the gaze and exclude classifications based on eye movements.

In order to test the potential of CNNs in dealing with high-dimensional EEG data, we created two datasets: one to represent high-dimensional, minimally preprocessed data and one to represent low-dimensional data preprocessed with a feature extraction method. For the first dataset, the data were only downsampled to half of the original sampling frequency (254.3 Hz) after applying an antialiasing FIR lowpass filter, and we call it dataset f250. For the second dataset, the data were decimated by a factor of 10 by averaging over blocks of 10 time points, which essentially is equivalent to downsampling after applying a moving average filter. These averages were then used as the temporal features [31]. This process led to a sampling frequency of 50.8 Hz, and hence we call it dataset f50_avg. This method is justified based on the knowledge that the P300 response—the signal of interest—is of low frequency relative to the sampling frequency [24, 31]. Each trial, comprising 60 stimuli flashes, was then segmented into 60 segments from 100 ms pre-stimulus to 700 ms post-stimulus. Each segment represents an input example, x, whose corresponding output, y , is either 1 if the participant attended this location or 0 otherwise. Each segment was then baseline-corrected by subtracting the average of the pre-stimulus interval. Since the dataset was imbalanced, we oversampled the minority class in the training set by randomly replicating examples until the fractions of both classes were equal before training the neural network models [20]. The input data matrix was then normalized by subtracting the mean and dividing by the standard deviation, both determined from the training set.

A convolutional layer of a CNN is formed from multiple feature maps. Each feature map comprises artificial neurons that share the same set of weights, but they apply their weights to a different patch of the input. These weights are called a filter, a kernel, or a feature detector, and their input patches are the equivalent of receptive fields, in the analogy between CNNs and the visual system. Each feature map thus represents the result of a feature detector striding through the input to scan it and outputs a similarity measure to this feature in all possible locations. These filters are learned from the data with backpropagation and gradient descent in the same way as fully connected networks [12]. That is why a CNN is considered a feature extractor as well as a classifier or regressor, based on the task for which it is used. Convolutional layers are usually followed by pooling layers. A pooling layer is a downsampling operation that is applied to each feature map. The pooling operation is defined by a pooling size and a pooling stride. The pooling size controls the input patch size, and the pooling stride controls how to move to the next input patch. The pooling layer serves to reduce the dimensionality of the network and therefore the complexity of the model. It also causes the model to be spatially invariant.

The branched model.

We designed a model consisting of five convolutional layers plus the input and output layers (figure 1). The first convolutional layer is purely spatial, i.e., the filter extends through all the channels at each time point separately. On top of the output of the spatial filters layer, we implemented two parallel temporal convolutional layers: one with a small filter size that is equal to 100 ms and the other one with a large filter size that is equal to roughly 400 ms. EEG data are thought to be formed from overlapping harmonics of different frequencies. The small and large filters should be able to detect high frequency and low frequency patterns in the data, respectively [53]. We then added two more deep convolutional layers after the small filters layer, so the network is deep and wide at the same time. We used tanh as the non-linear activation function for each layer, except for the output layer, for which we used a sigmoid function. The reason behind choosing tanh as an activation function is that its output range includes positive and negative values with a mean of zero. This is more suitable for representing features of EEG data, which include positive and negative values, than the commonly used ReLU function [17], whose output range has only positive values. Additionally, we used batch normalization [25] and dropout [51] of 0.5 after each layer to accelerate training and regularize the network. Moreover, we used L2 regularization with regularization strength 0.01 for each layer in the network.

Zoom In Zoom Out Reset image size

We validated the impact of different model components by omitting single components, retraining the model and comparing the results. We therefore trained five additional models:

• No_BN: batch normalization was omitted.
• No_Dropout: dropout was omitted.
• Not_Branched: the branch with the large temporal filter was omitted to validate the impact of the parallel convolutions.
• Not_Deep: the last two layers after the small temporal filters branch were omitted so that the model was formed only from the two parallel convolutional layers on top of the spatial convolutional layer.
• ReLU: we replaced the tanh activation function with the ReLU function [17].
• ELU: we replaced the tanh activation function with the ELU function [11].

We compared the performances of the branched model with a general CNN model for EEG-based BCIs, the EEGNet model [32], in order to test the potential of CNNs in general, and also with a traditional machine learning technique frequently found in the BCI literature, linear discriminant analysis (LDA) [26] and LDA with shrinkage [4].

To evaluate the performance of the models, we validated within-subject models as well as cross-subject models. For each participant, a model was trained only using the participant's individual data, rendering the model subject-specific. Since the data are limited per participant, we implemented a 5-fold sequential cross-validation scheme. For LDA, five models were trained per participant, each trained with data from four folds as the training set, and the models were then tested on the data of the held-out fold. For CNN models, we further divided the data from the four folds in a stratified random manner into 80% training data and 20% validation data.

To investigate the model's transfer learning ability, we implemented a leave-one-subject-out cross-validation scheme in which one participant's data were held back as a test set, and the data from the remaining 18 participants were used as training and validation sets. We then fine-tuned each cross-subject model with a subset from the test participant's data in order to evaluate whether we could achieve better performance than with random initializations.

We calculated a within-subject saliency map for each participant based on the participant's own data and one cross-subject saliency map based on the data of all the participants. In order to calculate the within-subject saliency maps, we trained one model per participant by dividing the participant's data in a random stratified manner into a 90% training set and a 10% validation set. For the cross-subject saliency map, we trained one cross-subject classifier by also randomly dividing the data of all the participants in a random stratified manner into a 90% training set and a 10% validation set. The saliency maps were then calculated by averaging over all the saliency maps of the target examples in the dataset.

LDA models were implemented using the Scikit-learn library [41]. CNN models were implemented using Keras API [9] with Tensorflow backend [1]. The training experiments were conducted on our university's Medusa CPU cluster.

To optimize the models, we used the Adam optimizer with the recommended settings (learning rate  =  10⁻³, $\beta_1=0.9$, $\beta_2=0.999$ and $\newcommand{\e}{{\rm e}} \epsilon=10^{-8}$) [27]. For fine-tuning, we used stochastic gradient descent with learning rate  =  10⁻⁴ and momentum  =  0.9. We used a decaying learning rate scheme that halves the learning rate if there was no decrease in the validation loss for 10 epochs. The early stopping technique was applied to avoid overfitting and to reduce the computational time by stopping the training when the validation loss did not decrease for 20 epochs. The batch size for the mini-batch gradient descent was set to 64 examples. The weights of the networks were initialized with the Glorot Uniform method [16].

With our branched model, we achieved average decoding accuracies of 90.6% (SD: 10%) and 90.7% (SD: 8.6%) on the lower-dimensional and higher-dimensional datasets, respectively (figure 2(a)). The diagrams show the decoding accuracies averaged over the participants. Decoding accuracy indicates the prediction of the target stimulus location as obtained from the binary classification of each stimulus, i.e., the chance level is at 8.3%. Since each stimulus location was flashed five times, we averaged the model outputs of the five repetitions to obtain more robust and accurate predictions. Asterisks above the bars represent the p-value of a Wilcoxon signed-rank test between this model and the branched model, Bonferroni corrected for multiple comparisons [13]. The branched model yielded significantly higher decoding accuracies than the EEGNet model on the f50_avg dataset and than all three models on the f250 dataset. It is important to note that the branched model was designed for the f50_avg dataset and then simply scaled up to fit the bigger f250 dataset. EEGNet was originally designed for a dataset with a sampling frequency of 128 Hz [32], and therefore, we scaled it up to fit the f250 dataset and down to fit the f50_avg dataset. The EEGNet model achieved significantly better decoding accuracy (p   <  0.001) on the f250 dataset than on the f50_avg dataset. In contrast, the difference of decoding accuracies between lower- and higher-dimensional datasets was not significant with the branched model. For the LDA classifier, the shrinkage regularization improved the decoding accuracy (p   <  0.001) on both datasets, which is particularly clear on the f250 dataset with higher dimensions, as expected. Nevertheless, on the higher-dimensional dataset, CNNs performed significantly better than LDA with shrinkage.

Zoom In Zoom Out Reset image size

With the cross-subject branched model, we achieved decoding accuracies of 81.2% (SD: 12.4%), and 79.9% (SD: 12.1%) using the f250 and f50_avg datasets, respectively (figure 2(b)). There was no significant difference between the branched model and any of the other three models on the f50_avg dataset, but it still yielded significantly higher accuracy than both LDA classifiers on the f250 dataset. While the branched model only showed a tendency towards better performance on the high-dimensional f250 dataset, the EEGNet model achieved a significantly higher accuracy (p   <  0.05).

Moreover, the effect of shrinkage on the LDA classifier is not significant, as in the case of within-subject classifiers on both datasets. This is because shrinkage works in situations where the number of training examples is small compared with the number of features. In this case, when the data were pooled from 18 participants, this ratio was high and therefore, LDA was less prone to overfitting, and the regularization was not advantageous.

We investigated the impact of different components of the branched model by comparing the decoding accuracies for different alternative within-subject models, in which we omitted or changed one of the model components (figure 3(a)). Again, we used Wilcoxon signed-rank tests with Bonferroni correction to test the significance of the differences. Omitting the dropout and changing the activation function to ReLU caused the most significant negative effect on both datasets. Removing the large temporal filter branch induced a significant negative effect on the f50_avg dataset. The effect of batch normalization was mainly negligible and differed across the datasets. Using the ELU activation function led to a non-significant improvement in the performance across both datasets. The same effect was observed when removing the deep convolutional layers (Not_deep).

Zoom In Zoom Out Reset image size

We also tested the impact of the model components for the cross-subject models. We observed a similar trend to that seen with the within-subject models: mainly the performance decrease of the ReLU activation function, No_dropout, and Not_branched and the performance increase of Not_deep and the ELU activation function (figure 3(b)).

Because the addition of a 400 ms large filter branch is comparable to applying a low pass filter, which is of relevance in ERP identification, we additionally evaluated the performance using only this extra large branch in the CNN model. As shown in figure 3(a), removing the large filter branch (the Not_branched model) led to a 3.1% and 0.8% decrease in the decoding accuracy of the within-subject models applied to the lower- and higher-dimensional datasets, respectively, while removing the deep layers of the small filter branch (the Not_deep model) yielded a 0.2% and 0.4% increase in the decoding accuracy, respectively. Additionally, training within-subject models on the lower- and higher-dimensional datasets after removing the whole small filter branch leaving only the large filter led to a decrease in decoding accuracy of 13.2% and 2.4%, respectively.

Training cross-subject branched models yielded average decoding accuracies of 79.9% and 81.2% on the f50_avg and f250 datasets, respectively (figure 2(b)). Starting with these models, we continued to fine-tune them with the test participants' data. We used 10, 20, 30, 50, 60 and 70 trials to further train the models and re-evaluated them using the remaining data (figure 4). This approach simulates the situation when a new participant uses the model out-of-the-box and tunes it with subject-specific training trials. Incrementally adding subject-specific trials, a significant improvement in accuracy was observed for both datasets from 30 additional trials. After using 70 trials, the models achieved decoding accuracies of 91.8% (SD: 9.2%), and 91.1% (SD: 9.6%) on the f50_avg and f250 datasets, respectively.

Zoom In Zoom Out Reset image size

These results cannot be compared directly to the results of the cross-validation scheme of the within-subject models trained from random initializations because of the difference in the test schemes. We therefore trained within-subject models with the same cross-validation scheme but starting from the cross-subject model as a starting point instead of random initializations. These new within-subject models achieved decoding accuracies of 91.9% (SD: 8.5%), and 93% (SD: 6.7%) on the f50_avg and f250 datasets, respectively. They showed improvements over the 90.6% and 90.7% reported for cross-subject models (figure 2(b)). These improvements were significant for both datasets (f50_avg: p   <  0.05; f250: p   <  0.001).

To create the within-subject saliency maps, the single-example saliency maps (see figure 5) for all the target examples in the participant's dataset were averaged. We examined the within-subject saliency maps of two participants for the branched model, who had decoding accuracies of 98.4% and 73.6% respectively, and were chosen to illustrate saliency maps for high- and low-performing participants (figure 6). The features expected to drive the classification using this paradigm are identifiable on inspection of the saliency map of the high-performing participant: central and parietal electrode locations (FC2, CP1, CP2, and P7) show salient features 300–500 ms post-stimulus. The saliency map of the low-performing participant, on the other hand, has a more scattered appearance across electrodes and over time, which could be due to noisy data leading to low decoding accuracy and poor noisy saliency maps.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The cross-subject saliency map (figure 7) is more suitable for visualizing more general task-relevant features, as the cross-subject model is less likely to pick up subject-specific, task-irrelevant features (see the difference for a single input example in figure 5). These subject-specific features could help improve the decoding accuracy of within-subject models but at the expense of their generalizability to other participants. Examining the grand average ERP at the most discriminative electrode, we observed some resemblance between the ERP waves and the gradients shown on the saliency map (figure 7). It is important to note that the resemblance is not exact as the models were trained on a set of single trials which differed from the average wave. We observe that the grand ERP for the target stimuli shows higher amplitude than the grand ERP of the standard stimuli around 200 ms and after 300 ms and lower amplitude around 100 ms and 300 ms. These differences are reflected on the saliency maps by positive and negative gradients. Note that the negative difference at around 100 ms is not represented on the saliency map which could be because the difference is small and not relevant for the classification task.

Zoom In Zoom Out Reset image size

CNN models were found to be especially powerful in dealing with high-dimensional data (f250), as they are capable of automatically extracting task-relevant features from the data. Therefore, besides gaining insights from the saliency maps by visual inspection, it would be also useful to quantify them in order to learn the relevant spatial and temporal features that drive the network decision. To this end, we quantified the saliency maps by computing the average of their absolute values across space and time.

Spatial features are defined as the most discriminative electrodes across the whole trial time period. The eight most discriminative electrodes for each participant separately computed on the within-subject saliency maps are illustrated in figure 8(a). We can observe that they were variable across participants, but were mainly located around central and parietal electrodes, with some participants having temporal, occipital, or frontal extensions. The eight most discriminative electrodes extracted from the cross-subject saliency map also showed central and parietal distribution (figure 8(b)).

Zoom In Zoom Out Reset image size

Temporal features are defined as the most discriminative time periods for the detection of the target stimuli. The most discriminative 100 ms time window across all the electrodes for within-subject models and for the cross-subject model fell in the interval between around 300 and 500 ms post-stimulus (figure 9).

Zoom In Zoom Out Reset image size

These spatial and temporal features correspond with the well-established spatial distribution and time period of the P300 component [43]. The distribution over time of the normalized gradients, averaged across all the electrodes for each time point for both the within-subject and cross-subject saliency maps, resembled a trimodal distribution, with the highest contribution coming from the time period between 300 and 500 ms, followed by contributions from the time period between 100 and 200 ms and around 600 ms (figure 10). Each peak in the distribution could be interpreted as reflecting a specific neural process in the brain that is modulated by covert attention at a specific time period.

Zoom In Zoom Out Reset image size

We validated the interpretation of the absolute values of the gradients in the saliency maps as the discriminative power of the features by comparing the decoding accuracy using all electrodes with that obtained using only the eight most and the eight least discriminative electrodes, setting all other electrodes to zero in the latter cases (figure 11(a)). Inside the cross-validation scheme of evaluating the performance of the within-subject branched models, we used the trained models to generate saliency maps, from which we computed the rank of the electrodes (as mentioned previously and shown in figure 8(a)). Visual inspection clearly reveals a difference in the decoding accuracies in the case of using the eight most discriminative versus the eight least discriminative electrodes (figure 11(b)). Using only eight out of the 29 electrodes means using only 27.5% of the features. Using the eight most discriminative electrodes, we retained 72% of the decoding accuracy versus only 16% in the case of using the eight least discriminative electrodes

Zoom In Zoom Out Reset image size

Furthermore, we compared the eight most discriminative electrodes determined with the saliency map approach to an approach that is based on considering the weights (spatial filters) of the first hidden layer [8, 36]. We trained within-subject LDA classifiers with shrinkage using only the eight most discriminative electrodes determined by both approaches. There was no significant difference between the two approaches, as using saliency map and spatial filter approaches yielded decoding accuracies of 80.70% and 81.19%, respectively (p   =  0.6). Moreover, by comparing the eight most discriminative electrode sites for the cross-subject model, there was only a difference in one electrode (replacing electrode C3 from figure 8(b) by electrode CP2).

Convolutional neural networks performed significantly better than traditional decoding techniques on the high-dimensional data. However, this superior performance comes at the expense of time and computational costs. Using the standard recommended model components does not guarantee the best performance and therefore manual optimization is still needed. More studies that validate and potentially replicate our design recommendations are also still needed. Thus, we believe that for cases, in which we have domain knowledge of the classification problem, like in the case of P300 classification, using a simple linear classifier on top of a feature extraction method that is based on this domain knowledge, offers a more practical solution. On the other hand, in novel cases, in which we do not have sufficient prior knowledge of the task-relevant effects on the EEG signal, using CNN models is justifiable and recommended. Training CNN models in these cases is not only more likely to produce performances competitive with other methods, but also to provide insight into the task-relevant features through visualization techniques like saliency maps.

For such models to be applicable in real world situations, they need to be tested on other datasets. Further investigation is required to establish whether they could be used 'out of the box' with reasonable performance, needing only some fine-tuning with a small amount of data, or whether retraining from scratch is necessary.

Moreover, there is still room for improvement, through exploring different neural network architectures, such as recurrent neural networks (RNN) [18], long short term memory networks (LSTM) [21], hybrid networks, and automatic neural architecture search techniques [35, 42].

Using CNN models for EEG decoding still needs further development before such models could become the standard approach and replace other techniques. The research in this area needs to go beyond intensive initial parameter adjustment and start to investigate the underpinnings of the models when they are applied to EEG data.