Effect of time windows in LSTM networks for EEG-based BCIs

New technological developments increasingly require non-physical interaction with the user. These types of applications not only cover routine or leisure aspects, but facilitate the lives of people who may have impaired motor function, (e.g. patients with amyotrophic lateral sclerosis (ALS) (Miao et al. 2020), spinal injuries, head trauma, rehabilitation, etc.) (Pfurtscheller et al. 1998; Craik et al. 2019). In recent years, many works have studied the development of brain–computer interfaces based on a non-invasive EEG device, from the standpoint of the design of both paradigms and algorithms for extracting features from EEG signals (Miao et al. 2020). In applications where the BCI is used by patients with some pathology, a BCI with maximum precision is essential for user safety. The difficulty with this application is that the classifier must make a decision in most cases based on a single trial (it is unable to obtain an averaged trial for the feature extraction). There are currently many research works aimed at finding better tools to identify an instruction and make a decision (Taniguchi et al. 2007; Li et al. 2016). A wide variety of methods for acquiring brain activity focus on non-invasive devices, such as magnetoencephalography (MEG), positron emission tomography (PET), and functional magnetic resonance imaging (fMRI). Even though electroencephalography (EEG) signals by nature have a poor signal–noise ratio, especially when acquired with wireless EEG devices, we conducted our research with this device due to the low cost, comfortability and ease of installation for a future user.

The first fact to consider is that the EEG signal patterns vary among users, and do not even remain constant over time for the same user. They always change in any patient state: when someone is at rest, performing some task (such as moving a hand) or is acoustically stimulated. Therefore, finding user responses to stimuli in real time, as well as determining the duration required for the signal recording to be classified properly, are essential to building an efficient BCI. All this time-dependent information and changes in the patient’s status extracted from the EEG signals have to be taken into account to design a classifier that improves decision-making in any application executed in real time. Real-time execution poses additional problems. The first one, as Netzer et al. defined in their paper, is a common issue in BCI processing, namely that the probability of success in the detection depends on capturing time (Netzer et al. 2020). If the BCI has short reaction time, the classification algorithm will perform worse, while if a better algorithm is used, the reaction time increases considerably. For example, they propose an algorithm based on coreset to calculate the common spatial patterns (CSP), keeping its representation with better results. The second problem is fatigue, as Roy et al. demonstrated. Mental fatigue can affect the signal intensity in EEG due to the distribution of these band power features, which are affected by the emotional and physical states (Roy et al. 2013). Complementarily, Myrden and Chau (2015) found a lower BCI performance caused by fatigue in their experiments involving a navigation game in which the participants had to report their level of fatigue, inter alia.

As a result of the complexity of the EEG interpretation, especially in a real-time application, the extraction and classification of information requires a complex methodology based on machine learning (ML) or deep learning (DL) techniques. Despite many existing classification algorithms in artificial intelligence-based applications, we could consider several of the most commonly applied algorithms. Lotte and others in their articles (Lotte et al. 2007, 2018) review the most frequently used classification algorithms in EEG-based applications, highlighting the following methods: discriminant analysis (DA), support vector machine (SVM), artificial neural network (ANN), k-nearest neighbor classification algorithm (KNN), decision tree learning (DTL), etc., which have yielded high accuracies, although not enough to apply these systems in an actual environment. The DA and SVM algorithms rely on a discriminant hyperplane search to identify classes, and they are prevalent in EEG-based BCI. In DA, it attempts to separate the data into different classes using a hyperplane, assuming that the data has a normal distribution and each class has an equal covariance matrix. The SVM searches for the hyperplane that maximizes the margin of the training data to the hyperplane. Algorithms based on artificial neural networks (ANN) assemble artificial neurons that are used to establish non-linear decision limits, but these are generic algorithms that could be negatively affected by noisy and non-stationary signals, such as those from an EEG. The KNN algorithm classifies new data according to the distance of the k nearest neighbors (Poorna et al. 2017). And the DTL is based on recursive partitioning of the instance space to predict the response (Wang et al. 2009; Ishfaque et al. 2013).

All these algorithms need a set of features extracted from the EEG signals as input data. However, due to the large number of features that can be obtained, the features needed for each classification method have to be pre-screened; that is, from each trace performed, a number of specific features is extracted (average power of a band in electrodes, connectivity, symmetry, etc.), losing the temporal information present in the brain when executing a task.

Against this backdrop, and given the importance of deep learning in a biomedical context (Zemouri et al. 2019), a classifier based on neural networks that can remember the information could provide a suitable alternative for an improved BCI. There is a type of recurrent neural network (RNN)—Long short-term memory (LSTM)—which is able to meet this requirement. The LSTM was developed by Hochreiter and Schmidhuber (1997). It is an RNN with the ability to remember information for long periods and forget unnecessary information, which means that the result outputs are based on a temporal pattern (Hochreiter and Schmidhuber 1997; Gers et al. 2000). It has been applied to EEG records, for example, to predict epileptic seizures (Tsiouris et al. 2018) or for emotion recognition (Alhagry et al. 2017; Li et al. 2018; Yang et al. 2018; Xing et al. 2019). Additionally, other researchers have applied this kind of RNN in similar contexts to ours: Kumar et al. (2019), who applied it in the BCI competition IV dataset 1 by combining common spatial pattern (CSP) with this type of network; Nagabushanam et al. (2020), who compare two specific neural network architectures (one with an LSTM layer) with the SVM; Li et al. (2016) defined an architecture and applied an LSTM net to classify motor imagery tasks in a public database, and something similar was applied by Wang et al. (2018) who classify motor imagery tasks of a public dataset by using their proposed framework based on LSTM, although they used a fixed time window.

The aims of this work are to study the effectiveness of using an LSTM with a low-cost wireless EEG device like Emotiv EPOC + working in real time, and to find the minimal optimal time window that maximizes the accuracy of the LSTM classifier. These objectives will lay the foundations for achieving a low-cost BCI that could be used by people with impaired motor function, for example, in a smart electric wheelchair with basic commands based on a simple code of opening and closing the user’s eyes. In order to determine the performance of using LSTM, we compared our results with popular ‘classical’ classification algorithms applied to EEG-based BCI, such as DA, SVM, KNN and DTL. In addition, the results obtained with the LSTM algorithms are compared with those obtained with a conventional recurrent neural network (RNN).

This section aims to define and explain the methods carried out in this research in an effort to highlight the relationship between the time window size and the robustness of a BCI, which follows the workflow shown in Fig. 1.

The flowchart for EEG signal pre-processing, feature extraction and classification is given in Fig. 1, which starts with the data acquisition using the Emotiv EPOC + to register the entire 14-channel EEG signal at a sample frequency of 128 Hz. Each data trial was divided into two seconds. A denoising of eye-blinks, a band-pass filter and a threshold were applied to remove any possible noise and artifacts. The flow continues by generating all the individual datasets with this processed signal, which consisted of different sequence sizes. The signal is then subject to a frequency analysis to obtain its power. Finally, a common train, validation and test processing is carried out to review the performance of the system.

The temporal variation of the EEG recordings in BCI applications suggests that classification algorithms that take their historical evolution into account (such as LSTM) would give better results. This section shows the feasibility of using LSTM in BCI systems with a low-cost, wireless EEG device (Emotiv EPOC+).

EEG recordings of 13 volunteers were taken using the protocols defined in the previous section. After pre-processing, the time evolution of the alpha-band power is computed using the wavelet transform. Figure 8 shows that the magnitude of the power of the alpha band in the visual area (sensors O1 and O2) is not constant over time and has larger values in the EC state than in the EO state, although, the EC state also has low values that could be confused with the EO state if the power is analyzed with a small time window.

A good BCI must react ideally to the user's action (command) instantly, or at least in the shortest time possible. If the system takes too long to react, the user could run into trouble. In fact, if the BCI’s response is not quick and accurate, a person with reduced mobility could spend considerable time and effort defining a simple action. Consequently, a long journey will take time and be exhausting.

Thus, to deal with these situations and have a more efficient BCI using an LSTM, it is important to define the shortest time window that generates optimal control of the system. Different time windows were used to create the datasets for the LSTM and find the best one.

The time evolution of the alpha-band power recorded in the O1 and O2 positions was used as a dataset to train and validate the classical algorithms and the LSTM network. The dataset was randomly divided to train (75%) and validate (25%) the algorithms. Four metrics, m, were calculated: accuracy (ACC), sensitivity (SEN), specificity (SPE) and Matthew correlation coefficient (MCC). The process of randomly dividing the dataset to train and validate the algorithms was repeated 20 times for each subject. Then, the mean value of the metric for each subject m_i is the average of the 20 metrics calculated, m_ij, where i indicates the subject and j identifies the iteration. Finally, the mean and standard deviation using the "n − 1" method (sd.s) of the set of subjects is calculated aswhere i = 1,…,I, with I = 13 and j = 1,…,J, with J = 20. Table 2 shows the mean result of the classical algorithms and Table 3 shows the means for the LSTM network with different time windows. In Table 3, we can see that the accuracy is between 0.7661 and 0.9214, while in Table 2, the accuracy of the classical algorithms is between 0.4854 and 0.6872.

The Friedman test yielded a statistically significant difference in the accuracy depending on the time windows, with a test statistic value F = 20.462 > χ²(2) = 7.815 and a significance level p-value = 0.000036. Then, to locate time windows with significant differences, post hoc studies with Wilcoxon signed-rank test were applied between different windows. The p-values obtained from these comparisons are shown in Table 4. p-values greater than 0.05 mean that there is no significant improvement to the LSTM algorithm when comparing the classifier accuracies using two different time windows. Therefore, as this table shows, there is no significant improvement for window sizes longer than 7 s.

Once the classifier algorithms were trained and validated, they were tested with new records. The paradigm used for the test recordings consisted of three EC states, each of which is t seconds long, with an EO state in between. Figure 9 shows the classification output for a user with different algorithms. The signals at the top (Fig. 9a) are EEG power records from the O1 and O2 sensors. Figure 9b shows the results using the best classifiers according to Table 2, and an LSTM classifier with a time window of 7 s.

Figure 10 represents the classification of the same recordings as in Fig. 9 with the LSTM and different time windows (w): 2, 4 and 7 s. Also, long EC states (t) were defined with different durations: 7, 15 and 20 s.

In addition, to test the network architecture, the number of LSTM layers has been increased (using eight neurons in each one) and their accuracy has been evaluated, yielding the result shown in Fig. 11 with the different time windows used. Note the elbow that is formed in the time windows between 5 and 7 s. As for the results, when applying several layers (2, 3 and 4), the evolution of the accuracy is similar, while using a single layer has lower values.

In this study, we have chosen a low-cost wireless EEG device, namely the Emotiv EPOC + , because it is wireless and easy to fit, which facilitates its use in BCI applications, for example in situations where people with reduced mobility wish to interact with their environment through computers. In addition, because it is a low-cost device, it can be used by a greater number of potential customers who will have less purchasing power. However, a low-cost wireless EEG device has a number of disadvantages compared to clinical devices in terms of the quality of the acquired signal. Therefore, in this study, we looked at the possibility of applying an LSTM network to the EEG recorded by a low-cost wireless EEG device and the need to choose the appropriate time window to improve accuracy. We also wanted to show that in this case, an LSTM network can yield better results than classical classification algorithms.

In order to study and compare the selected classification algorithms, the brain activity of 13 volunteers was measured while keeping their eyes closed or open, depending on the external stimuli received. Then, the purpose of the classifier is to decide whether the subject has their eyes closed or open. As was shown at the beginning of the results section, there is a clear contrast in brain signals when a user is in the EO or EC state, which is present in the alpha-band power signal, Fig. 8. The results obtained by Gopika Gopan et al. (2017) are similar, demonstrating a clear difference between these two states with an accuracy of 77.92%. But when comparing the classic classification algorithms, Table 2, with the LSTM networks, Table 3, we see that the latter are more accurate, yielding an accuracy that is at least 8% higher than standard algorithms. Although, Fig. 8 shows a clear difference between EEG recordings in the eyes open and eyes closed states, if we analyze shorter time intervals, such as one second or less, the difference is not so obvious. Since the power does not remain constant, a low power may be measured in both the EO and EC states. In this case, traditional classification algorithms cannot discern the difference between the two states. Therefore, when the classification algorithms are trained directly with the values of the alpha-band power signal that change with time, their accuracy is worse and the records must be processed to extract other characteristics that can improve the classification over a longer time window. When we focus on the sensitivity, specificity and MCC of these classic algorithms, Table 2, we see this behavior reflected: the accuracy average is 59.71% and the sensitivity (ratio of correct positives and all the elements classified as positive) and specificity (negative elements in relation to all elements classified as negative) are 52.38% and 67.04%, respectively. But in the case of MCC, we see that most of these algorithms have a value very close to 0, which means that the classification of these algorithms is almost random. Furthermore, it is observed that the application of the RNN, which works with previous information of the data, does not depart from the accuracies obtained with the rest of the algorithms. However, a classifier such as the LSTM, which takes into account the information in the previous time instants, produces better results. Table 3 shows that the accuracy (76.61–92.14%), sensitivity (70.75–91.62%) and specificity (82.47–93.63%) of LSTM networks are higher than the classical algorithms. In addition, the MCC is distributed between the range of 0.53 and 0.85, which means that its prediction is quite good (a perfect prediction is when MCC is equal to 1).

An important factor in properly training an LSTM network is choosing the duration (time window) of the records that are passed to the network. Other author, like Kumar et al. (2019), used a single time window that was 25% of the trial duration and with an overlap of approximately 5%. However, in this paper, the time window size is changed to compare its relationship with the algorithm’s performance.

The results show that LSTM classifiers generate a better accuracy with longer windows—from a mean accuracy of 76.61% with a 1-s window size to 90.94% with 7 s. From this point on, the accuracy values are around 91%. In our case, no pre-selection of the signal was done when applying the proposed methodology. Other authors do select the best segments to build their dataset, which is not applicable in real-time systems; for example, Piątek et al. (2018) obtained an accuracy of over 96% with different classification algorithms with a time window of 10 s.

From the results presented in Tables 3 and 4, we concluded that after 7 s there is no significant improvement in the classification.

Finally, EEG records presenting a more realistic situation with EO and EC states were used to test how the different classifiers will work in a real BCI (Fig. 9a). Figure 9b shows the outcomes of the Coarse DTL, SVM Fine Gaussian, Coarse KNN algorithms and RNN with a delay of 1 s, which had the best accuracy in Table 2. Although the user is in a fixed state (EC or EO), the responses of these four classifiers change continuously between EC and EO. In contrast, the LSTM results are more stable and closer to the user’s behavior. It should be noted that the RNN applies a dynamic temporal behavior as well, but does not achieve the constancy of actions present in the LSTM.

However, there is one aspect to highlight in Fig. 10, where different lengths of the EC state (t) are used in the protocol and the applied LSTM uses different time windows (w). Although the optimal result obtained in Table 3 is 7 s, we see how lengthening the window size affects the EO transition by adding a substantial delay, which could decrease the accuracy. It is necessary that the sequences to be classified in the LSTM be formed by exclusive characteristics of each state in order to be correctly classified. If they are formed by several classes, delays would be created because until the new state is predominant, the result of the classification will not change. That means that in a real system, it is necessary to establish a compromise between accuracy and response times since an LSTM-based BCI user has to know the duration of the time window over which to give his commands so they can be detected.

It should be noted that the network used and applied in the real context has a single LSTM layer which obtains higher accuracy than the classical algorithms. The results can be improved by adding more layers, as seen in Fig. 11.

In general, the algorithms applied can be considered, on the one hand, as classical algorithms, such as ML algorithms and, on the other hand, as DL algorithms, such as LSTM. In both cases, the quality and reliability of the data are essential to ensure efficient performance, even in the case involved in this research. In general, DL techniques usually require a large dataset and have a higher computational cost due to the complexity of the operations they perform. In this case study, the training was not significantly different, and the results were comparable, with the LSTM results standing out thanks to the sequential processing performed by this layer.

In a BCI context, although there is an extensive literature on EEG-based BCI with LSTM networks, there are not enough studies focused on the time window parameter, which is essential for its success. Our results highlight different aspects that we must take into account to implement an LSTM network in a BCI.

In our case study, the task of distinguishing the EO and EC states is complicated due to the changing evolution of the EEG signal. Most of the peaks in the alpha-band power in the EO state are low compared to those in the EC state. However, in an EC state, some values of power may be similar to those in the EO state. Therefore, this fact makes the classifier require extra information to differentiate the two states, since the discrete values of power do not have enough information to differentiate the states correctly.

There are different approaches to solve this problem; for example, using a threshold dependent on the maximum value of the peak powers detected in each state. When several consecutive power peaks of the EEG signal are detected and exceed this threshold in a predefined period, it is considered to be in the EC state, and if not, in the EO state. However, this method involves making several assumptions such as: assuming the behavior of the power, determining how often the power peaks appear, defining a threshold, etc. Therefore, a detailed study of each user’s EEG is necessary. In this work, we demonstrate how an LSTM neural network can elegantly solve this problem.

The outcomes obtained with the LSTM have demonstrated that it is able to obtain a temporal pattern in the EEG signals, as other authors have demonstrated (Li et al. 2016; Wang et al. 2018). Based on our results, the importance of the time window in an LSTM is vital because there is a relationship between the time window sequence used and the accuracy of the classifier. In addition, the LSTM models proposed yielded better outcomes than the classical algorithms, which would need a longer pre-processing of the data to obtain the same level of performance. On the other hand, unlike an RNN network, LSTM can store information over long periods of time, improving the classification results. In summary, there is a relationship between the time window and the accuracy of the LSTM classifier which determines the efficacy and accuracy of a BCI system.

And finally, as a generic conclusion, despite the good results obtained with the EEG, there are other methods, such as EMG (Usman et al. 2021) or video eye tracking (Ibrahim et al. 2021) that could obtain better results in the proposed task. But we must not forget the potential of the system proposed here, which can be applied to other tasks such as providing feedback to aid with motor diseases (Miao et al. 2020; Pimenta et al. 2021), neuromarketing (Gers et al. 2000) or the classification of more elaborate brain tasks (Pfurtscheller et al. 1998; Kumar et al. 2019; Netzer et al. 2020) (to which the same study conducted in this work can be applied).