A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers

Figure 1 shows the general paradigm of brain signal decoding, which receives brain signals and produces the user's latent informatics. The workflow includes several key components: brain signal collection, signal preprocessing, feature extraction, classification, and data analysis. The brain signals are collected from humans and sent to the preprocessing component for denoising and enhancement. Then, the discriminating features are extracted from the processed signals and sent to the classifier for further analysis.

Zoom In Zoom Out Reset image size

The collection methods differ from signal to signal. For example, electroencephalogram (EEG) signals measure the voltage fluctuation resulting from ionic current within the neurons of the brain. Collecting EEG signals requires placing a serievs of electrodes on the scalp of the human head to record the electrical activity of the brain. Since the ionic current generated within the brain is measured at the scalp, obstacles (e.g. skull) greatly decrease the signal quality—the fidelity of the collected EEG signals, measured as signal-to-noise ratio (SNR), is only approximately 5% of that of original brain signals [1]. The collection methods of more non-invasive signals can be found in appendix A.

Therefore, brain signals are usually preprocessed before feature extraction to increase the SNR. The preprocessing component contains multiple steps such as signal cleaning (smoothing the noisy signals or resolving the inconsistencies), signal normalization (normalizing each channel of the signals along time-axis), signal enhancement (removing direct current), and signal reduction (presenting a reduced representation of the signal).

Feature extraction refers to the process of extracting discriminating features from the input signals through domain knowledge. Traditional features are extracted from time-domain (e.g. variance, mean value, kurtosis), frequency-domain (e.g. fast Fourier transform), and time-frequency domains (e.g. discrete wavelet transform). They will enrich distinguishable information regarding user intention. Feature extraction is highly dependent on the domain knowledge. For example, neuroscience knowledge is required to extract distinctive features from motor imagery EEG signals. Manual feature extraction is also time-consuming and difficult. Recently, deep learning provides a better option to automatically extract distinguishable features.

The classification component refers to the machine learning algorithms that classify the extracted features into logical control signals recognizable by external devices. Deep learning algorithms are shown to be more powerful than traditional classifiers [2–4].

The classification results reflect the user's psychological or physical status and can inspire further information analysis. This is widely used in real-world applications such as neurological disorder diagnosis, emotion measuring, and driving fatigue detection. Appropriate treatment, therapy, and precaution could be conducted based on the analysis results.

In specific, the system is called a brain-computer interface (BCI) while the decoded brain signals are converted into digital commands to control the smart equipment and react with the user (dashed lines in figure 1). BCI ⁷ systems interpret the human brain patterns into messages or commands to communicate with the outer world [5]. BCI is generally a closed-loop system with an external device (e.g. wheelchair and robotic arm), which can directly serve the user. In contrast, brain signal analysis does not require a specific device as long as the analysis results can benefit society and individuals.

In this survey, we summarize the state-of-the-art studies which adopt deep learning models: (1) for feature extraction only; (2) for classification only; (3) for both feature extraction and classification. The details will be introduced in section 4. Brain signal underpins many novel applications that are important to people's daily life. For example, the brain signal-based user identification system, with high fake-resistance, allows normal people to enjoy enhanced entertainment and security [6]; for people with psychological/physical deceases or disabilities, brain signals enable them to control smart device such as wheelchairs, home appliances, and robots. We present a wide range of deep learning-based brain signal applications in section 5.

Although traditional brain signal system has made tremendous progress [7, 8], it still faces significant challenges. First, brain signals are easily corrupted by various biological (e.g. eye blinks, muscle artifacts, fatigue, and the concentration level) and environmental artifacts (e.g. noises) [7]. Therefore, it is crucial to distill informative data from corrupted brain signals and build a robust system that works in different situations. Second, it faces the low SNR of non-stationary electrophysiological brain signals [9]. The low SNR cannot be easily addressed by traditional preprocessing or feature extraction methods due to the time complexity of those method and the risk of information loss [10]. Third, feature extraction highly depends on human expertise in the specific domain. For example, it requires the basic biological knowledge to investigate sleep state through electroencephalogram (EEG) signals. Human experience may help on certain aspects but fall insufficient in more general circumstances. An automatic feature extraction method is highly desirable. Moreover, most existing machine learning research focuses on static data and therefore, cannot classify rapidly changing brain signals accurately. For instance, the state-of-the-art classification accuracy for multi-class motor imagery EEG is generally below 80% [11]. It requires novel learning methods to deal with dynamical data streams in brain signal systems.

Until now, deep learning has been applied extensively in brain signal applications and shown success in addressing the above challenges [12, 13]. Deep learning has two advantages. First, it works directly on raw brain signals, thus avoiding the time-consuming preprocessing and feature extraction. Second, deep neural networks can capture both representative high-level features and latent dependencies through deep structures.

We conduct this survey for three reasons. First, there lacks a comprehensive survey on the non-invasive brain signals. Table 1 shows a summary of the existing survey on brain signals. As our best knowledge, the limited existing surveys [5, 7, 8, 11, 14, 15, 24] only focus on partial EEG signals. For example, Lotte et al [11] and Wang et al [18] focus on general EEG without analyzing EEG subtypes; Cecotti et al [28] focus on event-related potentials (ERPs); Haseer et al [29] focus on functional near-infrared spectroscopy (fNIRS); Mason et al [15] brief the neurological phenomenons like event-related desynchronization (ERD), P300, SSVEP, visual evoked potentials (VEPs), auditory evoked potentials (AEPs) but have not organized them systematically; Abdulkader et al [7] present a topology of brain signals but have not mentioned spontaneous EEG and rapid serial visual presentation (RSVP); Lotte et al [5] have not considered ERD and RSVP; VEP should be a subtype of ERP in [8]. Ahn et al [21] review the performance variation in MI-EEG based BCI systems. Roy et al [17] list some deep learning-based EEG studies but present little technical inspirations and have less analysis on deep learning algorithms, they also failed to investigate other non-invasive brain signals beyond EEG. In particular, compared to [17], this work provides a better introduction of deep learning including the basic concepts, algorithms, and popular models (section 3 and appendix B). Moreover, this paper discusses the high-level guidelines in brain signal analysis in terms of the brain signal paradigms, the suitable deep learning frameworks and the promising real-world applications (section 6).

Second, few research has investigated the association between deep learning ([30, 31]) and brain signals ([5, 7, 8, 11, 15, 32]). To the best of our knowledge, this paper is in the first batch of comprehensive survey on recent advances on deep learning-based brain signals. We also point out frontiers and promising directions in this area.

Lastly, the existing surveys focus on specific areas or applications and lack an overview of broad scenarios. For example, Litjens et al [16] summarize several deep neural network concepts aiming at medical image analysis; Soekadar et al [20] review the BCI systems and machine learning methods for stroke-related motor paralysis based on sensori-motor rhythms; Vieira et al [33] investigate the application of brain signals on the neurological disorder and psychiatric.

This survey can mainly benefit: (1) the researchers with computer science background who are interested in the brain signal research; (2) the biomedical/medical/neuroscience experts who want to adopt deep learning techniques to solve problems in basic science.

To our best knowledge, this survey is the first comprehensive survey of the recent advances and frontiers of deep learning-based brain signal analysis. To this end, we have summarized over 200 contributions, most of which were published in the last five years. We make several key contributions in this survey:

• We review brain signals and deep learning techniques to help readers gain a comprehensive understanding of this area of research.
• We discuss the popular deep learning techniques and state-of-the-art models for brain signals, providing practical guidelines for choosing the suitable deep learning models given a specific subtype of signal.
• We review the applications of deep learning-based brain signal analysis and highlight some promising topics for future research.

The rest of this survey is structured as followed. Section 2 briefly introduces an taxonomy of brain signals in order to help the reader build a big picture in this field. Section 3 overviews the commonly used deep learning models to present the basic knowledge for researchers (e.g. neurological and biomedical scholars ) who are not familiar with deep learning. Section 4 presents the state-of-the-art deep learning techniques for brain signals and section 5 discusses the applications related to brain signals. Section 6 provides a detailed analysis and gives guidelines for choosing appropriate deep learning models based on the specific brain signal. Section 7 points out the opening challenges and future directions. Finally, section 8 gives the concluding remarks. We provide a tutorial ⁸ on how to use popular deep learning models to analyze brain signals.

Due to the advantages of high portability and low price, EEG signals have attracted much attention. Most of the latest publications on non-invasive brain signals are related to EEG. In this section, we summarize two aspects of EEG signals: spontaneous EEG and EPs. As implied by the name, the former are spontaneous and the latter requires outside stimuli.

4.1.1. Spontaneous EEG

4.1.2. Evoked potential

Up to now, only a few of researchers paid attention on deep learning-based fNIRS. Naseer et al [38] analyzed the difference between two mental tasks (mental arithmetic and rest) based on fNIRS signals. The authors manually extracted six features from the prefrontal cortex fNIRS and compared six different classifiers. The results demonstrated that the MLP with the accuracy of 96.3% outperformed all the traditional classifiers, including SVM, KNN, naive Bayes, etc Huve et al [198] classified the fNIRS signals, which were collected from the subjects during three mental states, including substractions, word generation, and rest. The employed MLP model achieved the accuracy of 66.48% based on the hand-crafted features (e.g. the concentration of OxyHb/DeoxyHb). After that, the authors study the mobile robot control through fNIRS signals and got the binary classification accuracy of 82% (offline) and 66% (online) [199]. Chiarelli et al [71] exploited the combination of fNIRS and EEG for left/right MI EEG classification. Sixteen features extracted from fNIRS signals (eight from OxyHb and eight from DeoxyHb) were fed into an MLP classifier with four hidden layers.

On the other hand, Hiroyasu et al [201] attempted to detect the gender of the subject through their fNIRS signals. The authors employed a denoising D-AE with three hidden layers to extract distinctive features to be fed into an MLP classifier for gender detection. The model was evaluated over a local dataset and gained the average accuracy of 81%. In this study, the authors also pointed out that, compared with Positron emission tomography (PET) and fMRI, fNIRS has higher time resolution and more affordable [201].

Recently, several deep learning methods have been applied to fMRI analysis, especially on the diagnosis of cognitive impairment [14, 33].

(1) Discriminative models. Among the discriminative models, CNN is a promising model to analyze fMRI [206]. For example, Havaei et al built a segmentation approach for brain tumor based on fMRI with a novel CNN algorithm which can capture both the global features and the local features simultaneously [205]. The convolutional filters have different size. Thus, the small-size and large-size filter could exploit the local and global features, independently. Sarraf et al [207, 225] applied deep CNN to recognize Alzheimer's Disease based on fMRI and MRI data. Morenolopez et al [226] employed a CNN model to deal with fMRI of brain tumor patients for three-class recognition (normal, edema, or active tumor). The model was evaluated over BRATS dataset and obtained the F1 score of 88%. Hosseini et al [117] employed CNN for feature extraction. The extracted features were classified by SVM for the detection of an epileptic seizure.

Furthermore, Li et al proposed a data completion method based on CNN. In particular, utilizing the information from fMRI data to complete PET, then train the classifier based on both fMRI and PET [208]. In the model, the input data of the proposed CNN is the fMRI patch, and the output is a PET patch. There are two convolutional layers with ten filters mapping the fMRI to PET. The experiments illustrated that the classifier trained by the combination of fMRI and PET (92.87%) outperformed the one trained by solo fMRI (91.92%) Moreover, Koyamada et al used a non-linear MLP to extract common features from different subjects. The model is evaluated over a dataset from the Human Connectome Project [202].

(2) Representative models. A wide range of publications demonstrated the effectiveness of representative models in recognition of fMRI data. Hu et al [216] used demonstrated that deep learning outperforms other machine learning methods in the diagnosis of neurological disorders such as Alzheimer's disease. Firstly, the fMRI images were converted to a matrix to represent the activity of 90 brain regions. Secondly, a correlation matrix is obtained by calculating the correlation between each pair of brain regions to represent the functional connectivity between different brain regions. Furthermore, a targeted AE is built to classify the correlation matrix, which is sensitive to AD. The proposed approach achieved an accuracy of 87.5%. Plis et al [211] employed a DBN-RBM with three RBM components to extract the distinctive features from ICA processed fMRI and finally achieved an average F1 measure of above 90% over four public datasets. Suk et al compared the effectiveness of DBN-RBM and DBN-AE on Alzheimer's disease detection and the experimental results showed that the former obtained the accuracy of 95.4%, which is slightly lower than the latter (97.9%) [210]. Suk et al [209] applied a D-AE model to extract latent features from the resting-state fMRI data on the diagnosis of MCI. The latent features are fed into a SVM classifier which achieved the accuracy of 72.58%. Ortiz et al [212] proposed a multi-view DBN-RBM to receives the information of MRI and PET simultaneously. The learned representations were sent to several simple SVM classifiers which were ensembled to form a high-level stronger classifier by voting.

(3) Generative models. The reconstruction of natural image (e.g. fMRI) has been attracted lots of attention [88, 203, 214]. Seeliger et al [213] proposed a deep convolutional GAN (DCGAN) for reconstructing visual stimuli from fMRI, which aimed at training a generator to create an image similar to the visual stimuli. The generator contains four convolutional layers in order to convert the input fMRI to a natural image. Han et al [214] focused on the generation of synthetic multi-sequence fMRI using GAN. The generated image can be used for data augmentation for better diagnostic accuracy or physician training to help better understand various diseases. The authors applied the existing DCGAN [227] and WGAN [228] and found that the former works better. Shen et al [203] presented another image recovery approach by minimizing the distance between the real image and the image generated based on real fMRI.

Garg et al [217] worked on the refining of MEG signals by removing the artifacts like eye-blinks and cardiac activity. The MEG singles were decomposed by ICA first and then classified by a 1-D CNN model. At last, the proposed approach achieved the sensitivity of 85% and specificity of 97% over a local dataset. Hasasneh et al [219] also focused on artifacts detection (cardiac and ocular artifacts). The proposed approach uses CNN to capture temporal features and MLP to extract spatial information. Shu et al [218] employed a sparse AE to learn the latent dependencies of MEG signals in the task of single word decoding. The results demonstrated that the proposed approach is advantageous for some subjects, although it did not produce an overall increase in decoding accuracy. Cichy et al [204] applied a CNN model to recognize visual object based on MEG and fMRI signals.

In the health care area, the deep learning-based brain signal systems mainly works on the detection and diagnosis of mental diseases such as sleep disorders, Alzheimer's Disease, epileptic seizure, and other disorders. In the first place, for the sleep disorder detection, most studies are focused on the sleep stage detection based on sleep spontaneous EEG. In this situation, the researchers do not need to recruit patients with sleep disorder because the sleep EEG signals can be easily collected from healthy individuals. In terms of the algorithm, it can be observed from table 5 that the DBN-RBM and CNN are widely adopted for feature selection and classification. Ruffini et al [111] walk one step further by detecting the RBD, which may cause neurodegenerative diseases such as PD. They achieved an average accuracy of 85% in recognition of the RBD from healthy controls.

Moreover, fMRI is widely applied in the diagnosis of Alzheimer's Disease. By taking advantage of the high spatial resolution of fMRI, the diagnosis achieved the accuracy of above 90% in several studies. Another reason that contributes to competitive performance is the binary classification scenario. Apart from that, there are several publications diagnose the AD based on spontaneous EEG [115, 126].

Besides, the diagnosis of epileptic seizure attracted much attention. The seizure detection mainly based on spontaneous EEG. The popular deep learning models in this scenario contain the independent CNN and RNN, along with hybrid models combined RNN and CNN. Some models integrated the deep learning models for feature extraction and traditional classifier for detection [125, 127]. For example, Yuan et al [121] applied a D-AE in feature extraction followed by SVM for seizure diagnosis. Ullah et al [112] adopted the voting for post-processing, which proposed several different CNN classifiers and predicted the final result by voting.

Furthermore, there are a lot of other healthcare issues can be solved by brain signal research. The cardiac artifacts in MEG can be automatically detected by deep learning models [217, 219]. Several modified CNN structures are proposed to detect brain tumor based on fMRI from the public BRATS dataset [205, 206]. Researchers have demonstrated the effectiveness of deep learning models in the detection of a wide number of mental diseases such as depression [113], interictal epileptic discharge (IED) [229], schizophrenia [211], CJD [123], and MCI [209].

The smart environment is a promising application scenario for brain signals in the future. With the development of Internet of things, an increasing number smart environment can be connected to brain signals. For example, the assisting robot can be used in smart home [2, 65], in which the robot can be controlled by brain signals of the individuals. Moreover, Behncke et al [139] and Huve et al [199] investigated the robot control problem based on the visual stimulated spontaneous EEG and fNIRS signals. The brain signal controlled exoskeleton could help the disabilities who damaged the motor system in sub-limb in walking and daily activities [191]. In the future, the research on brain-controlled appliances may be beneficial to the elders or disabilities in smart home and smart hospital.

One of the biggest advantages of brain signals, compared to other human–machine interface techniques, is that brain signal enables the patient who lost most motor abilities like speaking to communicate with the outer world. The deep learning technology improved the efficiency of brain signal based communications. One typical diagram which enables individual typing without any motor system is P300 speller, which can convert the user's intent into text [162]. The powerful deep learning models empower the brain signal systems to recognize the P300 segment from the non-P300 segment while the former contains the communication information of the user [166]. In a higher level, the representative deep learning models can help to detect what character the user is focusing on and print it on the screen to chat with others [164, 166, 170]. Additionally, Zhang et al [10] proposed a hybrid model that combined RNN, CNN, and AE to extract the informative features from MI EEG to recognize what letter the user wants to speak.

Brain signals can be used in security scenarios such as identification (or recognition) and authentication (or verification). The former conducts multi-class classification to recognize a person's identity [6]. The latter conducts binary classification to decide whether a person is authorized [61].

The majority of the existing biometric identification/authentication systems rely on individuals' intrinsic physiological features such as face, iris, retina, voice, and fingerprint [6]. They are vulnerable to various attacks based on anti-surveillance prosthetic masks, contact lenses, vocoder, and fingerprint films. EEG-based biometric person identification is a promising alternative given its highly resilient to spoofing attacks—individual's EEG signals are virtually impossible for an imposter to mimic. Koike et al [161] have adopted deep neural networks to identify the user's ID based on the VEP signals; Mao et al [180] applied CNN for person identification based on RSVP signals; Zhang et al [6] proposed an attention-based LSTM model and evaluated it over both public and local datasets. EEG signals are also combined with gait information in a hybrid deep learning model for a dual-authentication system [61].

Affective states of a user provide critical information for many applications such as personalized information (e.g. multimedia content) retrieval or intelligent human–computer interface design [99]. Recent research illustrated that deep learning models can enhance the performance in affective computing. The most widely used circumplex model believe the emotions are distributed in two dimensions: arousal and valence. The arousal refers to the intensity of the emotional stimuli or how strong is the emotion. The valence refers to the relationship within the person who experiences the emotion. In some other models, the dominance and liking dimensions are deployed.

Some research [89–91] attempts to classify users' emotional state into two (positive/negative) or three categories (positive, neutral, and negative) based on EEG signals using deep learning algorithms such as CNN and its variants [87]. DBN-RBM is the most representative deep learning model to discover the concealed features from emotional spontaneous EEG [96, 99]. Xu et al [99] applied DBN-RBM as feature extractors to classify affective states based on EEG.

Further, some researchers aim to recognize the positive/negative state of each specific emotional dimension. For example, Yin et al [107] employed an ensemble classifier of AE in order to recognize the user's affection. Each AE uses three hidden layers to filter out noises and to derive stable physiological feature representations. The proposed model was evaluated over the benchmark, DEAP, and achieved the arousal of 77.19% and valence of 76.17%.

Vehicle drivers' ability to keep alert and maintain optimal performance will dramatically affect the traffic safety [145]. EEG signals have proven useful in evaluating the human's cognitive state in different context. Generally, a driver is regarded as in an alert state if the reaction time is lower than 0.7 s and in fatigue state if it is higher than 2.1 s. Hajinoroozi et al [153] considered the detection of driver's fatigue from EEG signals by discovering the distinct features. They explored an approach based on DBN for dimension reduction.

Detecting driver fatigue is crucial because the drowsiness of the driver may lead to disaster. Driver fatigue detection is feasible in practice. In the hardware aspect, the collection equipment of EEG singles is off-the-shelf and portable enough to be used in a car. Moreover, the price of an EEG headset is affordable for most people. In the algorithm aspect, deep learning models have enhanced the performance of fatigue detection. As we summarized, the EEG based driving drowsiness can be recognized with high accuracy (82%–95%).

Future scope of drive fatigue detection is in the self-driving scenario. As we know, in the most situation of self-driving (e.g. Automation level 3 ¹¹ ), the human driver is expected to respond appropriately to a request to intervene, which indicates that the driver should keep alert state. Therefore, we believe the application of brain signal-based drive fatigue detection will benefit the development of the self-driving car.

The EEG oscillations can be used to measure the mental workload level, which can sustain decision making and strategy development in the context of human–machine interaction [150]. Additionally, the appropriate mental workload is essential for maintaining human health and preventing accidents. For example, the abnormal mental workload of the human operator may result in performance degradation which could cause catastrophic accidents [231]. Evaluation of operator Mental Workload levels via ongoing EEG is quite promising in human–machine collaborative task environment to alarm the temporal operator performance degradation.

Several researchers have been paid attention to this topic. The mental workload can be measured from fNIRS signals or spontaneous EEG. Naseer et al adopted a MLP algorithm for fNIRS-based binary mental task level classification (mental arithmetic and rest) [38]. The experiment results showed that the MLP outperformed the traditional classifiers like SVM, KNN, and achieved the highest accuracy of 96.3%. Bashivan et al [159] presented a statistical approach, a DBN model, for the recognition of mental workload level based on single-trial EEG. Before the DBN, the authors manually extracted the wavelet entropy and band-specific power from three frequency bands (theta, alpha, and beta). At last, the experiments demonstrated the recognition of mental workload achieved an overall accuracy of 92%. Zhang et al [156] investigate the mental load measurement across multiple mental tasks via a recurrent-convolutional framework. The model simultaneously learns EEG features from the spatial, spectral, and temporal dimensions, which results in the accuracy of 88.9% in binary classification (high/low workload levels).

There are plenty of interesting scenarios beyond the above where deep learning-based brain signals can apply, such as recommender system [135] and emergency braking [144]. One possible topic is the recognition of a visual object, which may be used in guilty knowledge test [195] and concealed information test [168]. The neurons of the participant will produce a pulse when he/she suddenly watch a similar object. Based on the theory, the visual target recognition is mainly used RSVP signals. Cecotti et al [177] aimed to build a common model for target recognition, which can work for various subjects instead of a specific subject.

Besides, researchers have investigated to distinguish the subject's gender by the fNIRS [201] and spontaneous EEG [146]. Hiriyasu et al [201] adopted deep learning to recognize the gender of the subject based on the cerebral blood flow. The experiment results suggested that the cerebral blood flow changes in different ways for male and female. Putten et al [146] tried to discover the sex-specific information from the brain rhythms and adopted a CNN model to recognize the participant's gender. This paper illustrated that fast beta activity (20–25 Hz) is one of the most distinctive attributes.

We have extensively explored the benchmark datasets usable for deep learning-based brain signals (table 6). We provide a bunch of public datasets with download links, which cover most brain signal types. In particular, BCI competition IV (BCI-C IV) contains five datasets via the same link. For better understanding, we present the number of subjects, the number of class (how many categories), sampling rate, and the number of channels of each dataset. In the '# Channel' column, the default channel is for EEG signals. Some datasets contain more biometric signals (e.g. ECG), but we only list the channels related to brain signals.

Among the non-invasive signals, the studies on EEG is far more than the sum of all the other brain signal paradigms (fNIRS, fMRI, and MEG). Furthermore, there are about 70% of the EEG papers pay attention to the spontaneous EEG (133 publications). For better understanding, we split the spontaneous EEG into several aspects: the sleep, the motor imagery, the emotional, the mental disease, the data augmentation, and others.

First, the classification of the sleep EEG mainly depends on the discriminative and the hybrid models. Among the nineteen studies about sleep stage classification, there are six employed CNN and the modified CNN models independently while two papers adopted RNN models. There are three hybrid models built on the combination of CNN and RNN.

Second, in terms of the research on MI EEG (30 publications), the independent CNN and CNN-based hybrid models are widely used. As for the representative models, DBN-RBM is often applied to capture the latent features from the MI EEG signals.

Third, there are 25 publications related to spontaneous emotional EEG. More than half of them employed representative models (such as D-AE, D-RBM, especially DBN-RBM) for unsupervised feature learning. The most typical state recognition works recognize the user's emotion as positive, neutral, or negative. Some researchers take a further step to classify the valence, and the arouse rate, which is more complex and challenging.

Fourth, the research on mental disease diagnosis is promising and attracting. The majority of the related research focuses on the detection of epileptic seizure and Alzheimer's Disease. Since the detection is a binary classification problem which is rather easier than multi-class classification, many studies can achieve a high accuracy like above 90%. In this area, the standard CNN model and the D-AE are prevalent. One possible reason is that CNN and AE are the most well-known and effective deep learning models for classification and dimensionality reduction.

Fifth, several publications pay attention to the GAN based data augmentation. At last, about 30 studies are investigating other spontaneous EEG such as driving fatigue, audio/visual stimuli impact, cognitive/mental load, and eye state detection. These studies extensively apply standard CNN models and variants.

Moreover, apart from spontaneous EEG, EPs also attracted much attention. On the one hand, in ERP, VEP and the subcategory RSVP has drawn lots of investigations because visual stimuli, compared to other stimuli, is easier to be conducted and more applicable in the real world (e.g. P300 speller can be used for brain typing). For VEP (21 publications), there are 11 studies applied discriminative models, and six works adopted hybrid models. In terms of RSVP, the sole CNN dominates the algorithms. Apart from them, five papers focused on the analysis of AEP signals. On the other hand, among the steady-state related researches, only SSVEP has been studied by deep learning models. Most of them only applied discriminative models on the recognition of the target image.

Furthermore, beyond the diverse EEG diagrams, a wide range of papers paid attention to fNIRS and fMRI. The fNIRS images are rarely studied by deep learning, and the major studies just employed the simple MLP models. We believe more attention should be paid to the research on fNIRS for the high portability and low cost. As for the fMRI, 23 papers proposed deep learning models to the classification. The CNN model is widely used for its outstanding performance in feature learning from images. There are also several papers interested in image reconstruction based on fMRI signals. One reason why fMRI is so hot is that several public datasets are available on the Internet, although the fMRI equipment is expensive. The MEG signals are mainly used in the medical area, which is insensitive to the deep learning algorithm. Thus, we only found very few studies on MEG. The sparse AE and CNN algorithms have a positive influence on the feature refining and classification of MEG.

Our investigation shows that discriminative models are most frequent in the summarized publications. This is reasonable at a high level because a large proportion of brain signal issues can be regarded as a classification problem. Another observation is that CNN and its variants are adopted in more than 70% of the discriminative models, for which we provide reasons as follows.

First, the design of CNN is powerful enough to extract the latent discriminative features and spatial dependencies from the EEG signals for classification. As a result, CNN structures are adopted for classification in some studies while adopted for feature extraction in some other studies.

Second, CNN has been achieved great success in some research areas (e.g. computer vision), which makes it extremely famous and feasible (public codes). Thus, the brain signal researchers have more chance to understand and apply CNN on their works.

Third, some brain signal diagrams (e.g. fMRI) are naturally formed as two-dimension images that are conducive to be processed by CNN. Meanwhile, other 1D signals (e.g. EEG) could be converted into 2D images for further analysis by CNN. Here, we provide several methods converting 1D EEG signals (with multiple channels) to the 2D matrix: (1) convert each time-point ²⁵ to a 2D image; (2) convert a segment into a 2D matrix. In the first situation, suppose we have 32 channels, and we can collect 32 elements (each element corresponding to a channel) at each time-point. As described in [89], the collected 32 elements could be converted into a 2D image based on the spatial position. In the second situation, suppose we have 32 channels, and the segment contains 100 time-points. The collected data can be arranged as a matrix with the shape of [32, 100] where each row and column refers to a specific channel and time-point, respectively.

Fourth, there are a lot of variants of CNN which are suitable for a wide range of brain signal scenarios. For example, the single-channel EEG signals can be processed by 1D CNN. In terms of RNN, only about 20% of discriminative model-based papers adopted RNN, which is much less than we expected since RNN has demonstrated powerful in temporal feature learning. One possible reason for this phenomena is that processing a long sequence by RNN is time-consuming and the EEG signals are generally formed as a long sequence. For example, the sleep signals are usually sliced into segments with 30 s, which has 3000 time-points under 100 Hz sampling rate. For a sequence with 3000 elements, through our preliminary experiments, RNN takes more than 20 folds training time than CNN. Moreover, MLP is not popular due to its inferior effectiveness (e.g. non-linear ability) to the other algorithms its simple deep learning architecture.

As for representative models, DBN, especially DBN-RBM, is the most popular model for feature extraction. DBN is widely used in brain signal for two reasons: (1) it learns the generative parameters that reveal the relationship of variables in neighboring layers efficiently; (2) it makes it straightforward to calculate the values of latent variables in each hidden layer [31]. However, most works that employed the DBN-RBM model were published before 2016. It can be inferred that the researchers prefer to use DBN for feature learning followed by a non-deep learning classifier before 2016; but recently, an increasing number of studies would like to adopt CNN or hybrid models for both feature learning and classification.

Moreover, generative models are rarely employed independently. The GAN- and VAE-based data augmentation and image reconstruction are mainly focused on fMRI and EEG signals. It is demonstrated that the trained classifier will achieve more competitive performance after data augmentation. Therefore, this is a promising research prospect in the future.

Last but not the least, there are 53 publications proposed hybrid models for brain signal studies. Among them, the combinations of RNN and CNN take about one-fifth proportion. Since RNN and CNN are illustrated having excellent temporal and spatial feature extraction ability, it is natural to combine them for both temporal and spatial feature learning. Another type of hybrid models is the combination of representative and discriminative models. This is easy to understand because the former is employed for feature refining, and the latter is employed for classification. There are 28 publications which almost covered all the brain signals proposed this type of hybrid deep learning models. The adopted representative models are mostly AE or DBN-RBM; at the meanwhile, the adopted discriminative models are mostly CNN. Apart from that, there are 12 papers proposed other hybrid models such as two discriminative models. For example, several studies proposed the combination of CNN and MLP where a CNN structure is used for extract spatial features and an MLP is used for classification.

In order to have a closer observation of the recent advances on deep learning-based brain signal analysis, we analyze the brain signal acquisition methods and the deep learning algorithms in terms of application performance. In some cases, various studies adopt the same deep architecture working on the same dataset but results in different performance, which maybe caused by the different pre-processing methods and hyper-parameter settings.

To begin with, the most appealing and hot field is that using brain signal analysis on health care area. For sleep quality evaluation, the dominate brain signals are spontaneous EEG which are measured while the patient is sleeping. The single RNN or CNN models seem have a good discriminative feature learning ability and lead to a comprehensive performance. Generally, most of the deep learning algorithms can achieve the accuracy of above 85% in the context of multiple sleep stage scenario. Upon this, the combined hybrid models (e.g. CNN integrates with LSTM) can only have incremental improvements.

One key method to detect Alzheimer's Disease is brain signal analysis by measuring the functions of specific brain regions. In detail, the diagnosis can be conducted by spontaneous EEG signals or fMRI images. For MD EEG, DBN is supposed to outperform CNN since the EEG signals contains more temporal instead of spatial information. As for the fMRI pictures, CNN have great advantages in the grid-arranged spatial information learning, which makes it obtain a very comprehensive classification accuracy (above 90%). As for epileptic seizure, the diagnosis are generally based on EEG signals. The single RNN classifier (e.g. LSTM or GRU) seems work better than its counterparts due to the excellent temporal dependency representing ability. Here, the complex hybrid models indeed outperform the single component. For example, [130] achieves a better specification than [116] on the same dataset because of combing with RNN. Most of the epileptic seizure detection models claim a rather high classification accuracy (above 95%). One possible reason is that the binary recognition scenario is much easier than multi-class classification.

The brain signal-controlled smart environment only appear in a small number of publications. Among them, the brain signals are collected through very different methods. This is an emerging but promising field because it is easy to integrate with smart home and smart hospital to benefit the individuals whether healthy or disable. Another advantage of brain signals is bridging people's inside and outer world by communication techniques. In this area, lots of investigations are focusing on the VEP signals because the VEP is obvious and easy to be detected. One important data source is from the third BCI competition. In addition, brain signal analysis can be widely implement in security systems since the brain signals are invisible and very hard to be mimicked. The characteristic of high fake-resistance enables brain signal a raising star in the identification/authentication in confidential scenarios. The drawbacks of brain signal-based security systems are the expensive equipment and inconvenient (e.g. the subject have to wear an EEG headset to monitor the brainwaves).

Affective computing has drawn much attention in recent years. The EEG signals have high temporal resolution and able to capture the quick-varying emotions. Therefore, almost all the studies are based on spontaneous EEG signals. The signals are gathered when the subject is watching video which is supposed to arouse the subject's specific emotion. Another reason for this phenomenon is that there are several open-source EEG-based affecting analysis datasets (e.g. DEAP and SEED) which greatly promote the investigation in this area. The EEG-based affective computing contains two mainstreams. One of them focuses on developing powerful discriminative classifiers (such as hierarchical CNN) which are designed to perform feature extraction and classification in the same step. The other tries to learn the latent features through deep representative models (e.g. DBN-RBM) and then send the learned representations into a powerful classifier (such as HMM and MLP). It can be observed that the former models ([88, 201]) seem outperform the latter methods ([96]) with a small margin on the SEED dataset.

Drive fatigue detection can be easily integrated in the platforms such as self-driving vehicles. Nevertheless, there are only a few publications in this area due to the expensive experimental cost and the lack of accessible dataset. Moreover, there are a lot of interesting applications (e.g. guilty knowledge test and gender detection) have been explored by deep learning models.

Until now, we have introduced several types of brain signals (e.g. spontaneous EEG, ERP, fMRI) and deep learning models that have been applied for each type. One promising research direction for deep learning-based brain signal research is to develop a general framework that can handle various brain signals regardless of the number of channels used for signal collection, the sample dimensions (e.g. 1-D or 2D sample), and stimulation types (e.g. visual or audio stimuli), etc The general framework would require two key capabilities: the attention mechanism and the ability to capture latent feature. The former guarantees the framework can focus on the most valuable parts of input signals, and the latter enables the framework to capture the distinctive and informative features.

The attention mechanism can be implemented based on attention scores or by various machine learning algorithms such as reinforcement learning. The attention scores can be inferred from the input data and work as a weight to help the framework to pay attention to the parts with high attention scores. Reinforcement learning has shown to be able to find the most valuable part through a policy search [85]. CNN is the most suitable structure for capturing features at various levels and ranges. In the future, CNN could be used as a fundamental feature learning tool and be integrated with suitable attention mechanisms to form a general classification framework.

One additional direction we may consider is how to interpret the feature representation derived by the deep neural network, what is the intrinsic relationship between the learned features and the task-related neural pattern, or neuropathology of mental disorders. More and more people are realizing that interpretation could be even more important than prediction performance, since we usually just treat deep learning as a black box.

Until now, most brain signal classification tasks focus on person-dependent scenarios, where the training samples and testing samples are collected from the identical individual. The future direction is to realize person-independent classification so that the testing data will never appear in the training set. High-performance person-independent classification is compulsory for the wide application of brain signals in the real world.

One possible solution to achieving this goal is to build a personalized model with transfer learning. A personalized affective model can adopt a transductive parameter transfer approach to construct individual classifiers and to learn a regression function that maps the relationship between data distribution and classifier parameters [232]. Another potential solution is mining the subject-independent component from the input data. The input data can be decomposed into two parts: a subject-dependent component, which depends on the subject and a subject-independent component, which is common for all subjects. A hybrid multi-task model can work on two tasks simultaneously, one focusing on person identification and the other on class recognition. A well-trained and converged model is supposed to extract the subject-independent features in the class recognition task.

The performance of deep learning highly depends on the size of training data, which, however, requires expensive and time-consuming manual labeling to collect abundant class labels for a wide range of scenarios such as sleep EEG. While supervised learning requires both observations and labels for the training, unsupervised learning requires no labels, and semi-supervised learning only requires partial labels [98]. They are, therefore, more suitable for problems with little ground truth.

Zhang et al proposed an adversarial variational embedding framework that combines a VAE++ model (as a high-quality generative model) and semi-supervised GAN (as a posterior distribution learner) [233] for robust and effective semi-supervised learning. Jia et al proposed a semi-supervised framework by leveraging the data distribution of unlabelled data to prompt the representation learning of labelled data [98].

Two methods may enhance the unsupervised learning: one is to employ crowd-sourcing to label the unlabeled observations; the other is to leverage unsupervised domain adaption learning to align the distribution of source brain signals and the distribution of target signals with a linear transformation.

Most of the existing brain signal systems focus on offline procedure which means that the training and testing dataset are pre-collected and evaluated offline. However, in the real-world scenarios, the brain signal systems are supposed to receive live data stream and produce classification results in real time, which is still very challenging.

For EEG signals, in the online system, compared to the offline procedure, the gathered live signals are more noisy and unstable due to lots of factors such as the less-concentrating of the subject [234] and the inherent destabilization of the equipment (e.g. fluctuating sampling rate). Through our empirical experiments, online brain signal systems generally perform a lower accuracy of 10% than their counterparts. One future scope of online implementation is to develop a batch of robust algorithms in order to handle the influence factors and discover the latent distinctive patterns underlying the noisy live brain signals. Aliakbaryhosseinabadi et al [235] implemented an EEG-based online system that achieves comparable performance, however, this work only investigates a very high-level target (i.e. human attention). Discovering the latent invariant representations through covariance matrices of EEG signals can help to mitigate the influence of extinct perturbations [236]. Some post-processing methods (e.g. voting and aggregating) [149, 166] can help to improve the decoding performance by averaging the results from multiple continues samples. However, these methods will inevitably bring higher latency. Thus, the post-processing requires a trade-off between the high-accuracy and low-latency.

For fNIRS and fMRI, the online evaluation is relatively less challenging since they have a rather low temporal resolution. The online images with less dynamic can be regarded as static images to some extent, which makes the online system approximating to the offline system. Furthermore, most fMRI and MEG signals are used to evaluate the user's neurological status (e.g. detect the effects of tumor) which does not require an instantaneous response. Thus, they have less demand for a real-time monitoring system.

Poor portability of hardware has been preventing brain signals from wide application in the real world. In most scenarios, users would like to use small, comfortable, or even wearable brain signal hardware to collect brain signals and to control appliances and assistant robots.

Currently, there are three types of EEG collection equipment: the unportable, the portable headset, and ear-EEG sensors. The unportable equipment has high sampling frequency, channel numbers, and signal quality but is expensive. It is suitable for physical examination in a hospital. The portable headsets (e.g. Neurosky, Emotiv EPOC) have 1–14 channels and 128–256 sampling rate but has inaccuracy readings and cause discomfort after long-time use. The ear-EEG sensors, which are attached to the outer eat, have gained increasing attention recently but remain mostly at the laboratory stage [237]. The ear-EEG sensors contain a series of electrodes which are placed in each ear canal and concha [238]. The EEGrids, to the best of our knowledge, is the only commercial ear-EEG. It has multi-channel sensor arrays placed around the ear using an adhesive ²⁶ and is even more expensive. A promising future direction is to improve the usability by developing a cheaper (e.g. lower than 200$) and more comfortable (e.g. can last longer than 3 h without feeling uncomfortable) wireless ear-EEG equipment.

EEG is the most commonly used non-invasive technique for measuring brain activities. EEG monitors the voltage fluctuations generated by an electrical current within human neurons. Electrodes placed on the scalp measure the amplitude of EEG signals. EEG signals have a low spatial resolution due to the effect of volume conduction which refers to the complex effects of measuring electrical potentials a distance from the source generators [239, 240]. EEG electrode locations generally follow the international 10–20 system [241]. The specific placement of electrodes is presented in figure 5 [10]. The EEG signals are collected while the subject is undertaking imagination task. Each line represents the signal stream collected from a single EEG electrode (also called 'channel') over time.

Zoom In Zoom Out Reset image size

The temporal resolution of EEG signals is much better than the spatial resolution. The ionic current changes rapidly, which offers a temporal resolution higher than 1000 Hz. The SNR of EEG is generally very poor due to both objective and subjective factors. Objective factors include environmental noises, the obstruction of the skull and other tissues between cortex and scalp, and different stimulations. Subjective factors contain the subject's mental stage, fatigue status, the variance among different subjects, and so on.

EEG recording equipment can be installed in a cap-like headset. The EEG headset can be mounted on the user's head to gather signals. Compared to other equipment used to measure brain signals, EEG headsets are portable and more accessible for most applications.

The EEG signals collected from any typical EEG hardware have several non-overlapping frequency bands (Delta, Theta, Alpha, Beta, and Gamma) based on the strong intra-band correlation with a distinct behavioral state [10]. Each EEG pattern contains signals associated with particular brain information. Table A1 shows EEG frequency patterns and the corresponding characteristics. Here, the degree of awareness denotes the perception of individuals when presented with external stimuli.

Compared to other signals (e.g. fMRI, fNIRS, MEG), EEG has several important advantages: (1) the hardware has higher portability with much lower price; (2) the temporal resolution is very high (milliseconds level). Among other non-invasive techniques, only MEG has the same level of temporal resolution; (3) EEG is relatively tolerant of subject movement and artifacts, which can be minimized by existing signal processing methods; (4) the subject does not need to be exposed to high-intensity ($\gt$1 T) magnetic fields, therefore, EEG can serve subjects that have metal implants in their body (such as metal-containing pacemakers).

As the most commonly used signals, there are a large number of sub-classes of EEG signals. In this section, we present a methodical introduction of EEG sub-class signals. As shown in figure 2, we divided EEG signals into spontaneous EEG and EPs. EPs can be split into EVP and steady-state EPs based on the frequency of the external stimuli [7]. Each potential contains visual-, auditory-, and somatosensory- potentials based on the external stimuli types. The dashed quadrilaterals in figure 2, such as Intracortical, SEP, SSAEP, SSSEP, and rapid serial auditory presentation (RSAP), are not included in this survey because there are very few existing studies working on them with deep learning algorithms. We list these signals for systematic completeness.

A.1.1. Spontaneous EEG

A.1.2. Evoked potential (EP)

EPs or evoked responses refers to the EEG signals which are evoked by an external stimulus instead of spontaneously. An EP is time-locked to the external stimulus while the aforementioned spontaneous EEG is non-time-locked. In contrast to spontaneous EEG, EP generally has higher amplitude and lower frequency. As a result, the EP signals are more robust across subjects.

According to the stimulation method, there exist two categories of EP: the ERP and the SSEP. ERP records the EEG signals in response to an isolated discrete stimulus event (or event change). To achieve this isolation, stimuli in an ERP experiment are typically separated from each other by a long inter-stimulus interval, allowing for the estimation of a stimulus-independent baseline reference [248]. The stimuli frequency of ERP is generally lower than 2 Hz. In contrast, SSEP is generated in response to a periodic stimulus at a fixed rate. The stimuli frequency of SSEP generally ranges within 3.5–75 Hz.

Event-related potential (ERP). There are three kinds of EPs in extensive research and clinical use: VEPs; AEPs; and somatosensory evoked potentials (SEPs) [28]. The VEP signals are mainly on the occipital lobe, and the highest signal amplitudes are collected at the Calcarine sulcus.

(1) Visual evoked potentials (VEP). VEPs are a specific category of ERP which is caused by visual stimulus (e.g. an alternating checkerboard pattern on a computer screen). VEP signals are hidden within the normal spontaneous EEG. To separate VEP signals from the background EEG readings, repetitive stimulation and time-locked signal-averaging techniques are generally employed.

RSVP [249] can be regarded as one kind of VEP. An RSVP diagram is commonly used to examine the temporal characteristics of attention. The subject is required to stare at a screen where a series of items (e.g. images) are presented one-by-one. There is a specific item (called the target) separates from the rest of the other items (called distracters). The subject knows which is the target before the RSVP experiment. For instance, the distracters can be a color change or letters among numbers. RSVP contains a static mode (the items appear on the screen and then disappear without moving) and a moving mode (the items appear on the screen, move to another place, and finally disappear). Nowadays, brain signal research mainly focuses on the static mode RSVP. Usually, the frequency of RSVP is 10 Hz which means that each item will stay on the screen for 0.1 s.

(2) Auditory evoked potentials (AEPs). AEPs are a specific subclass of ERP in which responses to auditory (sound) stimuli are recorded. AEP is mainly recorded from the scalp but originates at the brainstem or cortex. The most common AEP measured is the auditory brainstem response which is generally employed to test the hearing ability of newborns and infants. In the brain signal area, AEP is mainly used in clinical tests for its accuracy and reliability in detecting unilateral loss [250]. Similar to RSVP, RSAP refers to the experiments with rapid serial presentation of sound stimuli. The task for the subject is to recognize the target audio among the distracters.

(3) Somatosensory evoked potentials (SEPs). Generally, SEPs is abbreviated as SSEP or SEP. In this paper, we choose SEP as the abbreviation in case of the conflict with SSEPs. SEP are another commonly used subcategory of ERP which is elicited by electrical stimulation of the peripheral nerves. SEP signals conclude a series of amplitude deflection that can be elicited by virtually any sensory stimuli.

P300. P300 (also called P3) is an important component in ERP [251]. Here we introduce P300 signal separately since it is widely-used in brain signal analysis. Figure 6(a) shows the ERP signal fluctuation in the 500 ms after the stimuli onset. The waveform mainly concludes five components, P1, N1, P2, N2, and P3. The capital character P/N represents positive/negative electrical potentials. The following number refers to the occurrence time of the specific potential. Thus, P300 denotes the positive potential of ERP waveform at approximately 300 ms after the presented stimuli. Compared to other components, P300 has the highest amplitude and is easiest to detect. Thus, a large number of brain signal studies focus on P300 analysis. P300 is more of an informative feature instead of a type of brain signal (e.g. VEP). Therefore, we do no list P300 in figure 2. P300 can be analyzed in most of ERP signals such as VEP, AEP, SEP.

Zoom In Zoom Out Reset image size

In practice, P300 can be elicited by rare, task-relevant events in an 'oddball' paradigm (e.g. P300 speaker). In the oddball paradigm, the subject receives a series of stimuli where low-probability target items are mixed with high-probability non-target items. Visual and auditory stimuli are the most commonly used in the oddball paradigm. Figure 6(b) shows an example of visual-based P300 speller which enables the subject the spell letters/numbers directly through brain signals [247]. The 26 letters of the alphabet and the Arabic numbers are displayed on a computer screen which serves as the keyboard. The subject focuses attention successively on the characters they wish to spell. The computer detects the chosen character online in real time. This detection is achieved by repeatedly flashing rows and columns of the matrix. When the elements containing the selected characters are flashing, a P300 fluctuation is elicited. In the 6 × 6 matrix screen, the rows and columns flash in mixed random order. The flash duration and interval among adjacent flashes are generally set as 100 ms [252]. The columns and rows flash separately. First, the columns flash six times with each column flashing one time. Second, the rows will flash for six times. After that, this paradigm repeats for several times (e.g. N times). The P300 signals of the total 12 N flash will be analyzed to output a single outcome (i.e. one letter/number).

Steady state evoked potentials (SSEP). SSEP is another subcategory of EPs, which are periodic cortical responses evoked by certain repetitive stimuli with a constant frequency. It has been demonstrated that the brain oscillations generally maintain a steady level over time while the potentials are evoked by steady state stimuli (e.g. a flickering light with fixed frequency). Technically, SSEP is defined as a form of response to repetitive sensory stimulation in which the constituent frequency components of the response remain constant over time in both amplitude and phase [37]. Depending on the type of stimuli, SSEP divides into three subcategories: steady-state visually evoked potentials (SSVEPs), steady-state auditory evoked potentials (SSAEPs), and steady-state somatosensory evoked potentials (SSSEPs). In the brain signal area, most studies are focused on visual evoked steady potentials, and only rarely do papers focus on auditory and somatosensory stimuli. Therefore, in this survey, we mainly introduce SSVEP rather than SSAEP and SSSEP.

Commonly used visual-related potentials. VEPs are the most common used potentials. Therefore, it is essential to distinguish the three different VEP paradigms: VEP, RSVP, SSVEP. Here, we theoretically introduce the characteristics of each paradigm and then give three demonstration videos to provide a better understanding. First, the frequencies are different: the frequency of VEP is less than 2 Hz while the frequency of RSVP is around 10 Hz, and the frequency of SSVEP ranges from 3.5 to 75 Hz. Second, they have various presentation protocols. In the VEP paradigm, different visual patterns will be presented on the screen to check the user's brain signals changes. For instance, in this video ²⁷ , the image pattern is full of the screen and changes dramatically. In RSVP diagram, several items will be presented on a screen one-by-one. All the items are shown in the same place and share the same frequency. For example, the video ²⁸ shows an RSVP scenario which is called speed reading. In SSVEP paradigm, several items will be presented on a screen at the same time while the items are shown at variant positions with different frequencies. For example, in this demonstration video ²⁹ , there are four circles distributed on the up, down, left, and right sides of a screen and the frequency of each item differs from each other.

fNIRS is a non-invasive functional neuro-imaging technology using NIR light [38]. In specific, fNIRS employs NIR light to measure the aggregation degree of oxygenated Hb and deoxygenated-hemoglobin (deoxy-Hb) because Hb and deoxy-Hb have higher absorbence of light than other head components such as the skull and scalp. fNIRS relies on blood-oxygen-level-dependent (BOLD) response or hemodynamic response to form a functional neuro-image. The BOLD response can detect the oxygenated or deoxygenated blood level in the brain blood. The relative levels reflect the blood flow and neural activation, where increased blood flow implies a higher metabolic demand caused by active neurons. For example, when the user is concentrating on a mental task, the prefrontal cortex neurons will be activated, and the BOLD response in the prefrontal cortex area will be stronger [200].

Single or multiple emitter-detector pairs measure the Hb and deoxy-Hb: the emitter transmits NIR light through the blood vessels to the detector. Most existing studies use fNIRS technologies to measure the status of prefrontal and motor cortex. The former response to mental tasks and music/image imagery while the latter is a response to motor-related tasks (e.g. motor imagery). The monitored Hb and deoxy-Hb change slowly since the blood speed varies in a relatively slow ratio compared to electrical signals. Temporal resolution refers to the smallest time of neural activity reliably separated by the signal. The fNIRS has lower temporal resolution compared with electrical or magnetic signals. The spatial resolution depends on the number of emitter-detector pairs. In current studies, three emitters and eight detectors would suffice for adequately acquiring the prefrontal cortex signals; and six emitters and six detectors would suffice for covering the motor cortex area [29]. fNIRS has a drawback in that it cannot be used to measure cortical activity occurring deeper than 4 cm in the brain, due to the limitations in light emitter power and spatial resolution.

fMRI monitors brain activities by detecting changes associated with blood flow in brain areas [14]. Similar to fNIRS, fMRI relies on the BOLD response. The main differences between fNIRS and fMRI are as follows [24]. First, as the name implies, fMRI measures BOLD response through magnetic instead of optical methods. Hb differs in how it responds to magnetic fields, depending on whether it has a bound oxygen molecule. The magnetic fields are more sensitive to and are more easily distorted by deoxy-Hb than Hb molecules. Second, the magnetic fields have higher penetration than NIR light, which gives fMRI greater ability to capture information from deep parts of the brain than fNIRS. Third, fMRI has a higher spatial resolution than fNIRS since the latter's spatial resolution is limited by the emitter-detector pairs. However, the temporal resolutions of fMRI and fNIRS are at an equal level because they both constrained by the blood flow speed. fMRI has several flaws compared to fNIRS: (1) fMRI requires an expensive scanner to generate magnetic fields; (2) the scanner is heavy and has poor portability. In order to measure the signal of interest, CNR (contrast-to-noise ratio) has been investigated to measure the image quality of fMRI because researchers are more interested in the contrast between images rather than the raw images. So for fMRI data, using the CNR of the time series instead of (t)SNR is more preferred because CNR compares a measure of the activation fluctuations to the noise [253].

MEG is a functional neuroimaging technique for mapping brain activity by recording magnetic fields produced by electrical currents occurring naturally in the brain, using very sensitive magnetometers [254]. The ionic currents of active neurons will create weak magnetic fields. The generated magnetic fields can be measured by magnetometers like SQUIDs (superconducting quantum interference devices). However, producing a detectable magnetic field requires massive (e.g. 50 000) active neurons with similar orientation. The source of the magnetic field measured by MEG is the pyramidal cells which are perpendicular to the cortex surface.

MEG has a relatively low spatial resolution since the signal quality highly depends on the measurement factors (e.g. brain area, neuron orientations, neuron depth). However, MEG can provide very high temporal resolution (≥1000 Hz) since MEG directly monitors the brain activity from the neuron level, which is in the same level of intracortical signals. MEG equipment is expensive and not portable which limits its real-world deployment.

Since the main task of brain signal analysis is brain signal recognition, the discriminative deep learning models are the most popular and powerful algorithms. Suppose we have a dataset of brain signal samples $\{\mathbb{X}, \mathbb{Y}\}$ where $\mathbb{X}$ denotes the set of brain signal observations and $\mathbb{Y}$ denotes the set of sample ground truth (i.e. labels). Suppose an specific sample-label pair $\{\boldsymbol{x} \in \mathbb{R}^N, \boldsymbol{y} \in \mathbb{R}^M \}$ where N and M denote the dimension of observations and the number of sample categories, respectively. The aim of discriminative deep learning models is to learn a function with the mapping: $\boldsymbol{x} \rightarrow \boldsymbol{y}$. In short, the discriminative models receive the input data and output the corresponding category or label. All the discriminative models introduced in this section are supervised learning techniques which require the information of both the observations and the ground truth.

B.1.1. Multi-layer perceptron (MLP)

The most basic neural network is fully-connected neural networks (figure 7(a)) which only contains one hidden layer. The input layer receives the raw data or extracted features of brain signals while the output layer shows the classification results. The term 'fully-connected' denotes each node in a specific layer is connected with all the nodes in the previous and next layer. This network is too 'shallow' and generally not regarded as 'deep' neural networks.

Zoom In Zoom Out Reset image size

MLP is the simplest and the most basic deep learning model. The key difference between MLP and the fully-connected neural network is that MLP has more than one hidden layers. All the nodes are fully-connected with the nodes of the adjacent layers but without connection with the other nodes of the same layer. MLP includes multiple hidden layers. As shown in figure 7(b), we take a structure with two hidden layers as an example to describe the data flow in MLP.

The input layer receives the observation $\boldsymbol{x}$ and feeds forward to the first hidden layer,

$$\begin{equation} \boldsymbol{x^{h1}} = \sigma(\mathcal{T}(\boldsymbol{x})) \end{equation} \tag{ 3 }$$

where $\boldsymbol{x^{h1}}$ denotes the data flow in the first hidden layer and σ represents the non-linear activation function. There are several commonly used activation functions such as sigmoid/Logistic, Tanh, ReLU, we choose sigmoid activation function as an example in this section. Then, the data flow to the second hidden layer and the output layer,

$$\begin{equation} \boldsymbol{x^{h2}} = \sigma(\mathcal{T}(\boldsymbol{x^{h1}})) \end{equation} \tag{ 4 }$$

$$\begin{equation} \boldsymbol{y^{^{\prime}}} = \sigma(\mathcal{T}(\boldsymbol{x^{h2}})) \end{equation} \tag{ 5 }$$

where $\boldsymbol{y^{^{\prime}}}$ denotes the predict results in one-hot format. The error (i.e. loss) could be calculated based on the distance between $\boldsymbol{y^{^{\prime}}}$ and the ground truth $\boldsymbol{y}$. For instance, the Euclidean-distance based error can be calculated by

$$\begin{equation} error = \left \| \boldsymbol{y^{^{\prime}}}- \boldsymbol{y} \right \|_2 \end{equation} \tag{ 6 }$$

where $\left \| \cdot \right \|_2$ denotes the Euclidean norm. Afterward, the error will be back-propagated and optimized by a suitable optimizer. The optimizer will adjust all the weights and basis in the model until the error converges. The most widely used loss functions includes cross-entropy, negative log likelihood, mean square estimation, etc. The most widely used optimizers include Adaptive moment estimation (Adam), stochastic gradient descent, Adagrad (Adaptive subgradient method), etc

Several terms may be easily confused with each other: artificial neural network (ANN), deep neural network (DNN), and MLP. These terms have no strict difference and often mixed in literature and commonly used as synonyms. Generally, ANN and DNN can be used to describe deep learning models overall, including not only fully-connected networks but also other networks (e.g. recurrent, convolutional networks), but MLP can only refer to fully-connected network. Additionally, ANN contains all the models of neural networks, can be either shallow (one hidden layer) or deep (multiple hidden layers) while DNN does not cover shallow neural network [30, 31].

B.1.2. Recurrent neural networks (RNNs)

RNN is a specific subclass of discriminative deep learning model which are designed to capture temporal dependencies among input data [41]. Figure 8(a) describes the activity of a specific RNN node in the time domain. At each time ranges from [1, t + 1], the node receives an input I (the subscript represents the specific time) and a hidden state c from the previous time (except the first time). For instance, at time t it receives not only the input I_t but also the hidden state of the previous node c_t − 1. The hidden state can be regarded as the 'memory' of the nodes which can help the RNN 'remember' the historical input.

Zoom In Zoom Out Reset image size

Next, we will report two typical RNN architectures which have attracted much attention and achieved great success: long short-term memory (LSTM) and gated recurrent units (GRUs). They both follow the basic principles of RNN, and we will pay our attention to the complicated internal structures in each node. Since the structure is much more complicated than general neural nodes, we call it a 'cell.' Cells in RNN are equivalent to nodes in MLP.

Long short-term memory (LSTM). Figure 9(a) shows the structure of a single LSTM cell at time t [255]. The LSTM cell has three inputs (I_t , O_t − 1, and c_t − 1) and two outputs (c_t and O_t ). The operation is as follows:

$$\begin{equation} I_t, O_{t-1}, c_{t-1} \rightarrow c_t, O_t. \end{equation} \tag{ 7 }$$

I_t denotes the input value at time t, O_t − 1 denotes the output at the previous time (i.e. time t − 1), and c_t − 1 denotes the hidden state at the previous time. c_t and O_t separately denote the hidden state and the output at time t. Therefore, we can observe that the output O_t at time t not only related to the input I_t but also related to the information at the previous time. In this way, LSTM is empowered to remember the important information in the time domain. Moreover, the essential idea of LSTM is to control the memory of specific information. For this aim, LSTM cell adopts four gates: the input gate, forget gate, output gate, and input modulation gate. Each gate is a weight to control how much information can flow through this gate. For example, if the weight of the forget gate is zero, the LSTM cell would remember all the information passed from the previous time t − 1; if the weight is one, the LSTM cell would remember nothing. The corresponding activation function determines the weight. The detailed data flow as follows:

$$\begin{equation} f = \sigma(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 8 }$$

$$\begin{equation} i = \sigma(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 9 }$$

$$\begin{equation} o = \sigma(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 10 }$$

$$\begin{equation} m = \tanh(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 11 }$$

$$\begin{equation} c_t = f*c_{t-1} + i*m \end{equation} \tag{ 12 }$$

$$\begin{equation} h_t = o*\tanh(c_t) \end{equation} \tag{ 13 }$$

where i, f, o and m represent the input gate, forget gate, output gate and input modulation gate, respectively.

Zoom In Zoom Out Reset image size

Gated recurrent units (GRUs). Another widely used RNN architecture is GRU [256]. Similar to LSTM, GRU attempts to exploit the information from the past. GRU does not require hidden states, however, it receives temporal information only from the output of time t − 1. Thus, as shown in figure 9(b), GRU has two inputs (I_t and O_t − 1) and one output (O_t ). The mapping can be described as:

$$\begin{equation} I_t, O_{t-1} \rightarrow O_t. \end{equation} \tag{ 14 }$$

GRU contains two gates: reset gate r and update gate z. The former decides how to combine the input with previous memory. The latter decides how much of previous memory to keep around, which is similar to the forget gate of LSTM. The data flow as follows:

$$\begin{equation} z = \sigma(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 15 }$$

$$\begin{equation} r = \sigma(\mathcal{T}(I_t, O_{t-1})) \end{equation} \tag{ 16 }$$

$$\begin{equation} \bar{O_t} = \tanh(\mathcal{T}(I_t, r*O_{t-1})) \end{equation} \tag{ 17 }$$

$$\begin{equation} O_t = (1-z)*O_{t-1} + z*\bar{O_t}. \end{equation} \tag{ 18 }$$

It can be observed that there is an intermediate variable $\bar{O_t}$ which is similar to the hidden state of LSTM. However, $\bar{O_t}$ only works on this time point and unable to pass to the next time point.

We here give a brief comparison between LSTM and GRU since they are very similar. First, LSTM and GRU have comparable performance as studied by literature. For any specific task, it is recommended to try both of them to determine which provides better performance. Second, GRU is lightweight since it only has two gates and without the hidden state. Therefore, GRU is faster to train and requires few data for generalization. Third, in contrast, LSTM generally works better if the training dataset is big enough. The reason is that LSTM has better non-linearity than GRU since LSTM has two more control gates (input modulation gate and forget gate). As a result, LSTM, compared with GRU, is more powerful to discover the latent distinct information from large-level training dataset.

B.1.3. Convolutional neural networks (CNNs)

The term of representative deep learning refers to use DNN for representation learning. It aims to learn representations of input data that makes it easier to perform a downstream task (e.g. classification, generation, and clustering) [257].

The essential blocks of representative deep learning models are AEs, and RBMs ³⁰ . DBNs are composed of AE or RBM. The representative models including AE, RBM ³¹ , and DBN, are unsupervised learning methods. Thus, they can learn the representative features from only the input observations $\boldsymbol{x}$ without the ground truth $\boldsymbol{y}$. In short, representative models receive the input data and output a dense representation of the data. There are various definitions in different studies for several models (such as DBN, Deep RBM, and Deep AE), in this survey, we choose the most understandable definitions and will present them in detail in this section.

B.2.1. Autoencoder (AE)

As shown in figure 10(a), A AE is a neural network that has three layers: the input layer, the hidden layer, and the output layer [43]. It differs from the standard neural network, in that the AE is trained to reconstruct its inputs, which forces the hidden layer to try to learn good representations of the inputs.

Zoom In Zoom Out Reset image size

The structure of AE contains two blocks. The first block is called the encoder, which embeds the observation to a latent representation (also called 'code'),

$$\begin{equation} \boldsymbol{x^{h}} = \sigma(\mathcal{T}(\boldsymbol{x})) \end{equation} \tag{ 19 }$$

where $\boldsymbol{x^{h}}$ represents the hidden layer. The second block is called the decoder, which decodes the representation into the original space,

$$\begin{equation} \boldsymbol{y^{^{\prime}}} = \sigma(\mathcal{T}(\boldsymbol{x^{h}})) \end{equation} \tag{ 20 }$$

where $\boldsymbol{y^{^{\prime}}}$ represents the output.

AE forces $\boldsymbol{y^{^{\prime}}}$ to be equal to the input $\boldsymbol{x}$ and calculates the error based on the distance between them. Thus, AE can compute the loss function only by $\boldsymbol{x}$ without the ground truth $\boldsymbol{y}$

$$\begin{equation} error = \left \| \boldsymbol{y^{^{\prime}}}- \boldsymbol{x} \right \|_2. \end{equation} \tag{ 21 }$$

Compared to equation (6), this equation does not involve the variable $\boldsymbol{y}$ because it takes the input $\boldsymbol{x}$ as the ground truth. This is why AE is able to perform unsupervised learning.

Naturally, one variant of AE is deep-AE (D-AE) which has more than one hidden layer. We present the structure of D-AE with three hidden layers in figure 10(c). From the figure, we can observe that there is one more hidden layer in both the encoder and the decoder. The symmetrical structure ensures the smoothness of encoding and decoding procedure. Thus, D-AE generally has an odd number of hidden layers (e.g. 2n + 1) where the first n layers belong to the encoder, the (n + 1)th layer works as the code which belongs to both encoder and decoder, and the last n layers belong to the decoder. The data flow of D-AE (figure 10(c)) can be represented as

$$\begin{equation} \boldsymbol{x^{h1}} = \sigma(\mathcal{T}(\boldsymbol{x})) \end{equation} \tag{ 22 }$$

$$\begin{equation} \boldsymbol{x^{h2}} = \sigma(\mathcal{T}(\boldsymbol{x^{h2}})) \end{equation} \tag{ 23 }$$

where $\boldsymbol{x^{h2}}$ denotes the median hidden layer (the code). Then decode the hidden layer, we can get

$$\begin{equation} \boldsymbol{x^{h3}} = \sigma(\mathcal{T}(\boldsymbol{x^{h2}})) \end{equation} \tag{ 24 }$$

$$\begin{equation} \boldsymbol{y^{^{\prime}}} = \sigma(\mathcal{T}(\boldsymbol{x^{h3}})). \end{equation} \tag{ 25 }$$

It is almost the same as AE except that D-AE has more hidden layers. Apart from D-AE, AE has many other variants like denoising AE, sparse AE, contractive AE, etc Here we only introduce the D-AE because it is easily confused with the AE-based DBN. The key difference between them will be provided in section B.2.3.

The core idea of AE and its variants is simple, which is that condensing the input data $\boldsymbol{x}$ into a code $\boldsymbol{x^{h}}$ (generally the code layer has lower dimension) and then reconstructing the data based on the code. If the reconstructed $\boldsymbol{y^{^{\prime}}}$ can approximate to the input data $\boldsymbol{x}$, it can be demonstrated that the condensed code $\boldsymbol{x^{h}}$ carries enough information about $\boldsymbol{x}$, thus, we can regard $\boldsymbol{x^{h}}$ as a representation of the input data for future operation (e.g. classification).

B.2.2. Restricted Boltzmann machine (RBM)

RBM is a stochastic ANN that can learn a probability distribution over its set of inputs [44]. It contains two layers including one visible layer (input layer) and one hidden layer, as shown in figure 10(b). From the figure, we can see that the connection lines between the two layers are bidirectional. RBM is a variant of Boltzmann Machine with stronger restriction of being without intra-layer connections. In a general Boltzmann machine, the nodes in the same hidden layer will connect. Similar to AE, the procedure of RBM also includes two steps. The first step condenses the input data from the original space to the hidden layer in a latent space. After that, the hidden layer is used to reconstruct the input data in an identical way. Compared to AE, RBM has a stronger constraint which is that the encoder weights and the decoder weights should be equal. We have

$$\begin{equation} \boldsymbol{x^{h}} = \sigma(\mathcal{T}(\boldsymbol{x})) \end{equation} \tag{ 26 }$$

$$\begin{equation} \boldsymbol{x^{^{\prime}}} = \sigma(\mathcal{T}(\boldsymbol{x^h})). \end{equation} \tag{ 27 }$$

In the above two equations, the weights of $\mathcal{T}(\cdot)$ are the same. Then, the error for training can be calculated by

$$\begin{equation} error = \left \| \boldsymbol{x^{^{\prime}}}- \boldsymbol{x} \right \|_2. \end{equation} \tag{ 28 }$$

We can observe from the figure 10(d) that the Deep-RBM (D-RBM) is an RBM with multiple hidden layers. The input data from the visible layer firstly flow to the first hidden layer and then the second hidden layer. Then, the code will flow backward into the visible layer for reconstruction.

B.2.3. Deep belief networks (DBN)

A DBN is a stack of simple networks, such as AEs or RBMs [258]. Thus, we divided DBN into DBN-AE (also called stacked AE) which is composed of AE and DBN-RBM (also called stacked RBM) which is composed of RBM.

As shown in figure 11(a), the DBN-AE contains two AE structures while the hidden layer of the first AE works as the input layer of the second AE. This diagram has two stages. In the first stage, the input data feed into the first AE follows the rules introduced in section B.2.1. The reconstruction error is calculated and back propagated to adjust the corresponding weights and basis. This iteration continues until the AE converges. We get the mapping,

$$\begin{equation} \boldsymbol{x^1} \rightarrow \boldsymbol{x^{h1}}.\end{equation} \tag{ 29 }$$

Zoom In Zoom Out Reset image size

Then, we move on to the second stage where the learned representative code in the hidden layer $\boldsymbol{x^{h1}}$ will be used as the input layer of the second AE, which is

$$\begin{equation} \boldsymbol{x^{2}} = \boldsymbol{x^{h1}} \end{equation} \tag{ 30 }$$

and then, after the second AE converges, we have

$$\begin{equation} \boldsymbol{x^2} \rightarrow \boldsymbol{x^{h2}} \end{equation} \tag{ 31 }$$

where $\boldsymbol{x^{h2}}$ denotes the hidden layer of the second AE, meanwhile, it is the final outcome of the DBN-AE.

The core idea of AE is that of learning a representative code with lower dimensionality but containing most information of the input data. The idea behind DBN-AE is to learn a more representative and purer code.

Similarly, the DBN-RBM is composed of several single RBM structures. Figure 11(b) shows a DBN with two RBMs where the hidden layer of the first RBM is used as the visible layer of the second RBM.

Compare the DBN-RBM (figure 11(b)) and D-RBM (figure 10(d)). They almost have the same architecture. Moreover, DBN-AE (figure 11(a)) and D-AE (figure 10(c)) have similar architecture. The most important difference between the DBN and the deep AE/RBM is that the former is trained greedily while the latter is trained jointly. In particular, for the DBN, the first AE/RBM is trained first, after it converges, the second AE/RBM is trained [44]. For the deep AE/RBM, jointly training means that the whole structure is trained together, no matter how layers it has.

Generative deep learning models are mainly used to generate training samples or data augmentation. In other words, generative deep learning models play a supporting role in the brain signal field to enhance the training data quality and quantity. After the data augmentation, the discriminative models will be employed for the classification. This procedure is created to improve the robustness and effectiveness of the trained deep learning networks, especially when the training data is limited. In short, the generative models receive the input data and output a batch of similar data. In this section, we will introduce two typical generative deep learning models: VAE and GANs.

B.3.1. Variational autoencoder (VAE)

VAE, proposed in 2013 [46], is an important variant of AE, and one of the most powerful generative algorithms. The standard AE and its other variants can be used for representation but fail in generation for the reason that the learned code (or representation) may not be continuous. Therefore, we cannot generate a random sample which is similar to the input sample. In other words, the standard AE does not allow interpolation. Thus, we can replicate the input sample but cannot generate a similar one. VAE has one fundamentally unique property that separates it from other AEs, and it is this property that makes VAE so useful for generative modeling: the latent spaces are designed to be continuous which allows easy random sampling and interpolation. Next, we will introduce how VAE works.

Similar to the standard AE, VAE can be divided into an encoder and decoder where the former embeds the input data to a latent space and the latter transfers the data from the latent space to the original space. However, the learned representation in the latent space is forced to approximate a prior distribution $\boldsymbol{\bar{p(z)}}$ which is generally set as Standard Gaussian distribution. Based on the reparameterization trick [46], the first hidden layer of VAE is designed to have two parts where one denotes the expectation $\boldsymbol{\mu}$ and another denotes the standard deviation $\boldsymbol{\sigma}$, thus we have

$$\begin{equation} \boldsymbol{\mu} = \sigma(\mathcal{T}(\boldsymbol{x})) \end{equation} \tag{ 32 }$$

$$\begin{equation} \boldsymbol{\sigma} = \sigma(\mathcal{T}(\boldsymbol{x})). \end{equation} \tag{ 33 }$$

Then, the latent code in the hidden layer is not directly calculated but sampled from a Gaussian distribution $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$. The statistic code

$$\begin{equation} \boldsymbol{z} = \boldsymbol{\mu} + \boldsymbol{\sigma}*\boldsymbol{\varepsilon} \end{equation} \tag{ 34 }$$

where $\boldsymbol{\varepsilon} \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})$. The representation $\boldsymbol{z}$ is forced to a prior distribution, and the distance error_KL is measured by Kullback–Leibler divergence,

$$\begin{equation} error_{KL} = D_{KL}(z, \boldsymbol{\bar{p(z)}}) \end{equation} \tag{ 35 }$$

where $\boldsymbol{\bar{p(z)}}$ denotes the prior distribution. In the decoder, $\boldsymbol{z}$ is decoded into the output $\boldsymbol{y}^{^{\prime}}$,

$$\begin{equation} \boldsymbol{y^{^{\prime}}} = \sigma(\mathcal{T}(\boldsymbol{z})) \end{equation} \tag{ 36 }$$

and the reconstruction error is

$$\begin{equation} error_{recon} = \left \| \boldsymbol{y^{^{\prime}}}- \boldsymbol{x} \right \|_2. \end{equation} \tag{ 37 }$$

The overall error for VAE is combined by the DL divergence and the reconstruction error,

$$\begin{equation} error = error_{KL} + error_{recon}. \end{equation} \tag{ 38 }$$

The key point of VAE is that all the latent representations $\boldsymbol{z}$ are forced to obey the normal distribution. Thus, we can randomly sample a representation $\boldsymbol{z^{^{\prime}}} \in \boldsymbol{\bar{p(z)}}$ from the prior distribution and then reconstruct a sample based on $\boldsymbol{z^{^{\prime}}}$. This is why VAE is so powerful in generation.

B.3.2. Generative adversarial networks (GAN)

GANs [47] is proposed in 2014 and achieved great success in a wide range of research areas (e.g. computer vision and natural language processing). GAN is composed of two simultaneously trained neural networks with a generator and a discriminator. The generator captures the distribution of the input data, and the discriminator is used to estimate the probability that a sample came from the training data. The generator aims to generate fake samples while the discriminator aims to distinguish whether the sample is genuine. The functions of the generator and the discriminator are opposite; that's why GAN is called 'adversarial.' After the convergence of both the generator and the discriminator, the discriminator ought to be unable to recognize the generated samples. Thus, the pre-trained generator can be used to create a batch of samples and use them for further operations such as as classification.

Figure 12(b) shows the procedure of a standard GAN. The generator receives a noise signal $\boldsymbol{s}$ which is randomly sampled from a multimodal Gaussian distribution and outputs the fake brain signals $\boldsymbol{x}_F$. The distributor receives the real brain signals $\boldsymbol{x}_R$ and the generated fake sample $\boldsymbol{x}_F$, and then it predicts whether the received sample is real or fake. The internal architecture of the generator and discriminator are designed depending on the data types and scenarios. For instance, we can build the GAN by convolutional layers on fMRI images since CNN has an excellent ability to extract spatial features. The discriminator and the generator are trained jointly. After the convergence, numerous brain signals $\boldsymbol{x}_G$ can be created by the generator. Thus, the training set is enlarged from $\boldsymbol{x}_R$ to $\{\boldsymbol{x}_R, \boldsymbol{x}_G\}$ to train a more effective and robust classifier.

Zoom In Zoom Out Reset image size

Hybrid deep learning models refers to models which are composed of at least two deep basic learning models where the basic model is a discriminative, representative, or generative deep learning model. Hybrid models comprise two subcategories based on their targets: classification-aimed (CA) hybrid models and the non-classification-aimed (NCA) hybrid models.

Most of the deep learning related studies in brain signal area are focused on the first category. Based on the existing literature, the representative and generative models are employed to enhance the discriminative models. The representative models can provide more informative and low dimensional features for the discrimination while the generative models can help to augment the training data quality and quantity which supply more information for the classification. The CA hybrid models can be further subdivided into: (1) several discriminative models combined to extract more distinctive and robust features (e.g. CNN+RNN); (2) representative model followed by a discriminative model (e.g. DBN+MLP); (3) generative + representative model followed by a discriminative model; (4) generative + representative model followed by a non-deep learning classifier. In which, a representative model followed by a non-deep learning classifier is regarded as a representative deep learning model.

A few NCA hybrid models aim for brain signal reconstruction. For example, St-yves et al [259] adopted GAN to reconstruct visual stimuli based on fMRI images.