Top

Neuroinformatics

Published in:

Open Access 10-08-2021 | Original Article

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Authors: Parikshat Sirpal, Rafat Damseh, Ke Peng, Dang Khoa Nguyen, Frédéric Lesage

Published in: Neuroinformatics | Issue 3/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Patentsearch

Off

Abstract

In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model’s fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Introduction

Functional near infrared spectroscopy (fNIRS) is a non-invasive, mobile, and cost-effective neuroimaging technology that uses near infrared light to continually monitor changes in cerebral hemodynamic parameters (i.e., oxygenated (HbO) and deoxygenated hemoglobin (HbR), and total hemoglobin (HbT)) (Jobsis, 1977). The fNIRS method relies on the neurovascular coupling phenomenon which describes the intimate spatial and temporal relationship between neural activity and cerebral blood flow to map acute functional changes in the brain (Girouard & Iadecola, 2006). In a typical fNIRS setup, optodes corresponding to near-infrared light sources and their complimentary detectors are placed on the surface of the subject’s head. Infrared light emitted from the light source is absorbed or scattered as it enters cerebral tissue. Detected light is used to calculate the blood oxygenation changes associated with cerebral hemodynamic activity using the modified Beer-Lambert law (Kocsis et al., 2006; Scholkmann et al., 2014). Concentration changes in the oxygenation of hemoglobin quantifies the absorption of infrared light by the brain.

The fNIRS method offers several advantages as an alternative or complement to other functional imaging techniques (i.e., fMRI) (Strangman et al., 2002). fNIRS offers increased temporal resolution as compared to fMRI, and fNIRS hardware can be integrated with other modalities such as scalp electroencephalography (EEG) (Fazli et al., 2012; Khan & Hong, 2017; Miller, 2012). fNIRS signals have been recently used in studying brain state decoding as well as proven useful for brain computer interfacing over the last decade (Hong et al., 2015; Khan & Hong, 2015).

Scalp EEG technology is the clinical gold standard for studying the human brain (Müller-Putz, 2020) and EEG recordings can be classified into specific frequency bands: alpha, beta, delta, gamma, and theta (Cho et al., 2014; Freeman et al., 2003; Pedregosa et al., 2011; Zhao et al., 2018). The delta frequency range encompasses low frequencies with relatively high amplitude and slow waveforms ranging from 0.25–3.0 Hz. Delta frequencies are common in normal sleep and may incidentally appear with focal lesions, metabolic encephalopathy, or hydrocephalus (Amzica & Steriade, 1998; Hofle et al., 1997; Knyazev, 2012). The theta band includes frequencies between 4 and 7 Hz. While normal in young individuals, the theta frequency envelope is interpreted as slow activity in awake adults (Mantini et al., 2007; Pizzo et al., 2016; Sitnikova et al., 2016). As with delta waves, theta waves may be seen in focal lesions or in a more generalized distribution in diffuse neurological disorders. Alpha frequencies are between 8 and 13 Hz, representing the dominant rhythm in awake adults (Koch et al., 2008; Sigala et al., 2014). Beta activity ranges in frequency between 14–30 Hz and is usually observed in a bilaterally frontal symmetrical distribution (Canolty et al., 2006; Freeman et al., 2003; Merker, 2016). Higher frequency ranges represent gamma wave oscillations between 30–100 Hz. Gamma activity is seen during a wide range of activities, and is enhanced in rapid eye movement during sleep (Gross & Gotman, 1999; Hughes, 2008).

Multimodal EEG-fNIRS experimental setups record the spatiotemporal dynamics of brain activity, provide opportunities to observe the population dynamics of neural ensembles and offer increased benefit in fundamental and clinical analyses (Goldman et al., 2002; Laufs et al., 2003; Martinez-Montes et al., 2004; McKenna et al., 1994; Salek-Haddadi et al., 2003). In such setups, scalp EEG measures the brain’s electrical activity, and fNIRS signals encode the brain’s hemodynamic response (Chiarelli et al., 2017; Ogawa et al., 1992), with a delay of approximately 3 seconds post neural activity. Data from EEG-fNIRS setups have established causality between neuronal firing and changes in HbO, HbR, and HbT, reflecting electrical and hemodynamic fluctuations dictated by neurovascular coupling (Hughes, 2008; Logothetis et al., 2001; Mukamel et al., 2005; Singh, 2012). Recent interest has focused on determining spatial hemodynamic correlates from EEG recorded activity, particularly, in the blood oxygen level dependent signal (BOLD) (Czisch et al., 2004; Lemieux et al., 2001; Lövblad et al., 1999). Resting state studies have successfully demonstrated that low frequency EEG band signals are negatively correlated with modulations in the BOLD signal, particularly, infra-low gamma EEG band envelopes (Jia & Kohn, 2011; Niessing et al., 2005; Sumiyoshi et al., 2012).

The characterization of the relationship between electrophysiology and cerebral hemodynamics is clinically relevant in epilepsy. Seizures are self-terminating paroxysmal representations of aberrant brain activity (Moshé et al., 2015). It is believed that the neurovascular machinery causing seizures is similarly present in the brain interictally during normal function, suggesting to some extent that epilepsy is a dynamic disorder (Kobayashi et al., 2006; Richardson, 2012). The resting epileptic brain displays spontaneous neural activity believed to reflect its functional organization (Rojas et al., 2018; Tracy & Doucet, 2015). The interdependence of each component (i.e., neural and vascular) is a topic of interest to the wider clinical and neuroscience community. fMRI studies have shown that resting state networks in the epileptic brain undergo changes in their functional architecture (Luo et al., 2011; Wang et al., 2011). Increasingly, “task-free” resting state conditions in fMRI studies have been conducted with the assumption that functionally connected brain networks show similar profiles of activity over time (De Luca et al., 2006; He & Liu, 2008; Niu & He, 2014; Palva et al., 2010; Richardson, 2012; Shen, 2015).

In the context of epilepsy, resting state fMRI studies have shown that functional networks are abnormal (Bettus et al., 2009; Honda et al., 2021; Tracy & Doucet, 2015; Zhang et al., 2009, 2010a, b). Pre-clinical studies have proposed that there is a correlation between slow fluctuations in the resting state BOLD signal (~0.1 Hz) and slow fluctuations in neuronal firing rates in gamma band local field potentials (Richardson, 2012; Shmuel & Leopold, 2008; Zhang et al., 2020). This suggests that the resting state is related to physiologically active dynamic neuronal processes. Utilizing fNIRS signals for resting state functional connectivity has gained attention as a promising imaging tool to study brain function and provide valuable insight into the intrinsic networks present within the human epileptic brain (Fishburn et al., 2014; Geng et al., 2017; Niu & He, 2014; Wang et al., 2017).

In this study, we hypothesize that we can predict brain hemodynamics from electrical signals using a deep learning architecture from resting state multimodal EEG-fNIRS recordings collected from a cohort of 40 epileptic patients. Following which, we hypothesize that functional connectivity patterns derived from higher EEG frequency envelopes are increased as compared to lower EEG frequency envelopes.

Methods

Subjects and Protocol

Forty patients (27 males, 13 females; ranging in age of 11 to 62 years in age; mean age of 32.42 years, and standard deviation of 13.97 years) with refractory focal epilepsy were recruited for prolonged EEG-fNIRS recordings. Epilepsy diagnosis and epileptic focus localization was based on a comprehensive evaluation which included clinical history, video-EEG recording of interictal spikes and seizures, magnetic resonance imaging (MRI), positron emission tomography (PET) and for some patients ictal single photon emission computed tomography (SPECT) and magnetoencephalography (MEG) scans. Full details regarding patient profiles including age, gender, EEG and MRI findings are found in Table 1 of (Peng et al., 2014; Sirpal et al., 2019). A subset of patients had MRI evidence of encephalomalacia, cortical dysplasia, and/or hippocampal atrophy, a common finding in epilepsy, but this was neither an inclusion nor exclusion criterion. The presence of such findings (Dhamija et al., 2011; Woermann & Vollmar, 2009) is a common MRI finding in epileptic brains. The institutional review boards of Sainte-Justine Hospital and Centre Hospitalier de l’Université de Montréal approved the study.

Table 1

Detailed overview of the proposed convolutional neural network long-short term autoencoder (CNN-LSTM AE) model. The network receives as input resting state EEG time-series sequences, represented as a single matrix, and is trained to reconstruct the corresponding fNIRS resting state output. Model specifications and hyperparameters were heuristically determined. Convolutions and deconvolutions have kernels of size (1,2), and thus their effect is along the time dimension. Convolutions help in generating embeddings with higher level abstraction of the input EEG sequence. Deconvolutions reconstruct the fNIRS sequence at full resolution based on output embeddings. The decoder and encoder LSTM units have ReLU (Rectified linear units) activations

Layer	Description	Output size
Input	EEG sample sequence	(EEG sequence length, number of time points is 500, number of EEG channels is 21)
EEG Sequence Embedding
2-Dimensional convolution + Average Pooling + Dropout	$2D convolution: stride=(\mathrm{1,2});$ $kernel size=(\mathrm{1,7});$ $ReLU activation.$ $Dropout: 20\%.$ $Average Pooling kernel: (\mathrm{1,2})$	(EEG sequence length, 125, number of Features Maps)
2-Dimensional convolution + Average Pooling + Dropout	$2D convolution: stride=(\mathrm{1,2});$ $kernel size=(\mathrm{1,7});$ $ReLU activation.$ $Dropout: 20\%.$ $Average Pooling kernel: (\mathrm{1,2})$	(EEG sequence length, 62, number of Features Maps)
Reshape	Reshape into an elongated tensor	(EEG sequence length, 62 * number of Features Maps)
Encoder
LSTM 1 + Dropout	An LSTM layer with number of cells equal to the number of EEG sequence length. ReLU activation. Dropout: 20%	(EEG sequence length, 512)
LSTM 2 + Dropout	An LSTM layer with number of cells equal to the number of EEG sequence length. ReLU activation. Dropout: 20%	(1, 256)
Decoder
Repeat	Create repeated version of the latent vector	(fNIRS sequence length, 256)
LSTM 3 + Dropout	An LSTM layer with number of cells equal to the number of EEG sequence length. ReLU activation. Dropout: 20%	(fNIRS sequence length, 312)
LSTM 4 + Dropout	An LSTM layer with number of cells equal to the number of EEG sequence length. ReLU activation. Dropout: 20%	(fNIRS sequence length, 695)
fNIRS Sequence Reconstruction
Reshape	Reshape into a 2D tensor	(fNIRS sequence length, 5, number of Feature Maps 3)
Deconvolution1 + Dropout	$2D deconvolution: stride= (\mathrm{1,2}); ReLU activation;$ $kernel size = (\mathrm{1,2}). Dropout: 20\%.$	(fNIRS sequence length, 10, number of Feature Maps 4)
Deconvolution2 + Dropout	$2D deconvolution: stride=(\mathrm{1,2}); ReLU activation;$ $kernel size=(\mathrm{1,2}). Dropout: 20\%.$	(fNIRS sequence length, 20, number of fNIRS channels)

EEG-fNIRS Data Acquisition and Pre-Processing

Continuous EEG-fNIRS recordings were performed at the Optical Imaging Laboratory of Sainte-Justine Hospital in Montréal, Canada. Experimental protocol ensured ambient noise and lights were maintained at a minimum for patient comfort during data acquisition. Patients were further instructed to remain calm and placed in comfortable, climate-controlled ambulatory suites with curtains drawn to limit ambient light. Patients were continually telemonitored by trained clinical staff. fNIRS data was collected using the Imagent Tissue Oximeter system (ISS Inc.), a multi-channel frequency domain system recording at 19.5 Hz with wavelengths of 690 nm and 830 nm for sensitivity to HbR and HbO respectively. EEG data was recorded according to the standard 10–20 system using 21 electrodes (in positions Fp1, Fp2, F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, P8, O1, O2) at 500 Hz (Neuroscan Synamps 2TM system). Custom-made helmets, taking into consideration different head sizes and shapes were made to fit comfortably. Plastic and polyvinyl chloride manufacturing materials made them rigid and light. The helmets were equipped with a total of 64 light sources, 16 light detectors and 19 EEG electrodes that allowed for stable optical coupling between cortical regions and the scalp. This further prevented inter-optode shifting and movement artifacts to a large extent. Sensitivity of near-infrared light to cortical tissue was maintained by positioning the optodes approximately 3–4 cm apart. Electrodes were placed following the 10–20 EEG instrumentation standard, allowing for full head coverage (Barlow et al., 1974). Figure 1 shows the EEG-fNIRS helmet placed on a patient’s head.

EEG data was bandpass filtered between 0.1–100 Hz to remove instrumental noise and to remove drift related to physiological activity, particularly, higher frequencies. The unprocessed raw time series of the HbO and HbR signals was bandpass filtered to remove specific frequency components attributed to cardiac (approximately 1 Hz) and/or respiratory activity (approximately 0.2–0.3 Hz) (Gramfort et al., 2014; Lu et al., 2010; Peng et al., 2014). Signal fidelity was examined prior to analysis by channel-wise verification of signal intensity. Bandpass filtering was applied to EEG data to compute frequency bands of interest. We used a FIR bandpass filter and the lowcut and highcut values (Hz) for the delta, theta, alpha, beta, and gamma frequencies were set as: [1, 4], [4, 8], [8, 12], [12, 30], [30, 100] respectively (Gramfort et al., 2014). The signal to noise ratio (SNR) threshold applied in channel analysis was defined as those channels less than 30% of the mean SNR of all channels. fNIRS channels deemed to have SNR were eliminated and not included for analysis. This led to an average of 138 channels per patient. Changes in HbO and HbR were calculated via the HomER and MNE software packages (Gramfort et al., 2014; Huppert et al., 2009).

Multiple consecutive recordings were performed, with each recording approximately 15 min led to a compendium of 200 recordings totaling 50 hours of recording time. Data was bandpass filtered in the 0.01 to 0.1 Hz frequency range to be in the resting state range (Tong et al., 2012). The resting state period (indexed from patients when they were resting comfortably) ranged between 7 to 10 minutes with a mean of 8.35 minutes (Geng et al., 2017; Li et al., 2015; Zhang et al., 2010a, b). To correct for motion, we performed dimension reduction via principal component analysis on EEG-fNIRS data and removed components with the most variance. Further, artifact rejection with (10% variation from normalized intensity) was applied to remove additional motion artifacts. Artifact-free data points were then filtered for the effects of respiratory and cardiac signal with a cutoff frequency of 0.2 Hz. Finally, HbO concentrations were calculated for each channel using the modified Beer–Lambert law.

Structural MRI registration of optode and electrode position was done using neuro-navigation (Brainsight, Rogue-Research Inc.). Channel positions were cross-referenced with the patient MRI and adapted to ensure coverage of the epileptic focus, the contralateral homologous region, and as much area as possible of other brain regions. The MRI was segmented into six different layers: air, scalp, skull, CSF, gray matter and white matter. The gray matter layer was used to extract six two-dimensional cortical projections. The three-dimensional position of each channel was projected onto these two-dimensional topographic maps, of which the following views were considered: dorsal, frontal, left and right views.

Neural Network Architecture

We built a deep sequence-to-sequence multimodal autoencoder to predict fNIRS signals from input scalp EEG signals. Autoencoders are powerful machine learning models trained in a self-supervised fashion to reconstruct inputs by learning their abstract representations (Kocsis et al., 2006; Lindauer et al., 2010; Socher et al., 2011; Vincent et al., 2010). The autoencoder embedded signals in a low dimensional latent space, where both the encoder and decoder are formulated as deep neural networks.

Recurrent neural networks (RNN) have been widely used in time series modeling since they account for the temporal state within data (Baytas et al., 2017; Chung et al., 2016; Merity et al., 2018; Mikolov et al., 2010). The output depends on hidden states and feedback connections present within hidden units. Previous states can be used as inputs, thereby allowing RNNs to hold memory. In our model, we used backpropagation through time, a common gradient descent type training technique (Sutskever et al., 2014). The innate problem of RNN gradient based training is that derivatives propagated via recurrent connections either become exceedingly small or large (Goodfellow et al., 2016; Luong et al., 2015), causing a vanishing or exploding gradient respectively. Long short-term memory units (LSTM), a variant of the vanilla RNN architecture overcomes the vanishing gradient problem (Greff et al., 2017; Gregor et al., 2015; Lecun et al., 2015). LSTM units receive external inputs and generate hidden outputs via input, output, and forget gates and a memory cell. The gates and memory cell are internally connected with weighted links. The gates are connected with external sources, which are current state sequential inputs and previous hidden states. This prevents the LSTM from storing useless or noisy input information (Greff et al., 2017; Gregor et al., 2015; Lecun et al., 2015).

The LSTM autoencoder model (LSTM-AE) as proposed by Srivastava et al. consists of encoder LSTM units and decoder LSTM units (Srivastava et al., 2015). The encoder LSTM receives input sequences and encodes them into a feature vector as the LSTM generates hidden outputs (Lipton et al., 2016; Wang et al., 2016). Likewise, the decoder LSTM receives the feature vector and decodes it into the original input sequences. LSTM-AEs learn a compressed representation of sequential data and have been used in video, text, audio, and time series sequence data (Lipton et al., 2016; Srivastava et al., 2015; Wang et al., 2016). In this work, multiple LSTM layers were incorporated to learn temporal representations. Our model also includes convolutional layers to extract high level spatial percepts from channel combinations. We input EEG sequential data accounting for hemodynamic delays to perform sequence-to-sequence encoding (Luong et al., 2015; Truong et al., 2018; Vincent et al., 2008; Zhang, 2018). These input EEG sequences are convolved by two convolutional neural networks (CNN) and subsequently fed into the first two encoding long short-term memory (LSTM) modules. EEG data samples are projected in the latent space with fixed length vectors that provide more compressed representations, which are then used to decode and reconstruct the output fNIRS data, by the LSTM decoding modules.

After testing multiple architectures with exhaustive hyper-parameter optimization, we designed our model as follows: The encoder is comprised of LSTM layers preceded by convolutional blocks. Convolutions in each block have a kernel size of (1, 7) and stride size of (1, 2). The decoder is comprised of LSTM layers which manipulate the vectors in the latent space to provide a final output dimension equal to that of an fNIRS sample. We evaluated our model in terms of cross-modal reconstruction error (Zhao et al., 2018), denoted as RE. The objective is to simultaneously minimize the distance between fNIRS data samples and maximize the distance between each fNIRS and EEG data points (i.e., minimizing the RE is equivalent to maximizing the likelihood function). Once the model was trained, the corresponding RE was calculated on an independent testing subset (see below) by computing the sum of the Euclidean distance between ${x}_{t}$ and its corresponding reconstruction, $\widehat{x}$, over all L dimensions, as expressed in Eq. 1 below:

$${\in }_{t}=\sum_{l=1}^{L}\left|{\widehat{x}}_{t,l}-{x}_{t,l}\right|, t\epsilon\; T$$

(1)

Model output is denoted as ${\widehat{|x}}_{t}|$.

EEG data is processed as follows. First, matching EEG and fNIRS data are parsed from our data directory, following which the respective data (EEG or fNIRS) is labeled according to the resting state periods. Feature scaling is performed using the MinMaxScaler class (Pedregosa et al., 2011) on EEG input data which sets the range of values between 0 and 1. Input signals are mean centered prior to being fed into the model. Then, data is fed into the convolutional layers and travels to the LSTM and deconvolution modules. A detailed schematic view of our model is shown in Fig. 2 below.

Training Details

The model was designed to use patient specific EEG signals as input to decode fNIRS signals. For each patient, the data was randomly split into training, validation, and testing subsets, with a proportion of 60% training, 20% testing and 20% respectively. We experimented with various model depths and determined deep LSTMs to outperform shallow LSTMs. This is likely due to the larger hidden state which occurs because of increasing layers. Complete training details are given below.

We initialized the LSTM’s parameters with the uniform distribution between 0 and 1. This was done to counteract the exploding gradients problem intrinsic to LSTMs, thereby enforcing a hard constraint on the norm of the gradient by scaling it between 0 and 1. Simultaneously, we specified starting node values for the LSTM computations by preparing a feed dictionary which has input EEG data and a target label. It is important to note that the LSTM can learn how to map input sequences as model training is patient specific into a fixed dimensional vector representation and can learn temporal dependencies.
Backpropagation through time was used with a learning rate of 0.05, batch size of 60 and 50 epochs, all of which were heuristically determined.
Each fNIRS signal generated corresponds to an EEG sequence input. An element in the EEG sequence corresponds to 1 second of recording with 500 time points (sampling frequency is 500 Hz) for each EEG channel. Data batches were generated for sequence processing by using the utility class for batch generation in the Keras framework. Briefly, this class uses as input a sequence of data points to produce batches for training and validation. Data points outside of the start and end indices of resting state periods (as marked in our ground truth) are not used in the output sequences. The final EEG data used as input is two dimensional, i.e., [data points, channels].

To summarize, the model was trained as follows: (a) we designed LSTM layers with corresponding LSTM cells (b) model parameters were uniformly initialized in the range between [0,1], (c) dropout was applied with value of 0.2, and average pooling was applied to reduce the probability of model overfitting, (d) we used backpropagation through time with a learning rate of 0.05, (e) we used a batch size of 60 and 50 training epochs for each patient.

Model Validation

After training and saving our model’s weights, we validated the model’s intra-patient predictive capacity by using individual EEG recordings as input to predict fNIRS signals. This was possible since our dataset contains multiple recordings from each patient. To diagnose performance, we plotted learning curves to ensure we did not overfit during training. As an illustrative example, Fig. 3 shows the learning curves for patients 1, 4, and 23.

Model Predictions

The model predicts signals by appending ‘output state’, and ‘output prediction’ matrices. LSTM cells are connected recurrently to each other. Decoder inputs are two-dimensional matrices which are passed into decoder LSTM layers. fNIRS data is shifted one sequence ahead to hold data in LSTM memory and finally decoder outputs are returned due to the data passing through the deconvolution layers.

Functional Connectivity Validation

We chose the seed channel from a region of interest, defined to be a region which had adequate optode coverage confirmed by our source/detector montage and an acceptable level of signal fidelity. That is, signals that were $\pm$ 2 standard deviations of the mean and displayed low SNR (i.e., signal amplitude less than 30% of mean signal amplitude) were removed from analysis. We then computed the Pearson product-moment correlation coefficients between the experimental fNIRS timeseries of the seed channel and the experimental fNIRS timeseries of all other channels. Subsequently, the Pearson product-moment correlations (and corresponding Fisher z-scores) were computed between the experimental seed channel timeseries and our model’s predicted fNIRS timeseries for all other channels. The two sets of correlation coefficients were respectively projected to an MRI head template based on the three dimensional coordinates of the corresponding channels using Atlasviewer (Aasted et al., 2015). The connectivity value at each voxel of the cortex was obtained from the correlation coefficients of all channels with a weighted-average method using the reciprocal of the cube of the distance from the voxel to each fNIRS channel.

In order to quantitively evaluate and compare the results of our functional connectivity studies, we computed the root mean square error (Eq. 2) i.e., the standard deviation of the residuals between functional connectivity values in experimental fNIRS and reconstructed fNIRS time courses derived from full spectrum EEG and specific EEG frequency band signals for all patients in our cohort.

$${RMSE}_{{FC}_{C}}=\sqrt{\frac{{{\sum }_{i=c}\left({fc}_{i}-{\widehat{f}}_{c}\right)}^{2}}{C}}$$

(2)

where “C” is the number of channels per functional connectivity analysis, ${fc}_{i}$ is the connectivity value of experimental fNIRS and ${\widehat{f}}_{C}$ is the connectivity value of model fNIRS reconstructions.

Results

This section describes the reconstruction results obtained using full spectrum EEG and subsequently EEG frequency ranges as model input. Intra-patient reconstructions are also presented; we explore spatial reconstruction, resting state predictions, and functional connectivity.

Full Spectrum EEG Performance and Feature Analysis

Resting state full spectrum EEG signals from all channels were input in the model. To decode fNIRS channels from encoded EEG channels, the model’s decoder layers used the encoder’s latent state as input as data traveled through LSTM units. Figure 4 below quantifies performance on selected individual patients with full spectrum EEG signals as input.

Figure 5 provides the group estimate of reconstruction error for all patients given scalp full spectrum EEG recordings.

Intra-patient Reconstructions on Separate Recording Sessions

Here, we report results on intra-patient fNIRS reconstructions provided EEG resting state as input. Specifically, we hypothesized that our model when trained with a patient’s single recording was able to reconstruct fNIRS signals from a subsequent recording. To examine our model’s predictive capacity and to cross-validate our model, we first trained our network on a patient’s single recording. Next, we used our trained network and aimed to reconstruct fNIRS signals from a subsequent recording from the same patient. The data was partitioned into training, testing, and validation subsets in a 60/20/20 manner. This was done for all data across all patients and recordings. Figure 6 displays the group results for intra-patient fNIRS signal reconstructions and Fig. 7 displays the fNIRS reconstructions for channel 5 from patient 10 across recordings 1, 3, 4.

Spatial Variability of Reconstructions

We then explored the model’s predictions sensitivity to channel location on the head. The topographic robustness of the model suggests the predictions are reasonably invariant across the brain. Channel locations were chosen if they offered coverage of most of the brain within the constraints of the source/detector montage and had an acceptable level of signal fidelity as indicated in “EEG-fNIRS Data Acquisition and Pre-Processing” section. As an illustrative example, Fig. 8 shows the model’s spatial predictions for patient 10.

EEG Frequency Decomposition and Resting State Predictions

After model training and validation, we computed EEG frequency bands, namely: delta [0.5–3 Hz], theta [4–7 Hz], alpha [8–13 Hz], beta [14–30 Hz], and gamma [30–100 Hz]. To ensure the presence of appropriate power in the frequency ranges, the spectral power of EEG signals was obtained using the Welch’s power spectral density function. Welch's method was preferred over other methods (i.e., standard periodogram spectrum estimation and Bartlett's method) as Welch’s method offsets a reduced frequency resolution with a reduction in signal noise in the estimated power spectra in exchange for reducing the frequency resolution (Welch, 1967). The Welch method partitions the signal into overlapping segments thereby mitigating the loss of edge data. The overlapped data segments are then windowed in the time domain. Subsequent computation includes the discrete Fourier transform, followed by averaging the periodograms leading to a final $n x m$ array representing power measurements by frequency bins.

All computations (including Fourier decomposition, Welch’s power spectral density) were performed using the MNE software package (Gramfort et al., 2014). Figure 9 shows the model’s predictions from EEG frequency ranges input using patient 10 (fNIRS channel 10).

We calculated decoded fNIRS reconstruction error metrics, as shown in Fig. 10, for each EEG frequency range and calculated patient wise reconstruction error. The gamma and beta frequency bands demonstrated the lowest error rates and in the lower EEG frequency ranges, we noticed increased fNIRS reconstruction error, possibly owing to the fact that our model was possibly not able to learn appropriate features to reconstruct fNIRS signals.

To further determine which EEG frequency band can reconstruct fNIRS signals with the lowest reconstruction error on average, we calculated band wise reconstruction error for all patients, as shown in Fig. 11. Following which, we conducted one-tailed paired t-tests to test whether there is a statistical difference in reconstruction error between any two of the five bands when compared to gamma in the following combinations: [delta, gamma], [theta, gamma], [alpha, gamma], and [beta, gamma]. Bonferroni correction was then applied to control the family-wise error rate to be less than 0.05. The gamma frequency band reconstructs fNIRS signals with increased fidelity on average as compared to other frequency bands.

Functional Connectivity Results

We computed functional connectivity mappings for experimental fNIRS and our model’s fNIRS reconstructions. We compared experimental fNIRS and fNIRS reconstructions derived from both full spectrum EEG and the EEG gamma band for all patients. The root mean square error (standard deviation of the residuals) was used as an estimator of the error in our connectivity studies. On a group level, we noticed a lower error in functional connectivity analyses between experimental fNIRS and fNIRS reconstructions derived from full spectrum EEG as compared with experimental fNIRS and fNIRS reconstructions derived from the gamma band and are shown in Fig. 12.

Figures 13, 14 show examples of the functional connectivity mapping results generated using the correlations of the timeseries from patient 22, with the seed channel chosen as 20. First, we used full spectrum EEG predictions as input for functional connectivity computations (Fig. 13), followed by analysis using the EEG gamma frequency band as input (Fig. 14).

Discussion

Deep learning models obviate cumbersome and brittle feature engineering processes replacing them with hierarchical feature learning. In this work, we developed a deep learning CNN-LSTM sequence-to-sequence autoencoder to predict fNIRS signals from resting state EEG signals in the epileptic brain. Our model was trained using a 60/20/20 split for training, testing, and validation, respectively. The results here demonstrate that in the context of epileptic resting state recordings, fNIRS signals can be predicted using full spectrum as well as specific frequency range EEG signals to a certain extent. We further validated our method by reconstructing the functional connectivity in the brain using the predicted fNIRS and compared it to the functional connectivity using experimental fNIRS.

From a neurophysiological standpoint, the resting epileptic brain is in a dynamic state and cerebral blood flow is in constant flux (Wang et al., 2011). Recent work has shown the presence of abnormal functional networks in the interictal state (Murta et al., 2015; Richardson, 2012). Thus, even with removal of systemic physiological components underlying compensation by molecular and cellular mechanisms can possibly help predict components of systemic physiology in addition to hemodynamic brain activity (Pressl et al., 2019). Our experimental findings can be related to known physiological phenomena being generated at the frequency of Mayer waves (~0.1 Hz), as these oscillations reflect fluctuation in cerebral arterial blood pressure (Nikulin et al., 2014; Schwab et al., 2009). The presence of these oscillations persisting after filtering can be partly due to the fact that they share a common spectral range with typical hemodynamic responses (Yücel et al., 2016). On the other hand, these oscillations correspond to cerebral vasomotion (i.e., extra neuronal) and are possibly related to blood vessel tonal oscillation (Aalkjær et al., 2011; Julien, 2006; Quaresima & Ferrari, 2019; Sassaroli et al., 2012).

The exact mechanics of physiological signal presence within EEG signals has not been established with certainty. However, experimental results from this work suggest the following: our model can capture subtle hemodynamic dependencies within the EEG resting state signal and its fNIRS correlate via the neurovascular coupling phenomenon.

These nuanced features within the EEG signal are encoded and subsequently decoded by the architectural components of the model, particularly the convolutional LSTM parameters (Greff et al., 2017; Sutskever et al., 2014). The model’s encoder and decoder and parameters (e.g., the activation function) may have enhanced feature extraction in resting state EEG data and its corresponding correlate in fNIRS signals. In addition, the features computed by using the outputs or hidden states of the recurrent units and the model may extract long-term dependencies (electrical and/or physiological) in resting state EEG signals from the LSTM modules via the gating mechanism (Sutskever et al., 2014). Furthermore, when cerebral blood flow (CBF) varies, changes occur in both the metabolic and electrical activity of cortical neurons with corresponding EEG changes (Sassaroli et al., 2012).

Events responsible for evoking the fNIRS response can be divided into subthreshold synaptic and suprathreshold spiking activities (Curtin et al., 2019; Sharbrough et al., 1973). Excitatory and inhibitory neurons which are often located within close proximity in the brain are simultaneously active and may contribute to the hemodynamic response (Franaszczuk et al., 2003). Slower EEG frequency envelopes (i.e., delta and theta) are generated by the thalamus and cortical cells in layers II-VI. Faster frequencies (i.e., beta and gamma) arise from cells in layers IV and V of the cortex (Foreman & Claassen, 2012; Merker, 2016). Changes in electrical potential seen in EEG recordings are closely tied to cerebral blood flow (CBF) and when normal CBF declines to approximately 25– 35 ml/100 g/min, the EEG signal first loses faster frequencies, then as the CBF decreases to approximately 17–18 ml/100 g/min, slower frequencies gradually increase. The interdependent relationship between CBF and neuronal activity in the resting epileptic brain is theorized to be captured by the model used in this work. Exploring the spatial localization of EEG frequency oscillations can help to determine if the presence of physiological signals is variable across patients and electrodes thereby possibly lending credence to the hypothesis that these oscillations are unlikely to be generated by a single source.

We show spatial decoding is possible using our model. Examination of the LSTM memory units and the latent space architecture in autoencoders can demonstrate correlation between data that were previously unknown. Utilizing the architecture developed here to predict brain hemodynamics, a next step would be to understand the structure of the latent variable (multidimensional vector) to unpack the principal components of the fNIRS or EEG signal.

A second point for further investigation is to integrate an attention mechanism in our model. Since LSTM cells can lead to ambiguous memory activations, an attention mechanism allows for encoding input into a sequence of vectors and from this, we can choose a subset adaptively during decoding. In this condition, the model no longer needs to utilize fixed length vectors thereby increasing performance metrics at the cost of computational time. Attention implemented in our model would enable us to inspect the relationship between encoded and decoded sequences by model weight visualization.

In comparison with lower frequency range EEG signals, results here suggest that higher frequency EEG envelopes reconstruct fNIRS signals with less error. Our results corroborate that EEG gamma band based fNIRS reconstructions show a closer fit between the observed and predicted hemodynamic responses as opposed to other EEG frequency ranges (Ebisch et al., 2005; Murta et al., 2015; Niessing et al., 2005). This is possibly because higher frequencies engage an increased number of neurons, but it is less apparent if this is attributed to baseline network activity or part of a pivotal functional role. Gamma rhythms in the brain provide an indication of engaged networks and have been observed in several cortical and subcortical structures. These rhythms are typically stronger for some stimuli as compared to others, thereby displaying selectivity to that of nearby neuronal activity (Jia & Kohn, 2011; Whittingstall & Logothetis, 2009). GABA-ergic inhibitory interneuron activity is considered to be crucial to generate EEG gamma frequency activity and this may be increased via interactions with excitatory neurons (Jia & Kohn, 2011; Park et al., 2011; Ray & Maunsell, 2010). However, to fully interpret the impact of this activity warrants an investigation into the cellular mechanisms responsible for their generation.

In the second part of our work, we explored functional connectivity in the resting state of the epileptic brain. We hypothesized that our network’s predictions can help reveal functional connections and on a group level, predicted fNIRS from full spectrum EEG have higher connectivity as compared to predictions derived from the EEG gamma band. Experimental resting state fNIRS data and predicted fNIRS data was correlated to reveal similar connections near the set seed but metrics decreased generally as distance increased from the seed. This can be due to numerous factors: 1. noise causing a decrease in reconstruction quality, 2. a decrease in gamma activations at the region of interest, and 3. model parameters unable to completely learn the nuances present within the signal. Furthermore, systemic artifacts from the scalp and skull behave as dominant noise sources in resting state fNIRS signals, leading to inaccurate reconstruction. Utilizing an EEG-fNIRS experimental setup with short separation channels, measuring approximately 1–2 cm in spatial separation between source and detector could lead to sufficient noise reduction and improved signal sensitivity (Gagnon et al., 2012; Kohno et al., 2007). We hypothesize that reconstruction metrics and corresponding functional connectivity network measures stabilize with increased signal quality and resting state duration, thereby decreasing the disparities present between experimental and predicted time series.

The resting state epileptic brain and connectivity between brain networks is dynamic (Deco et al., 2011; McKenna et al., 1994). Typically, fMRI has been used for computing functional connectivity but there are inherent limitations of fMRI, particularly, slow dynamics, regional variability of the hemodynamic response to neuron firing and the fact that some patients are not able to undergo an fMRI scan easily (i.e., claustrophobia, paroxysmal seizure occurrence during scanning) (Pressl et al., 2019; Richardson, 2012). By showing the possibility of obtaining brain hemodynamic data from neural signals, the results here add an additional dimension for understanding the epileptic human brain, aid in clinical decision making, and provide a complementary measure to fMRI, particularly in locations where access to fMRI technology is scarce or not possible.

Scalp EEG technology remains the clinical gold standard for the noninvasive assessment of electrical brain activity (Dash et al., 2017). Using EEG signals in conjunction with predicted brain hemodynamics can possibly improve clinical management and ultimately patient outcomes (Connolly et al., 2015; Helbok & Claassen, 2013). Multimodal EEG-fNIRS analysis using deep learning frameworks, as the one presented in this work, can improve our understanding of cerebral neurovascular coupling and pathophysiology. The results from this work can be abstracted for applications to other neurological and neuropsychiatric pathologies, such as stroke, spinal cord injuries, traumatic brain injuries, Alzheimer’s disease, attention-deficit hyperactivity disorder, post-traumatic stress disorder, and dementia to name a few (Fair et al., 2013; Phillips et al., 2018; Siegel et al., 2016). Furthermore, hemodynamic predictions from electrical brain signals can be useful in treatment strategies utilizing neurofeedback (i.e., neuroprosthetics, transcranial direct current stimulation) as well as towards developing precision medicine strategies (DeBettencourt et al., 2015; Dutta et al., 2015; Kotliar et al., 2017; Nicholson et al., 2016; Ros et al., 2014; Sitaram et al., 2017; Thair et al., 2017). Predicting hemodynamics from EEG increases clinical diagnostic specificity, allowing differentiation between pathological conditions that may appear similar but require different treatments (Citerio et al., 2015; Le Roux, 2013). Currently, therapeutic strategies follow a ‘reactive’ model: corrective actions are triggered by abnormal values in single parameters (i.e., EEG signals) and a stepwise approach is used with increasing therapeutic intensity. Comprehensive signals (i.e., EEG and predicted hemodynamics) can shift this paradigm towards a ‘goal-directed’ management strategy (Le Roux, 2013; Maas et al., 2012; Schmidt & De Georgia, 2014).

Conclusion

We designed and implemented a deep learning model to predict resting state hemodynamics given specific resting state scalp EEG frequencies from a cohort of epileptic patients. The robust multidimensional dataset used here allowed us to investigate the relationship between brain hemodynamics and neural signals via neurovascular coupling. Using a deep learning architecture, we performed a thorough analysis of each EEG frequency range and its complementary fNIRS prediction; further we analyzed functional connectivity between brain regions using frequency range predictions. We noted that higher EEG frequency bands provided hemodynamic predictions with the highest metrics.

Sharing the data used in this study is bound by the ethics of the institutional review boards of Sainte-Justine Hospital and Centre Hospitalier de l’Université de Montréal which approved the study. The custom code used in this study is available upon reasonable request.

Acknowledgements

This project was generously supported by The Natural Sciences and Engineering Research Council of Canada grant NSERC: NSERC, 239876-and Canadian Institutes of Health Research grant 87183.

Declarations

Conflict of Interest

The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose.

Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

next article Modelling Cortical Laminar Connectivity in the Macaque Brain

Aalkjær, C., Boedtkjer, D., & Matchkov, V. (2011). Vasomotion – what is currently thought? In Acta Physiologica. https://doi.org/10.1111/j.1748-1716.2011.02320.x

Aasted, C. M., Yücel, M. A., Cooper, R. J., Dubb, J., Tsuzuki, D., Becerra, L., Petkov, M. P., Borsook, D., Dan, I., & Boas, D. A. (2015). Anatomical guidance for functional near-infrared spectroscopy: AtlasViewer tutorial. Neurophotonics. https://doi.org/10.1117/1.nph.2.2.020801CrossRefPubMedPubMedCentral

Amzica, F., & Steriade, M. (1998). Electrophysiological correlates of sleep delta waves. Electroencephalography and Clinical Neurophysiology, 107(2), 69–83.CrossRef

Barlow, J. S., Kamp, A., Morton, H. B., Ripoche, A., Shipton, H., & Tchavdarov, D. B. (1974). EEG instrumentation standards: Report of the Committee on EEG Instrumentation Standards of the International Federation of Societies for Electroencephalography and Clinical Neurophysiology. Electroencephalography and Clinical Neurophysiology, 37(5), 549–553. https://doi.org/10.1016/0013-4694(74)90100-XCrossRef

Baytas, I. M., Xiao, C., Zhang, X., Wang, F., Jain, A. K., & Zhou, J. (2017). Patient subtyping via time-aware LSTM networks. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. https://doi.org/10.1145/3097983.3097997CrossRef

Bettus, G., Guedj, E., Joyeux, F., Confort-Gouny, S., Soulier, E., Laguitton, V., Cozzone, P. J., Chauvel, P., Ranjeva, J. P., Bartolomei, F., & Guye, M. (2009). Decreased basal fMRI functional connectivity in epileptogenic networks and contralateral compensatory mechanisms. Human Brain Mapping, 30(5), 1580–1591. https://doi.org/10.1002/hbm.20625CrossRefPubMed

Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Kirsch, H. E., Berger, M. S., Barbare, N. M., & Knight, R. T. (2006). High gamma power is phase-locked to theta oscillations in human neocortex. Science. https://doi.org/10.1126/science.1128115CrossRefPubMedPubMedCentral

Chiarelli, A. M., Zappasodi, F., Di Pompeo, F., & Merla, A. (2017). Simultaneous functional near-infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: A review. Neurophotonics. https://doi.org/10.1117/1.nph.4.4.041411CrossRefPubMedPubMedCentral

Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. https://doi.org/10.3115/v1/d14-1179

Chung, Y. A., Wu, C. C., Shen, C. H., Lee, H. Y., & Lee, L. S. (2016). Audio Word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. https://doi.org/10.21437/Interspeech.2016-82

Citerio, G., Oddo, M., & Taccone, F. S. (2015). Recommendations for the use of multimodal monitoring in the neurointensive care unit. In Current Opinion in Critical Care (Vol. 21, Issue 2, pp. 113–119). Lippincott Williams and Wilkins. https://doi.org/10.1097/MCC.0000000000000179

Connolly, M., Vespa, P., Pouratian, N., Gonzalez, N. R., & Hu, X. (2015). Characterization of the relationship between intracranial pressure and electroencephalographic monitoring in burst-suppressed patients. Neurocritical Care, 22(2), 212–220. https://doi.org/10.1007/s12028-014-0059-8CrossRefPubMedPubMedCentral

Curtin, A., Tong, S., Sun, J., Wang, J., Onaral, B., & Ayaz, H. (2019). A systematic review of integrated functional near-infrared spectroscopy (fNIRS) and transcranial magnetic stimulation (TMS) studies. Frontiers in Neuroscience. https://doi.org/10.3389/fnins.2019.00084CrossRefPubMedPubMedCentral

Czisch, M., Wehrle, R., Kaufmann, C., Wetter, T. C., Holsboer, F., Pollmächer, T., & Auer, D. P. (2004). Functional MRI during sleep: BOLD signal decreases and their electrophysiological correlates. European Journal of Neuroscience. https://doi.org/10.1111/j.1460-9568.2004.03518.xCrossRefPubMed

Dash, D., Dash, C., Primrose, S., Hernandez-Ronquillo, L., Moien-Afshari, F., Ladino, L. D., Appendino, J. P., Mazepa, L., Elliott, C., Mirsattari, S. M., Federico, P., Bui, E., Hunter, G., Ramachandran Nair, R., Sharma, R., Melendres, P., Nikkel, J., Nguyen, D. K., Almubarak, S., … Téllez-Zenteno, J. F. (2017). Update on minimal standards for electroencephalography in Canada: A review by the Canadian Society of Clinical Neurophysiologists. In Canadian Journal of Neurological Sciences (Vol. 44, Issue 6, pp. 631–642). Cambridge University Press. https://doi.org/10.1017/cjn.2017.217

De Luca, M., Beckmann, C. F., De Stefano, N., Matthews, P. M., & Smith, S. M. (2006). fMRI resting state networks define distinct modes of long-distance interactions in the human brain. NeuroImage, 29(4), 1359–1367. https://doi.org/10.1016/j.neuroimage.2005.08.035CrossRefPubMed

DeBettencourt, M. T., Cohen, J. D., Lee, R. F., Norman, K. A., & Turk-Browne, N. B. (2015). Closed-loop training of attention with real-time brain imaging. Nature Neuroscience. https://doi.org/10.1038/nn.3940CrossRefPubMedPubMedCentral

Deco, G., Jirsa, V. K., & McIntosh, A. R. (2011). Emerging concepts for the dynamical organization of resting-state activity in the brain. In Nature Reviews Neuroscience (Vol. 12, Issue 1, pp. 43–56). Nature Publishing Group. https://doi.org/10.1038/nrn2961

Dhamija, R., Moseley, B. D., Cascino, G. D., & Wirrell, E. C. (2011). A population-based study of long-term outcome of epilepsy in childhood with a focal or hemispheric lesion on neuroimaging. Epilepsia, 52(8), 1522–1526. https://doi.org/10.1111/j.1528-1167.2011.03192.xCrossRefPubMedPubMedCentral

Dutta, A., Jacob, A., Chowdhury, S. R., Das, A., & Nitsche, M. A. (2015). EEG-NIRS based assessment of neurovascular coupling during anodal transcranial direct current stimulation - a stroke case series. Journal of Medical Systems, 39(4), 1–9. https://doi.org/10.1007/s10916-015-0205-7CrossRef

Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., Galuske, R. A. W., & Niessing, J. (2005). Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science.

Fair, D. A., Nigg, J. T., Iyer, S., Bathula, D., Mills, K. L., Dosenbach, N. U. F., Schlaggar, B. L., Mennes, M., Gutman, D., Bangaru, S., Buitelaar, J. K., Dickstein, D. P., Martino, A. D., Kennedy, D. N., Kelly, C., Luna, B., Schweitzer, J. B., Velanova, K., Wang, Y. F., & Milham, M. P. (2013). Distinct neural signatures detected for ADHD subtypes after controlling for micro-movements in resting state functional connectivity MRI data. Frontiers in Systems Neuroscience. https://doi.org/10.3389/fnsys.2012.00080CrossRefPubMedPubMedCentral

Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K. R., & Blankertz, B. (2012). Enhanced performance by a hybrid NIRS-EEG brain computer interface. NeuroImage, 59(1), 519–529. https://doi.org/10.1016/j.neuroimage.2011.07.084CrossRefPubMed

Fishburn, F. A., Norr, M. E., Medvedev, A. V., & Vaidya, C. J. (2014). Sensitivity of fNIRS to cognitive state and load. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2014.00076CrossRefPubMedPubMedCentral

Foreman, B., & Claassen, J. (2012). Quantitative EEG for the detection of brain ischemia. Critical Care. https://doi.org/10.1186/cc11230CrossRefPubMedPubMedCentral

Franaszczuk, P. J., Kudela, P., & Bergey, G. K. (2003). External excitatory stimuli can terminate bursting in neural network models. Epilepsy Research, 53(1–2), 65–80. https://doi.org/10.1016/S0920-1211(02)00248-6CrossRefPubMed

Freeman, W. J., Burke, B. C., & Holmes, M. D. (2003). Aperiodic phase re-setting in scalp EEG of beta-gamma oscillations by state transitions at alpha-theta rates. Human Brain Mapping. https://doi.org/10.1002/hbm.10120CrossRefPubMedPubMedCentral

Gagnon, L., Cooper, R. J., Yücel, M. A., Perdue, K. L., Greve, D. N., & Boas, D. A. (2012). Short separation channel location impacts the performance of short channel regression in NIRS. NeuroImage. https://doi.org/10.1016/j.neuroimage.2011.08.095CrossRefPubMed

Geng, S., Liu, X., Biswal, B. B., & Niu, H. (2017). Effect of resting-state fNIRS scanning duration on functional brain connectivity and graph theory metrics of brain network. Frontiers in Neuroscience, 11(JUL). https://doi.org/10.3389/fnins.2017.00392

Girouard, H., & Iadecola, C. (2006). Neurovascular coupling in the normal brain and in hypertension, stroke, and Alzheimer disease. Journal of Applied Physiology, 100(1), 328–335.CrossRef

Goldman, R. I., Stern, J. M., Engel, J., Jr., & Cohen, M. S. (2002). Simultaneous EEG and fMRI of the alpha rhythm. NeuroReport, 13(18), 2487.CrossRef

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.

Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., Parkkonen, L., & Hämäläinen, M. S. (2014). MNE software for processing MEG and EEG data. NeuroImage. https://doi.org/10.1016/j.neuroimage.2013.10.027CrossRefPubMed

Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2016.2582924CrossRefPubMed

Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. 32nd International Conference on Machine Learning, ICML 2015.

Gross, D. W., & Gotman, J. (1999). Correlation of high-frequency oscillations with the sleep-wake cycle and cognitive activity in humans. Neuroscience. https://doi.org/10.1016/S0306-4522(99)00343-7CrossRefPubMed

He, B., & Liu, Z. (2008). Multimodal functional neuroimaging: integrating functional MRI and EEG/MEG. IEEE Reviews in Biomedical Engineering, 1(2008), 23–40. https://doi.org/10.1109/RBME.2008.2008233CrossRefPubMedPubMedCentral

Helbok, R., & Claassen, J. (2013). Multimodal invasive monitoring in status epilepticus: What is the evidence it has a place? Epilepsia, 54(SUPPL. 6), 57–60. https://doi.org/10.1111/epi.12279CrossRefPubMed

Hofle, N., Paus, T., Reutens, D., Fiset, P., Gotman, J., Evans, A. C., & Jones, B. E. (1997). Regional cerebral blood flow changes as a function of delta and spindle activity during slow wave sleep in humans. Journal of Neuroscience. https://doi.org/10.1523/jneurosci.17-12-04800.1997CrossRefPubMed

Honda, Y., Nakamura, S., Ogawa, K., Yoshino, R., Tobler, P. N., Nishimura, Y., & Tsutsui, K. I. (2021). Changes in beta and high-gamma power in resting-state electrocorticogram induced by repetitive transcranial magnetic stimulation of primary motor cortex in unanesthetized macaque monkeys. Neuroscience Research. https://doi.org/10.1016/j.neures.2021.02.002CrossRefPubMed

Hong, K. S., Naseer, N., & Kim, Y. H. (2015). Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neuroscience Letters, 587, 87–92. https://doi.org/10.1016/j.neulet.2014.12.029CrossRefPubMed

Hughes, J. R. (2008). Gamma, fast, and ultrafast waves of the brain: Their relationships with epilepsy and behavior. In Epilepsy and Behavior (Vol. 13, Issue 1, pp. 25–31). Academic Press. https://doi.org/10.1016/j.yebeh.2008.01.011

Huppert, T. J., Diamond, S. G., Franceschini, M. A., & Boas, D. A. (2009). HomER: A review of time-series analysis methods for near-infrared spectroscopy of the brain. Applied Optics, 48(10), D280–D298.CrossRef

Jia, X., & Kohn, A. (2011). Gamma rhythms in the brain. PLoS Biology, 9(4), 2–5. https://doi.org/10.1371/journal.pbio.1001045CrossRef

Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198(4323), 1264–1267.CrossRef

Julien, C. (2006). The enigma of Mayer waves: Facts and models. Cardiovascular Research. https://doi.org/10.1016/j.cardiores.2005.11.008CrossRefPubMed

Khan, M. J., & Hong, K. -S. (2015). Passive BCI based on drowsiness detection: An fNIRS study. Biomedical Optics Express, 6(10), 4063. https://doi.org/10.1364/boe.6.004063CrossRefPubMedPubMedCentral

Khan, M. J., & Hong, K. S. (2017). Hybrid EEG-FNIRS-based eight-command decoding for BCI: Application to quadcopter control. Frontiers in Neurorobotics, 11, 6. https://doi.org/10.3389/fnbot.2017.00006CrossRefPubMedPubMedCentral

Knyazev, G. G. (2012). EEG delta oscillations as a correlate of basic homeostatic and motivational processes. Neuroscience & Biobehavioral Reviews, 36(1), 677–695.CrossRef

Kobayashi, E., Bagshaw, A. P., Grova, C., Dubeau, F., & Gotman, J. (2006). Negative BOLD responses to epileptic spikes. Human Brain Mapping, 27(6), 488–497. https://doi.org/10.1002/hbm.20193CrossRefPubMed

Koch, S. P., Koendgen, S., Bourayou, R., Steinbrink, J., & Obrig, H. (2008). Individual alpha-frequency correlates with amplitude of visual evoked potential and hemodynamic response. NeuroImage, 41(2), 233–242.CrossRef

Kocsis, L., Herman, P., & Eke, A. (2006). The modified Beer-Lambert law revisited. Physics in Medicine and Biology, 51(5). https://doi.org/10.1088/0031-9155/51/5/N02

Kohno, S., Miyai, I., Seiyama, A., Oda, I., Ishikawa, A., Tsuneishi, S., Amita, T., & Shimizu, K. (2007). Removal of the skin blood flow artifact in functional near-infrared spectroscopic imaging data through independent component analysis. Journal of Biomedical Optics. https://doi.org/10.1117/12814249CrossRefPubMed

Kotliar, K., Hauser, C., Ortner, M., Muggenthaler, C., Diehl-Schmid, J., Angermann, S., Hapfelmeier, A., Schmaderer, C., & Grimmer, T. (2017). Altered neurovascular coupling as measured by optical imaging: A biomarker for Alzheimer’s disease. Scientific Reports, 7(1), 1–11. https://doi.org/10.1038/s41598-017-13349-5CrossRef

Laufs, H., Krakow, K., Sterzer, P., Eger, E., Beyerle, A., Salek-Haddadi, A., & Kleinschmidt, A. (2003). Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.1831638100CrossRefPubMedPubMedCentral

Le Roux, P. (2013). Physiological monitoring of the severe traumatic brain injury patient in the intensive care unit. Current Neurology and Neuroscience Reports, 13(3), 331.CrossRef

Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature. https://doi.org/10.1038/nature14539CrossRefPubMed

Lemieux, L., Krakow, K., & Fish, D. R. (2001). Comparison of spike-triggered functional MRI BOLD activation and EEG dipole model localization. NeuroImage. https://doi.org/10.1006/nimg.2001.0896CrossRefPubMed

Li, Z., Liu, H., Liao, X., Xu, J., Liu, W., Tian, F., He, Y., & Niu, H. (2015). Dynamic functional connectivity revealed by resting-state functional near-infrared spectroscopy. Biomedical Optics Express. https://doi.org/10.1364/boe.6.002337CrossRefPubMedPubMedCentral

Lindauer, U., Dirnagl, U., Füchtemeier, M., Böttiger, C., Offenhauser, N., Leithner, C., & Royl, G. (2010). Pathophysiological interference with neurovascular coupling–when imaging based on hemoglobin might go blind. Frontiers in Neuroenergetics, 2, 25.CrossRef

Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2016). Learning to diagnose with LSTM recurrent neural networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings.

Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature. https://doi.org/10.1038/35084005CrossRefPubMed

Lövblad, K. O., Thomas, R., Jakob, P. M., Scammell, T., Bassetti, C., Griswold, M., Ives, J., Matheson, J., Edelman, R. R., & Warach, S. (1999). Silent functional magnetic resonance imaging demonstrates focal activation in rapid eye movement sleep. Neurology. https://doi.org/10.1212/wnl.53.9.2193CrossRefPubMed

Lu, C. M., Zhang, Y. J., Biswal, B. B., Zang, Y. F., Peng, D. L., & Zhu, C. Z. (2010). Use of fNIRS to assess resting state functional connectivity. Journal of Neuroscience Methods, 186(2), 242–249. https://doi.org/10.1016/j.jneumeth.2009.11.010CrossRefPubMed

Luo, C., Li, Q., Lai, Y., Xia, Y., Qin, Y., Liao, W., Li, S., Zhou, D., Yao, D., & Gong, Q. (2011). Altered functional connectivity in default mode network in absence epilepsy: A resting-state fMRI study. Human Brain Mapping, 32(3), 438–449. https://doi.org/10.1002/hbm.21034CrossRefPubMed

Luong, M. -T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2015). Multi-task Sequence to Sequence Learning. c, 1–10. http://arxiv.org/abs/1511.06114

Maas, A. I. R., Menon, D. K., Lingsma, H. F., Pineda, J. A., Sandel, M. E., & Manley, G. T. (2012). Re-orientation of clinical research in traumatic brain injury: Report of an international workshop on comparative effectiveness research. Journal of Neurotrauma, 29(1), 32–46. https://doi.org/10.1089/neu.2010.1599CrossRefPubMedPubMedCentral

Mantini, D., Perrucci, M. G., Del Gratta, C., Romani, G. L., & Corbetta, M. (2007). Electrophysiological signatures of resting state networks in the human brain. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.0700668104CrossRefPubMedPubMedCentral

Martinez-Montes, E., Valdes-Sosa, P. A., Miwakeichi, F., Goldman, R. I., & Cohen, M. S. (2004). Concurrent EEG/fMRI analysis by multiway partial least squares. NeuroImage, 22(3), 1023–1034. https://doi.org/10.1016/j.neuroimage.2004.03.038CrossRefPubMed

McKenna, T. M., McMullen, T. A., & Shlesinger, M. F. (1994). The brain as a dynamic physical system. Neuroscience, 60(3), 587–605. https://doi.org/10.1016/0306-4522(94)90489-8CrossRefPubMed

Merity, S., Keskar, N. S., & Socher, R. (2018). Regularizing and optimizing LSTM language models. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.

Merker, B. H. (2016). Cortical gamma oscillations: Details of their genesis preclude a role in cognition. Frontiers in Computational Neuroscience. https://doi.org/10.3389/fncom.2016.00078CrossRefPubMedPubMedCentral

Mikolov, T., Karafiát, M., Burget, L., Jan, C., & Khudanpur, S. (2010). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010.

Miller, C. M. (2012). Update on multimodality monitoring. Current Neurology and Neuroscience Reports. https://doi.org/10.1007/s11910-012-0274-7CrossRefPubMed

Moshé, S. L., Perucca, E., Ryvlin, P., & Tomson, T. (2015). Epilepsy: New advances. In The Lancet (Vol. 385, Issue 9971, pp. 884–898). Lancet Publishing Group. https://doi.org/10.1016/S0140-6736(14)60456-6

Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., & Malach, R. (2005). Neuroscience: Coupling between neuronal firing, field potentials, and fMRI in human auditory cortex. Science. https://doi.org/10.1126/science.1110913CrossRefPubMed

Müller-Putz, G. R. (2020). Electroencephalography. Handbook of Clinical Neurology. https://doi.org/10.1016/B978-0-444-63934-9.00018-4CrossRefPubMed

Murta, T., Leite, M., Carmichael, D. W., Figueiredo, P., & Lemieux, L. (2015). Electrophysiological correlates of the BOLD signal for EEG-informed fMRI. Human Brain Mapping, 36(1), 391–414. https://doi.org/10.1002/hbm.22623CrossRefPubMed

Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Théberge, J., Kluetsch, R. C., Jetly, R., & Lanius, R. A. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. NeuroImage: Clinical. https://doi.org/10.1016/j.nicl.2016.07.006

Niessing, J., Ebisch, B., Schmidt, K. E., Niessing, M., Singer, W., & Galuske, R. A. W. (2005). Neuroscience: Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science, 309(5736), 948–951. https://doi.org/10.1126/science.1110948CrossRefPubMed

Nikulin, V. V., Fedele, T., Mehnert, J., Lipp, A., Noack, C., Steinbrink, J., & Curio, G. (2014). Monochromatic Ultra-Slow (~ 0.1 Hz) Oscillations in the human electroencephalogram and their relation to hemodynamics. NeuroImage, 97, 71–80. https://doi.org/10.1016/J.NEUROIMAGE.2014.04.008CrossRefPubMed

Niu, H., & He, Y. (2014). Resting-state functional brain connectivity: Lessons from functional near-infrared spectroscopy. The Neuroscientist. https://doi.org/10.1177/1073858413502707CrossRefPubMed

Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Merkle, H., & Ugurbil, K. (1992). Intrinsic signal changes accompanying sensory stimulation: Functional brain mapping with magnetic resonance imaging. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.89.13.5951CrossRefPubMedPubMedCentral

Palva, J. M., Monto, S., Kulashekhar, S., & Palva, S. (2010). Neuronal synchrony reveals working memory networks and predicts individual memory capacity. Proceedings of the National Academy of Sciences of the United States of America, 107(16), 7580–7585. https://doi.org/10.1073/pnas.0913113107CrossRefPubMedPubMedCentral

Park, Y., Luo, L., Parhi, K. K., & Netoff, T. (2011). Seizure prediction with spectral power of EEG using cost-sensitive support vector machines. Epilepsia, 52(10), 1761–1770. https://doi.org/10.1111/j.1528-1167.2011.03138.xCrossRefPubMed

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research.

Peng, K., Nguyen, D. K., Tayah, T., Vannasing, P., Tremblay, J., Sawan, M., Lassonde, M., Lesage, F., & Pouliot, P. (2014). FNIRS-EEG study of focal interictal epileptiform discharges. Epilepsy Research. https://doi.org/10.1016/j.eplepsyres.2013.12.011CrossRefPubMed

Phillips, A. A., Squair, J. W., Sayenko, D. G., Edgerton, V. R., Gerasimenko, Y., & Krassioukov, A. V. (2018). An autonomic neuroprosthesis: Noninvasive electrical spinal cord stimulation restores autonomic cardiovascular function in individuals with spinal cord injury. Journal of Neurotrauma, 35(3), 446–451. https://doi.org/10.1089/neu.2017.5082CrossRefPubMedPubMedCentral

Pizzo, F., Frauscher, B., Ferrari-Marinho, T., Amiri, M., Dubeau, F., & Gotman, J. (2016). Detectability of fast ripples (>250 Hz) on the scalp EEG: A proof-of-principle study with subdermal electrodes. Brain Topography. https://doi.org/10.1007/s10548-016-0481-7CrossRefPubMed

Pressl, C., Brandner, P., Schaffelhofer, S., Blackmon, K., Dugan, P., Holmes, M., Thesen, T., Kuzniecky, R., Devinsky, O., & Freiwald, W. A. (2019). Resting state functional connectivity patterns associated with pharmacological treatment resistance in temporal lobe epilepsy. Epilepsy Research. https://doi.org/10.1016/j.eplepsyres.2018.11.002CrossRefPubMed

Quaresima, V., & Ferrari, M. (2019). A mini-review on functional near-infrared spectroscopy (fNIRS): Where do we stand, and where should we go? Photonics, 6(3), 87. https://doi.org/10.3390/photonics6030087CrossRef

Ray, S., & Maunsell, J. H. R. (2010). Differences in gamma frequencies across visual cortex restrict their possible use in computation. Neuron, 67(5), 885–896. https://doi.org/10.1016/j.neuron.2010.08.004CrossRefPubMedPubMedCentral

Richardson, M. P. (2012). Large scale brain models of epilepsy: Dynamics meets connectomics. Journal of Neurology, Neurosurgery and Psychiatry, 83(12), 1238–1248. https://doi.org/10.1136/jnnp-2011-301944CrossRefPubMed

Rojas, G. M., Alvarez, C., Montoya, C. E., de la Iglesia-Vayá, M., Cisternas, J. E., & Gálvez, M. (2018). Study of resting-state functional connectivity networks using EEG electrodes position as seed. Frontiers in Neuroscience. https://doi.org/10.3389/fnins.2018.00235CrossRefPubMedPubMedCentral

Ros, T., Baars, B. J., Lanius, R. A., & Vuilleumier, P. (2014). Tuning pathological brain oscillations with neurofeedback: A systems neuroscience framework. Frontiers in Human Neuroscience, 8, 1008. https://doi.org/10.3389/fnhum.2014.01008CrossRefPubMedPubMedCentral

Salek-Haddadi, A., Friston, K. J., Lemieux, L., & Fish, D. R. (2003). Studying spontaneous EEG activity with fMRI. Brain Research Reviews. https://doi.org/10.1016/S0165-0173(03)00193-0CrossRefPubMed

Sassaroli, A., Pierro, M., Bergethon, P. R., & Fantini, S. (2012). Low-frequency spontaneous oscillations of cerebral hemodynamics investigated with near-infrared spectroscopy: A review. IEEE Journal on Selected Topics in Quantum Electronics. https://doi.org/10.1109/JSTQE.2012.2183581CrossRef

Schmidt, J. M., & De Georgia, M. (2014). Multimodality Monitoring: Informatics, Integration Data Display and Analysis The Participants in the International Multidisciplinary Consensus Conference on Multimodality Monitoring. Neurocritical Care. https://doi.org/10.1007/s12028-014-0037-1

Scholkmann, F., Kleiser, S., Metz, A. J., Zimmermann, R., Pavia, J. M., Wolf, U., & Wolf, M. (2014). A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. NeuroImage, 85, 6–27.CrossRef

Schwab, K., Skupin, H., Eiselt, M., Walther, M., Voss, A., & Witte, H. (2009). Coordination of the EEG and the heart rate of preterm neonates during quiet sleep. Neuroscience Letters. https://doi.org/10.1016/j.neulet.2009.09.031CrossRefPubMed

Sharbrough, F. W., Messick, J. M., & Sundt, T. M. (1973). Correlation of continuous electroencephalograms with cerebral blood flow measurements during carotid endarterectomy. Stroke. https://doi.org/10.1161/01.STR.4.4.674CrossRefPubMed

Shen, H. H. (2015). Core concept: Resting-state connectivity. Proceedings of the National Academy of Sciences of the United States of America. https://doi.org/10.1073/pnas.1518785112CrossRefPubMedPubMedCentral

Shmuel, A., & Leopold, D. A. (2008). Neuronal correlates of spontaneous fluctuations in fMRI signals in monkey visual cortex: Implications for functional connectivity at rest. Human Brain Mapping, 29(7), 751–761. https://doi.org/10.1002/hbm.20580CrossRefPubMedPubMedCentral

Siegel, J. S., Snyder, A. Z., Ramsey, L., Shulman, G. L., & Corbetta, M. (2016). The effects of hemodynamic lag on functional connectivity and behavior after stroke. Journal of Cerebral Blood Flow and Metabolism, 36(12), 2162–2176. https://doi.org/10.1177/0271678X15614846CrossRefPubMed

Sigala, R., Haufe, S., Roy, D., Dinse, H. R., & Ritter, P. (2014). The role of alpha-rhythm states in perceptual learning: Insights from experiments and computational models. Frontiers in Computational Neuroscience, 8, 1–19. https://doi.org/10.3389/fncom.2014.00036CrossRef

Singh, K. D. (2012). Which “neural activity” do you mean? fMRI, MEG, oscillations and neurotransmitters. NeuroImage. https://doi.org/10.1016/j.neuroimage.2012.01.028CrossRefPubMed

Sirpal, P., Kassab, A., Pouliot, P., & Nguyen, D. K. (2019). fNIRS improves seizure detection in multimodal EEG-fNIRS recordings. Journal of Biomedical Optics, 24(05), 1. https://doi.org/10.1117/1.jbo.24.5.051408CrossRefPubMed

Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis-Peacock, J., Weiskopf, N., Blefari, M. L., Rana, M., Oblak, E., Birbaumer, N., & Sulzer, J. (2017). Closed-loop brain training: The science of neurofeedback. Nature Reviews Neuroscience. https://doi.org/10.1038/nrn.2016.164CrossRefPubMed

Sitnikova, E., Hramov, A. E., Grubov, V., & Koronovsky, A. A. (2016). Rhythmic activity in EEG and sleep in rats with absence epilepsy. Brain Research Bulletin. https://doi.org/10.1016/j.brainresbull.2015.11.012CrossRefPubMed

Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.

Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015). Unsupervised learning of video representations using LSTMs. 32nd International Conference on Machine Learning, ICML 2015.

Strangman, G., Boas, D. A., & Sutton, J. P. (2002). Non-invasive neuroimaging using near-infrared light. Biological Psychiatry. https://doi.org/10.1016/S0006-3223(02)01550-0CrossRefPubMed

Sumiyoshi, A., Suzuki, H., Ogawa, T., Riera, J. J., Shimokawa, H., & Kawashima, R. (2012). Coupling between gamma oscillation and fMRI signal in the rat somatosensory cortex: Its dependence on systemic physiological parameters. NeuroImage. https://doi.org/10.1016/j.neuroimage.2011.12.082CrossRefPubMed

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 4(January), 3104–3112.

Thair, H., Holloway, A. L., Newport, R., & Smith, A. D. (2017). Transcranial direct current stimulation (tDCS): A Beginner’s guide for design and implementation. Frontiers in Neuroscience, 11, 641. https://doi.org/10.3389/fnins.2017.00641CrossRefPubMedPubMedCentral

Tong, Y., Hocke, L. M., Licata, S. C., & Frederick, B. D. (2012). Low-frequency oscillations measured in the periphery with near-infrared spectroscopy are strongly correlated with blood oxygen level-dependent functional magnetic resonance imaging signals. Journal of Biomedical Optics, 17(10), 1060041. https://doi.org/10.1117/1.jbo.17.10.106004CrossRef

Tracy, J. I., & Doucet, G. E. (2015). Resting-state functional connectivity in epilepsy: Growing relevance for clinical decision making. Current Opinion in Neurology. https://doi.org/10.1097/WCO.0000000000000178CrossRefPubMed

Truong, N. D., Kuhlmann, L., Bonyadi, M. R., & Kavehei, O. (2018). Semi-supervised Seizure Prediction with Generative Adversarial Networks. 1–6. http://arxiv.org/abs/1806.08235

Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning. https://doi.org/10.1145/1390156.1390294

Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research.

Wang, J., Dong, Q., & Niu, H. (2017). The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports, 7(1). https://doi.org/10.1038/s41598-017-06340-7

Wang, Y., Huang, M., Zhao, L., & Zhu, X. (2016). Attention-based LSTM for aspect-level sentiment classification. EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings. https://doi.org/10.18653/v1/d16-1058

Wang, Z., Lu, G., Zhang, Z., Zhong, Y., Jiao, Q., Zhang, Z., Tan, Q., Tian, L., Chen, G., Liao, W., Li, K., & Liu, Y. (2011). Altered resting state networks in epileptic patients with generalized tonic-clonic seizures. Brain Research, 1374, 134–141. https://doi.org/10.1016/j.brainres.2010.12.034CrossRefPubMed

Welch, P. D. (1967). The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms. IEEE Transactions on Audio and Electroacoustics. https://doi.org/10.1109/TAU.1967.1161901

Whittingstall, K., & Logothetis, N. K. (2009). Frequency-band coupling in surface EEG reflects spiking activity in monkey visual cortex. Neuron. https://doi.org/10.1016/j.neuron.2009.08.016CrossRefPubMed

Woermann, F. G., & Vollmar, C. (2009). Clinical MRI in children and adults with focal epilepsy: A critical review. Epilepsy and Behavior, 15(1), 40–49. https://doi.org/10.1016/j.yebeh.2009.02.032CrossRefPubMed

Yücel, M. A., Selb, J., Aasted, C. M., Lin, P.-Y., Borsook, D., Becerra, L., & Boas, D. A. (2016). Mayer waves reduce the accuracy of estimated hemodynamic response functions in functional near-infrared spectroscopy. Biomedical Optics Express, 7(8), 3078. https://doi.org/10.1364/boe.7.003078CrossRefPubMedPubMedCentral

Zhang, H., Zhang, Y. J., Lu, C. M., Ma, S. Y., Zang, Y. F., & Zhu, C. Z. (2010a). Functional connectivity as revealed by independent component analysis of resting-state fNIRS measurements. NeuroImage. https://doi.org/10.1016/j.neuroimage.2010.02.080CrossRefPubMed

Zhang, W. (2018). Generative Adversarial Nets for Information Retrieval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 1375–1378. https://doi.org/10.1145/3209978.3210184

Zhang, X., Pan, W. J., & Keilholz, S. D. (2020). The relationship between BOLD and neural activity arises from temporally sparse events. NeuroImage, 207, 116390. https://doi.org/10.1016/j.neuroimage.2019.116390CrossRefPubMed

Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Liao, W., Wang, Z., Wang, Z., Li, K., Chen, H., & Liu, Y. (2010b). Altered spontaneous neuronal activity of the default-mode network in mesial temporal lobe epilepsy. Brain Research, 1323, 152–160. https://doi.org/10.1016/j.brainres.2010.01.042CrossRefPubMed

Zhang, Z., Lu, G., Zhong, Y., Tan, Q., Yang, Z., Liao, W., Chen, Z., Shi, J., & Liu, Y. (2009). Impaired attention network in temporal lobe epilepsy: A resting FMRI study. Neuroscience Letters, 458(3), 97–101. https://doi.org/10.1016/j.neulet.2009.04.040CrossRefPubMed

Zhao, F., Feng, J., Zhao, J., Yang, W., & Yan, S. (2018). Robust LSTM-Autoencoders for Face De-Occlusion in the Wild. IEEE Transactions on Image Processing. https://doi.org/10.1109/TIP.2017.2771408CrossRefPubMed

Title: Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals
Authors: Parikshat Sirpal
Rafat Damseh
Ke Peng
Dang Khoa Nguyen
Frédéric Lesage
Publication date: 10-08-2021
Publisher: Springer US
Published in: Neuroinformatics / Issue 3/2022
Print ISSN: 1539-2791
Electronic ISSN: 1559-0089
DOI: https://doi.org/10.1007/s12021-021-09538-3

Springer Professional

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Abstract

Publisher's Note

Introduction

Methods

Subjects and Protocol

EEG-fNIRS Data Acquisition and Pre-Processing

Neural Network Architecture

Training Details

Model Validation

Model Predictions

Functional Connectivity Validation

Results

Full Spectrum EEG Performance and Feature Analysis

Intra-patient Reconstructions on Separate Recording Sessions

Spatial Variability of Reconstructions

EEG Frequency Decomposition and Resting State Predictions

Functional Connectivity Results

Discussion

Conclusion

Acknowledgements

Declarations

Conflict of Interest

Publisher's Note

Premium Partner

Springer Professional

Abstract

Publisher's Note

Introduction

Methods

Subjects and Protocol

EEG-fNIRS Data Acquisition and Pre-Processing

Neural Network Architecture

Training Details

Model Validation

Model Predictions

Functional Connectivity Validation

Results

Full Spectrum EEG Performance and Feature Analysis

Intra-patient Reconstructions on Separate Recording Sessions

Spatial Variability of Reconstructions

EEG Frequency Decomposition and Resting State Predictions

Functional Connectivity Results

Discussion

Conclusion

Information Sharing Statement

Acknowledgements

Declarations

Conflict of Interest

Publisher's Note

Other articles of this Issue 3/2022

Design and Application of Automated Algorithms for Diagnosis and Treatment Optimization in Neurodegenerative Diseases

3dSpAn: An interactive software for 3D segmentation and analysis of dendritic spines

A Practical Workflow for Organizing Clinical Intraoperative and Long-term iEEG Data in BIDS

nGauge: Integrated and Extensible Neuron Morphology Analysis in Python

Shamo: A Tool for Electromagnetic Modeling, Simulation and Sensitivity Analysis of the Head

Correction to: Reproducible Inter-Personal Brain Coupling Measurements in Hyperscanning Settings With functional Near Infra-Red Spectroscopy

Premium Partner