Deep-MEG: spatiotemporal CNN features and multiband ensemble classification for predicting the early signs of Alzheimer’s disease with magnetoencephalography

Deep convolutional neural networks (CNNs) have become very popular in recent years thanks to their ability to decode images [1], video streams [2], and other biomedical signals [3], including 2D and 3D neuroimaging data [4]. The use of CNNs, in fact, offers the possibility to recognize the presence of patterns that other techniques are not able to reveal. In clinical scenarios, when data availability is limited, transfer learning (TL) can be applied to transfer the knowledge, previously learned by the CNN, for solving new problems faster or with different learning solutions [5, 6]. By combining the merit of multiple classifiers, ensemble learning can be an additional powerful instrument to improve the performance of predictive models [7‐10].

Thanks to these advantages, the use of CNNs is becoming predominant for the analysis of many types of biomedical images, in particular for decoding encephalographic signals, in which it is important to recognize the various neural activation patterns in relation to diseases. Many studies [11‐14] have used deep learning to 1D electroencephalography (EEG) signals to reach this goal, looking for solutions and architectures of different neural networks in order to extract the most discriminating features.

In recent years, the analysis of brain activity has been extended to an innovative technique, known as magnetoencephalography (MEG) [15‐18]. MEG is a powerful non-invasive diagnostic tool that possesses the unique advantage of providing a direct measure of the neural activity of the pyramidal neurons in the brain, ensuring high spatial and temporal resolutions (of order of mm and ms, respectively), and a fast preparation time [18]. A set of MEG recordings, together with the positions of the corresponding sources, encompass complex high-dimensional information on the brain network functioning, which can be difficult to uncover via standard methodologies, as for the case of the Alzheimer’s disease (AD).

AD is a neurodegenerative disorder and the most common form of dementia worldwide [19]. AD may start decades before the symptoms occur and then gradually evolve, with progressive alteration of cognitive and functional abilities. A precursory condition to AD, named mild cognitive impairment (MCI), is known to indicate a deviation from normal aging and an increased risk of developing dementia in future [20]. MCI, which can be caused by disorders other than AD (such as frontotemporal dementia), can remain a stable condition over time (stable MCI or sMCI), or finally progress to AD (progressive MCI or pMCI). The full-blown AD is a disabling condition resulting from the synaptic disruption of local and large-scale networks of the brain for which there is no cure. Finding new methods to detect pre-symptomatic or prodromal phases, i.e., pMCI, and predict earlier their progression toward AD would facilitate the timely implementation of therapeutic strategies [26].

To date, the most effective approaches for early AD diagnosis involve the use of invasive techniques such as the cerebrospinal fluid analysis [21] or the positron emission tomography (PET) [22, 23], which require performing a lumbar puncture or the use of radioactive tracers, respectively. Non-invasive diagnostic tools are being explored as alternatives [24‐28], with MEG representing a promising technique to be taken into account [15‐18]. In the following section, we report an overview of the state-of-the-art methods relative to the analysis of MEG data, with particular reference to the early AD diagnosis.

A schematic representation of the pipeline of the proposed platform is given in Fig. 1. The main characteristics of the methods are as follows:

In this section, the obtained results are presented for different classification scenarios. First, we report the results obtained for the classification of MEG recordings of the MCI subjects as sMCI or pMCI with respect to two classification approaches: (1) individual Deep-MEG classifiers based on different FC maps and on the RGB images (2) Deep-MEG ensemble #1. Finally, for the early detection of AD within a more complete scenario also including HC subjects, the results obtained with ensemble #2 are reported. Results are labeled as post-conversion when the MCI data include pMCI patients who met the criteria for probable AD and as pre-conversion when the pMCI patients were still clinically undistinguishable from the sMCI patients.

We have presented a deep-feature transfer approach, named Deep-MEG, and a set of ensemble classification architectures for decoding MEG recordings based on a new visual perspective on FC for the early diagnosis of AD. Image-based representations of FC were derived starting from the MEG time series. The MEG signals were first processed and filtered to derive meaningful data for FC analysis and to quantify the spatiotemporal characteristics of the brain connectome, in conjunction with the anatomical information encoded in the MRI scans, as described in Sects. 3.1 and 3.2. Different versions of the \({\text{PLV}}\) and \({\text{MC}}\) indices were computed as FC descriptors and organized as RGB images, also relative to multiple frequency bands (see Sects. 3.3 and 3.4). Such images could be received as input data by the pre-trained CNN and pooling layers in the AlexNet network, used as feature extractors and decoders of FC patterns, as described in Sect. 3.5. Cooperative decision architectures among Deep-MEG modules allowed the integration of the brain signaling at multiple levels of frequency bands to derive increased performance (see Sect. 3.7).

The main novelty of the proposed study is the analysis of the MEG-FC patterns of the brain network via deep CNNs. We have shown that the information on the hypo- or hyper-synchronization is conveyed not only by the FC values, but it is also embedded in their spatial arrangement as FC maps, which gave us additional information on the connectome disruption related to AD. Individual Deep-MEG modules (see Sect. 4.1) allowed the discrimination of sMCI and pMCI patients at the post- and pre-conversion stages, with accuracy values of 0.87 and 0.78, respectively, using the \({\text{PLV}}\_{\text{cent}}\) in the α band and the \({\text{MC}}\_{\text{cent}}_{{{\text{am}}}} \) in the β₁ band. For the binary classification of MEG recordings as HC or MCI, using a single Deep-MEG module based on the \({\text{PLV}}_{{{\text{cen}}}}\) map in the θ band, we obtained an accuracy of 0.80.

A composed image, named RGB, was designed to encode multiple levels of information also avoiding redundancies. The new set of deep features so extracted, boosted the classification performance at the pre-conversion stage to 0.89. In this scenario, we found that when data were integrated in different color channels of a single image, the encoded information has to be similar and homogeneous to guarantee appropriate decoding by the CNNs, i.e., indicators relative to the β and β₁ bands.

As the connectivity patterns of different frequency bands were unique and differently informative in terms of activation patterns, multiband ensemble classifiers were used to integrate the information encoded in different image-based representations (see Sect. 4.2). By averaging the probability scores of the best image representations of FC at the decision level, increased accuracy values, i.e., 0.93 and 0.83 for the post- and pre-conversion stages, respectively, were obtained for the binary classification of MEG recordings as sMCI or pMCI. These results showed the fundamental role of integrating heterogeneous and diverse data (at the spatiotemporal and frequency levels) for better representation and decoding of the brain functional connectivity.

In our experiments in the three-class scenario also including the HC cases, none of the single image-based representations of FC was effective, neither was ensemble #1 among Deep-MEG modules based on LDA or SVM classifiers. More importantly, it was not possible to detect the pMCI cases at the pre-conversion stage, when also HC cases were present, i.e., the discrimination of pMCI cases from the HC and sMCI, that is the main goal for early detection of AD. The reasons of this finding may lie in the fact that the dynamic activity of the brain network in relation to diverse clinical conditions possesses diverse manifestations in terms of spatiotemporal and frequency responses and that the information relevant to each binary sub-problem are encoded into different frequency bands or FC indicators. Therefore, we used another ensemble logic, named ensemble #2, in which two classification modules solve the two different sub-tasks, as described in Sect. 3.6. Finally, we used an AND Logic to aggregate the assessment of the individual predictors. For the three-class results, accuracy value of 0.74 was obtained. The best discriminated class was the pMCI, with a percentage of detection of 0.82. The sMCI cases were mostly confounded with the HC cases. As the sMCI possess different severity levels, which may lie on a continuum from the cognitive perspective [68], different levels of cognitive impairments, in turn, may be associated with different FC connectivity patterns, some of which resulted to be similar to those of the HC cases. This is not surprising, especially considering the stability in terms of AD conversion of the sMCI cases over the three years of observation. When the HC and sMCI were considered as a single class (Fig. 7c), the accuracy of classification of ensemble #2 increased up to 0.87, maintaining a sensitivity for the pMCI class at 0.82. Ensemble #2 is the result of cooperation of only two Deep-MEG-based classifiers trained, on average, with eight deep features automatically selected during the rounds of LOPO cross-validation from the RGB images and the \({\text{PLV}}_{{{\text{cent}}}}\) map in the θ band. Data diversity and cooperation at the decision level were crucial to boost the recognition of pMCI cases at the pre-conversion stage and discriminate them from the HC and sMCI cases.

We have also shown that the proposed combination of deep spatiotemporal features and multiband ensemble classification showed superior performance as compared to other existing methods, including the standard approach to the analysis of FC, in which the FC indicators, also relative to different frequency bands, are used as feature descriptors. This was verified with respect to multiple combinations of FC features, even when the best FC features were selected on the complete pool based on their individual AUC values.

We tested multiple combinations of image representations of FC, and the higher results are reported in this study. A major advantage of this approach is that the learned models can be interpreted in neurophysiological terms. The results obtained with the present study support the notion of different functional brain connectivity patterns associated with different rates of progression and conversion to AD [68]. In line with other work in the literature [15, 16, 39‐44], we have observed the role of specific frequency bands as potential biomarkers for the different phases of progression of the disease. In particular, a different set of FC images and frequency bands was determinant for the two ensemble classifiers relative to the post- and pre-conversion phases. In fact, at the post-conversion stage, our results indicate that the phase synchrony in the α band can serve as a predominant sign of AD in symptomatic patients [15, 16], while, at the pre-conversion stage, the results indicate evidence of changes in the pattern signs relative to the amplitude correlation in the β and β₁ bands among the MEG signals [40‐43]. Moreover, the discrimination of the HC cases from the MCI cases, instead, was favored by the presence of the FC indicators in the θ band [39, 44], which were not informative for the sMCI vs pMCI scenario in the pre-conversion phase (Table 3).

To further validate the platform, it would be important to test the proposed methods in a larger study. In the present work, to avoid overfitting, we extracted the deep data features from a pre-trained AlexNet architecture. Such features were automatically selected at the training level within rounds of LOPO cross-validation, thus allowing the training of simpler LDA or SVM classification modules based on a small set of features. In addition, our effort was devoted to identify the important MEG-based FC representations that inform classification (top–down approach) as the aggregation was performed using the best combinations of FC maps. The results obtained using knowledge-based computer vision techniques can be used as reference for deriving possible biomarkers for AD (down–up approach).

The results obtained in the present work compare favorably with the standard approach, in which the FC indicators are used as mono-dimensional training features, and with previous studies in the literature on the pre-conversion phase based on MEG data [17] or other imaging modalities [22, 23], posing the basis for further investigations on the proposed Deep-MEG architectures.

MEG provides the unique advantage of measuring the brain function with a remarkable combination of spatial and temporal resolutions. With this work, we have presented a novel system for decoding MEG recordings based on image-based representations of FC, deep CNN features, and ensemble classification architectures. The proposed methods for deriving and codifying the MEG-based FC measures allow the generation of pictures that represent, visually and numerically, the intricate communication patterns among spatially separated brain regions, which could be decoded by deep CNN features. The derivation of different cooperative architectures for integrating the spatiotemporal and multi-frequency information encoded in such images was the key to recognize the early alterations of the brain connectome relative to patients who undergo conversion to AD over a 3-year follow-up period.

In future, the analysis in resting state used in the present work may be extended with the analysis of other task-related activation patterns in order to optimize future applications of Deep-MEG architectures for predicting early signs of AD. Our findings may also have implications for the use of MEG-based FC as a biomarker in therapeutic trials. Finally, the proposed methods can be applied in other predictive scenarios to decode early signs of diverse neurodegenerative or neuropsychiatric diseases as well as to decode EEG signals.