Skip to main content
Top
Published in: Medical & Biological Engineering & Computing 1/2024

20-09-2023 | Original Article

EEG-FMCNN: A fusion multi-branch 1D convolutional neural network for EEG-based motor imagery classification

Authors: Wenlong Wang, Baojiang Li, Haiyan Wang, Xichao Wang, Yuxin Qin, Xingbin Shi, Shuxin Liu

Published in: Medical & Biological Engineering & Computing | Issue 1/2024

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Motor imagery (MI) electroencephalogram (EEG) signal is recognized as a promising paradigm for brain-computer interface (BCI) systems and has been extensively employed in various BCI applications, including assisting disabled individuals, controlling devices and environments, and enhancing human capabilities. The high-performance decoding capability of MI-EEG signals is a key issue that impacts the development of the industry. However, decoding MI-EEG signals is challenging due to the low signal-to-noise ratio and inter-subject variability. In response to the aforementioned core problems, this paper proposes a novel end-to-end network, a fusion multi-branch 1D convolutional neural network (EEG-FMCNN), to decode MI-EEG signals without pre-processing. The utilization of multi-branch 1D convolution not only exhibits a certain level of noise tolerance but also addresses the issue of inter-subject variability to some extent. This is attributed to the ability of multi-branch architectures to capture information from different frequency bands, enabling the establishment of optimal convolutional scales and depths. Furthermore, we incorporate 1D squeeze-and-excitation (SE) blocks and shortcut connections at appropriate locations to further enhance the generalization and robustness of the network. In the BCI Competition IV-2a dataset, our proposed model has obtained good experimental results, achieving accuracies of 78.82% and 68.41% for subject-dependent and subject-independent modes, respectively. In addition, extensive ablative experiments and fine-tuning experiments were conducted, resulting in a notable 7% improvement in the average performance of the network, which holds significant implications for the generalization and application of the network.

Graphical Abstract

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Horki P et al (2011) Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med boil Eng Computing 49:567–577CrossRef Horki P et al (2011) Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med boil Eng Computing 49:567–577CrossRef
2.
go back to reference Altaheri H et al (2021) Deep learning techniques for classification of electroencephalogram (EEG) motor imagery (MI) signals: A review. Neural Computing App 35(20):14681–14722CrossRef Altaheri H et al (2021) Deep learning techniques for classification of electroencephalogram (EEG) motor imagery (MI) signals: A review. Neural Computing App 35(20):14681–14722CrossRef
3.
go back to reference Wang W, Li B, Wang H (2022) A novel end-to-end network based on a bidirectional GRU and a self-attention mechanism for denoising of electroencephalography signals. Neuroscience 505:10–20CrossRefPubMed Wang W, Li B, Wang H (2022) A novel end-to-end network based on a bidirectional GRU and a self-attention mechanism for denoising of electroencephalography signals. Neuroscience 505:10–20CrossRefPubMed
4.
go back to reference Nicolas-Alonso LF, Jaime G-G (2012) Brain computer interfaces, a review. Sensors 12.2:1211–1279CrossRef Nicolas-Alonso LF, Jaime G-G (2012) Brain computer interfaces, a review. Sensors 12.2:1211–1279CrossRef
5.
go back to reference Ferracuti F et al (2023) Augmenting robot intelligence via EEG signals to avoid trajectory planning mistakes of a smart wheelchair. J Ambient Intell Human Computing 14.1:223–235CrossRef Ferracuti F et al (2023) Augmenting robot intelligence via EEG signals to avoid trajectory planning mistakes of a smart wheelchair. J Ambient Intell Human Computing 14.1:223–235CrossRef
6.
go back to reference Riyad MMK, Abdellah A (2020) Incep-EEGNet: a convnet for motor imagery decoding. In Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, June 4–6, 2020, Proceedings 9 pp 103–111. Springer International Publishing Riyad MMK, Abdellah A (2020) Incep-EEGNet: a convnet for motor imagery decoding. In Image and Signal Processing: 9th International Conference, ICISP 2020, Marrakesh, Morocco, June 4–6, 2020, Proceedings 9 pp 103–111. Springer International Publishing
7.
go back to reference Vialatte F-B et al (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Progress Neurobiol 904:418–438CrossRef Vialatte F-B et al (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Progress Neurobiol 904:418–438CrossRef
8.
go back to reference Combaz A, Van Hulle M (2015) Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface. PLoS One 103:0121481 Combaz A, Van Hulle M (2015) Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface. PLoS One 103:0121481
9.
go back to reference Salelkar S, Ray S (2020) Interaction between steady-state visually evoked potentials at nearby flicker frequencies. Sci Rep 10(1):1–16CrossRef Salelkar S, Ray S (2020) Interaction between steady-state visually evoked potentials at nearby flicker frequencies. Sci Rep 10(1):1–16CrossRef
10.
go back to reference Mattioli F, Porcaro C, Baldassarre G (2022) A 1D CNN for high accuracy classification and transfer learning in motor imagery EEG-based brain-computer interface. J Neural Eng 18(6):066053CrossRef Mattioli F, Porcaro C, Baldassarre G (2022) A 1D CNN for high accuracy classification and transfer learning in motor imagery EEG-based brain-computer interface. J Neural Eng 18(6):066053CrossRef
12.
go back to reference Dos Santos EM, Rodrigo S-M, Francisco JF (2022) Comparison of subject-independent and subject-specific EEG-based BCI using LDA and SVM classifiers. Med Biol Eng Comput 61(3):835–845CrossRef Dos Santos EM, Rodrigo S-M, Francisco JF (2022) Comparison of subject-independent and subject-specific EEG-based BCI using LDA and SVM classifiers. Med Biol Eng Comput 61(3):835–845CrossRef
13.
go back to reference Liu X et al (2023) A compact multi-branch 1D convolutional neural network for EEG-based motor imagery classification. Biomed Signal Process Control 81:104456CrossRef Liu X et al (2023) A compact multi-branch 1D convolutional neural network for EEG-based motor imagery classification. Biomed Signal Process Control 81:104456CrossRef
14.
go back to reference M Hersche et al (2018) Fast and accurate multiclass inference for MI-BCIs using large multiscale temporal and spectral features. In 2018 26th European Signal Processing Conference (EUSIPCO) pp 1690–1694. IEEE M Hersche et al (2018) Fast and accurate multiclass inference for MI-BCIs using large multiscale temporal and spectral features. In 2018 26th European Signal Processing Conference (EUSIPCO) pp 1690–1694. IEEE
15.
go back to reference LeCun Y, Bengio Y, Hinton G (2015) Deep learning. nature 521(7553):436–444PubMed LeCun Y, Bengio Y, Hinton G (2015) Deep learning. nature 521(7553):436–444PubMed
16.
go back to reference Clevert DA, Unterthiner T, Hochreiter S (2015) Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 Clevert DA, Unterthiner T, Hochreiter S (2015) Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289
17.
go back to reference S Ioffe, S Christian (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning pp 448–456 S Ioffe, S Christian (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning pp 448–456
18.
go back to reference Alzahab NA et al (2021) Hybrid deep learning (hDL)-based brain-computer interface (BCI) systems: a systematic review. Brain Sci 111:75CrossRef Alzahab NA et al (2021) Hybrid deep learning (hDL)-based brain-computer interface (BCI) systems: a systematic review. Brain Sci 111:75CrossRef
19.
go back to reference S Vaid, S Preeti, K hamandeep (2015) EEG signal analysis for BCI interface: A review. In 2015 fifth international conference on advanced computing & communication technologies pp 143–147. IEEE S Vaid, S Preeti, K hamandeep (2015) EEG signal analysis for BCI interface: A review. In 2015 fifth international conference on advanced computing & communication technologies pp 143–147. IEEE
20.
go back to reference Dai G et al (2020) HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification. J Neural Eng 17.1:016025CrossRef Dai G et al (2020) HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification. J Neural Eng 17.1:016025CrossRef
21.
go back to reference Sun B et al (2021) EEG motor imagery classification with sparse spectrotemporal decomposition and deep learning. IEEE Transact Automat Sci Eng 18.2(2020):541–551CrossRef Sun B et al (2021) EEG motor imagery classification with sparse spectrotemporal decomposition and deep learning. IEEE Transact Automat Sci Eng 18.2(2020):541–551CrossRef
22.
go back to reference Schirrmeister RT et al (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping 38.11:5391–5420CrossRef Schirrmeister RT et al (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping 38.11:5391–5420CrossRef
23.
go back to reference Alzubaidi L et al (2021) Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. J Big Data 8:1–74CrossRef Alzubaidi L et al (2021) Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. J Big Data 8:1–74CrossRef
24.
go back to reference Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16.3:031001CrossRef Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16.3:031001CrossRef
25.
go back to reference J Hu, S Li, S Gang (2018) Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition pp 7132–7141 J Hu, S Li, S Gang (2018) Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition pp 7132–7141
26.
go back to reference S Woo et al (2018) Cbam: Convolutional block attention module. Proceedings of the European conference on computer vision (ECCV) pp 3–19 S Woo et al (2018) Cbam: Convolutional block attention module. Proceedings of the European conference on computer vision (ECCV) pp 3–19
27.
go back to reference Altuwaijri GAl et al (2022) A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for eeg-based motor imagery signals classification. Diagnostics 12.4:995CrossRef Altuwaijri GAl et al (2022) A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for eeg-based motor imagery signals classification. Diagnostics 12.4:995CrossRef
28.
go back to reference Altaheri H, Muhammad G, Alsulaiman M (2022) Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification. IEEE Trans Industr Inf 19(2):2249–2258CrossRef Altaheri H, Muhammad G, Alsulaiman M (2022) Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification. IEEE Trans Industr Inf 19(2):2249–2258CrossRef
29.
go back to reference Lawhern VJ et al (2018) EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng 15.5:056013CrossRef Lawhern VJ et al (2018) EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng 15.5:056013CrossRef
30.
go back to reference TM Ingolfsson et al (2020) EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) pp 2958–2965. IEEE TM Ingolfsson et al (2020) EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) pp 2958–2965. IEEE
31.
go back to reference Jia, Z et al (2021) MMCNN: A multi-branch multi-scale convolutional neural network for motor imagery classification. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part III pp 736–751. Springer International Publishing Jia, Z et al (2021) MMCNN: A multi-branch multi-scale convolutional neural network for motor imagery classification. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part III pp 736–751. Springer International Publishing
32.
go back to reference K He et al (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778 K He et al (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778
33.
go back to reference C Szegedy et al (2015) Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition pp 1–9 C Szegedy et al (2015) Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition pp 1–9
34.
go back to reference Zhang D et al (2020) Motor imagery classification via temporal attention cues of graph embedded EEG signals. IEEE J Biomed Health Inform 249:2570–2579CrossRef Zhang D et al (2020) Motor imagery classification via temporal attention cues of graph embedded EEG signals. IEEE J Biomed Health Inform 249:2570–2579CrossRef
35.
go back to reference Amin SU, Altaheri H, Muhammad G, Alsulaiman M, Abdul W (2021) Attention based Inception model for robust EEG motor imagery classification. In 2021 IEEE international instrumentation and measurement technology conference (I2MTC) pp 1–6 Amin SU, Altaheri H, Muhammad G, Alsulaiman M, Abdul W (2021) Attention based Inception model for robust EEG motor imagery classification. In 2021 IEEE international instrumentation and measurement technology conference (I2MTC) pp 1–6
37.
go back to reference Musallam YK et al (2021) Electroencephalography-based motor imagery classification using temporal convolutional network fusion. Biomed Signal Process Control 69:102826CrossRef Musallam YK et al (2021) Electroencephalography-based motor imagery classification using temporal convolutional network fusion. Biomed Signal Process Control 69:102826CrossRef
38.
go back to reference Brunner C, Leeb R, Müller-Putz G, Schlögl A, Pfurtscheller G (2008) BCI Competition 2008–Graz data set A. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces). Graz University of Technology 16:1–6 Brunner C, Leeb R, Müller-Putz G, Schlögl A, Pfurtscheller G (2008) BCI Competition 2008–Graz data set A. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces). Graz University of Technology 16:1–6
39.
go back to reference George O et al (2022) Data augmentation strategies for EEG-based motor imagery decoding. Heliyon 88:e10240CrossRef George O et al (2022) Data augmentation strategies for EEG-based motor imagery decoding. Heliyon 88:e10240CrossRef
40.
go back to reference Amin SU et al (2019) Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Gen Computer Systems 101:542–554CrossRef Amin SU et al (2019) Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Gen Computer Systems 101:542–554CrossRef
41.
go back to reference Li Chunguang et al (2020) Detecting self-paced walking intention based on fNIRS technology for the development of BCI. Med Biol Eng Computing 58:933–941CrossRef Li Chunguang et al (2020) Detecting self-paced walking intention based on fNIRS technology for the development of BCI. Med Biol Eng Computing 58:933–941CrossRef
42.
go back to reference Sindi Hatem et al (2021) A novel hybrid deep learning approach including combination of 1D power signals and 2D signal images for power quality disturbance classification. Expert Systems with App 174:114785CrossRef Sindi Hatem et al (2021) A novel hybrid deep learning approach including combination of 1D power signals and 2D signal images for power quality disturbance classification. Expert Systems with App 174:114785CrossRef
43.
go back to reference Nour M, Öztürk Ş, Polat K (2021) A novel classification framework using multiple bandwidth method with optimized CNN for brain–computer interfaces with EEG-fNIRS signals. Neural Comput Appl 33:15815–15829CrossRef Nour M, Öztürk Ş, Polat K (2021) A novel classification framework using multiple bandwidth method with optimized CNN for brain–computer interfaces with EEG-fNIRS signals. Neural Comput Appl 33:15815–15829CrossRef
44.
go back to reference Sindi Hatem et al (2021) An adaptive deep learning framework to classify unknown composite power quality event using known single power quality events. Expert Systems Appl 178:115023CrossRef Sindi Hatem et al (2021) An adaptive deep learning framework to classify unknown composite power quality event using known single power quality events. Expert Systems Appl 178:115023CrossRef
45.
go back to reference Salami A, Andreu-Perez J, Gillmeister H (2022) EEG-ITNet: An explainable inception temporal convolutional network for motor imagery classification. IEEE Access 10:36672–36685CrossRef Salami A, Andreu-Perez J, Gillmeister H (2022) EEG-ITNet: An explainable inception temporal convolutional network for motor imagery classification. IEEE Access 10:36672–36685CrossRef
Metadata
Title
EEG-FMCNN: A fusion multi-branch 1D convolutional neural network for EEG-based motor imagery classification
Authors
Wenlong Wang
Baojiang Li
Haiyan Wang
Xichao Wang
Yuxin Qin
Xingbin Shi
Shuxin Liu
Publication date
20-09-2023
Publisher
Springer Berlin Heidelberg
Published in
Medical & Biological Engineering & Computing / Issue 1/2024
Print ISSN: 0140-0118
Electronic ISSN: 1741-0444
DOI
https://doi.org/10.1007/s11517-023-02931-x

Other articles of this Issue 1/2024

Medical & Biological Engineering & Computing 1/2024 Go to the issue

Premium Partner