Skip to main content
Top
Published in:

24-01-2022 | Research Article

A multi-modal brain–computer interface based on threshold discrimination and its application in wheelchair control

Authors: Enzeng Dong, Haoran Zhang, Lin Zhu, Shengzhi Du, Jigang Tong

Published in: Cognitive Neurodynamics | Issue 5/2022

Login to get access

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this study, we propose a novel multi-modal brain–computer interface (BCI) system based on the threshold discrimination, which is proposed for the first time to distinguish between SSVEP and MI potentials. The system combines these two heterogeneous signals to increase the number of control commands and improve the performance of asynchronous control of external devices. In this research, an electric wheelchair is controlled as an example. The user can continuously control the wheelchair to turn left/right through motion imagination (MI) by imagining left/right-hand movement and generate another 6 commands for the wheelchair control by focusing on the SSVEP stimulation panel. Ten subjects participated in a MI training session and eight of them completed a mobile obstacle-avoidance experiment in a complex environment requesting high control accuracy for successful manipulation. Comparing with the single-modal BCI-controlled wheelchair system, the results demonstrate that the proposed multi-modal method is effective by providing more satisfactory control accuracy, and show the potential of BCI-controlled systems to be applied in complex daily tasks.

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
go back to reference Cao L, Li J, Ji H, Jiang C (2014) A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control. J Neurosci Methods 229:33–43PubMedCrossRef Cao L, Li J, Ji H, Jiang C (2014) A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control. J Neurosci Methods 229:33–43PubMedCrossRef
go back to reference Chen X, Wang Y, Gao S, Jung TP, Gao X (2015) Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J Neural Eng 12:046008PubMedCrossRef Chen X, Wang Y, Gao S, Jung TP, Gao X (2015) Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J Neural Eng 12:046008PubMedCrossRef
go back to reference Cheng M, Gao X, Gao S, Member S, Xu D (2002) Design and implementation of a brain–computer interface with high transfer rates. IEEE Trans Biomed Eng 49:1181–1186PubMedCrossRef Cheng M, Gao X, Gao S, Member S, Xu D (2002) Design and implementation of a brain–computer interface with high transfer rates. IEEE Trans Biomed Eng 49:1181–1186PubMedCrossRef
go back to reference Dong E, Li C, Li L, Du S, Belkacem AN, Chen C (2017) Classification of multi-class motor imagery with a novel hierarchical SVM algorithm for brain–computer interfaces. Med Biol Eng Compu 55:1809–1818CrossRef Dong E, Li C, Li L, Du S, Belkacem AN, Chen C (2017) Classification of multi-class motor imagery with a novel hierarchical SVM algorithm for brain–computer interfaces. Med Biol Eng Compu 55:1809–1818CrossRef
go back to reference Dong E, Zhu G, Chen C, Tong J, Jiao Y, Du S (2018) Introducing chaos behavior to kernel relevance vector machine (RVM) for four-class EEG classification. PLoS ONE 13:e0198786PubMedPubMedCentralCrossRef Dong E, Zhu G, Chen C, Tong J, Jiao Y, Du S (2018) Introducing chaos behavior to kernel relevance vector machine (RVM) for four-class EEG classification. PLoS ONE 13:e0198786PubMedPubMedCentralCrossRef
go back to reference Dong E, Zhou K, Tong J, Du S (2020) A novel hybrid kernel function relevance vector machine for multi-task motor imagery EEG classification. Biomed Signal Process Control 60:1746–8094CrossRef Dong E, Zhou K, Tong J, Du S (2020) A novel hybrid kernel function relevance vector machine for multi-task motor imagery EEG classification. Biomed Signal Process Control 60:1746–8094CrossRef
go back to reference Duan X, Xie S, Xie X, Meng Y, Xu Z (2019) Quadcopter flight control using a non-invasive multi-modal brain computer interface. Front Neurorobot 13:1662–5218CrossRef Duan X, Xie S, Xie X, Meng Y, Xu Z (2019) Quadcopter flight control using a non-invasive multi-modal brain computer interface. Front Neurorobot 13:1662–5218CrossRef
go back to reference Fazel-Reza R, Allison BZ, Guger C, Sellers EW, Kübler A (2012) P300 brain computing interface: current challenges and emerging trends. Front Neuroeng 17:5–14 Fazel-Reza R, Allison BZ, Guger C, Sellers EW, Kübler A (2012) P300 brain computing interface: current challenges and emerging trends. Front Neuroeng 17:5–14
go back to reference Fouad MM, Amin KM, El-Bendary N, Hassanien AE (2015) Brain computer interface: a review. Springer, Berlin, p 74 Fouad MM, Amin KM, El-Bendary N, Hassanien AE (2015) Brain computer interface: a review. Springer, Berlin, p 74
go back to reference Franois-Benot V, Monique M (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 90:418–438CrossRef Franois-Benot V, Monique M (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 90:418–438CrossRef
go back to reference Gao Q, Zhang Y, Wang Z, Dong E (2019) Channel projection-based CCA target identification method for an SSVEP-based BCI system of quadrotor helicopter control. Comput Intell Neurosci 16:2361282 Gao Q, Zhang Y, Wang Z, Dong E (2019) Channel projection-based CCA target identification method for an SSVEP-based BCI system of quadrotor helicopter control. Comput Intell Neurosci 16:2361282
go back to reference Han CH, Muller KR, Hwang HJ (2020) Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynchronous brain–computer interface. IEEE Trans Neural Syst Rehabil Eng 28:2102–2112PubMedCrossRef Han CH, Muller KR, Hwang HJ (2020) Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynchronous brain–computer interface. IEEE Trans Neural Syst Rehabil Eng 28:2102–2112PubMedCrossRef
go back to reference Horki P, Solis Escalante T, Neuper C, Müller Putz G (2011) Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med Biol Eng Compu 49:567–577CrossRef Horki P, Solis Escalante T, Neuper C, Müller Putz G (2011) Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med Biol Eng Compu 49:567–577CrossRef
go back to reference Huang Q, Zhang Z, Yu T, He S, Li Y (2019) An EEG-/EOG-based hybrid brain–computer interface: application on controlling an integrated wheelchair robotic arm system. Front Neurosci 22:1243CrossRef Huang Q, Zhang Z, Yu T, He S, Li Y (2019) An EEG-/EOG-based hybrid brain–computer interface: application on controlling an integrated wheelchair robotic arm system. Front Neurosci 22:1243CrossRef
go back to reference Jin J, Zhang H, Daly I, Wang X, Cichocki A (2017) An improved P300 pattern in BCI to catch user’s attention. J Neural Eng 14:036001PubMedCrossRef Jin J, Zhang H, Daly I, Wang X, Cichocki A (2017) An improved P300 pattern in BCI to catch user’s attention. J Neural Eng 14:036001PubMedCrossRef
go back to reference Kim KT, Suk HI, Lee SW (2018) Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials. IEEE Trans Neural Syst Rehabil Eng 26:654–665PubMedCrossRef Kim KT, Suk HI, Lee SW (2018) Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials. IEEE Trans Neural Syst Rehabil Eng 26:654–665PubMedCrossRef
go back to reference Ko LW, Komarov O, Lin SC (2019) Enhancing the hybrid BCI performance with the common frequency pattern in dual-channel EEG. IEEE Trans Neural Syst Rehabil Eng 27:1360–1369PubMedCrossRef Ko LW, Komarov O, Lin SC (2019) Enhancing the hybrid BCI performance with the common frequency pattern in dual-channel EEG. IEEE Trans Neural Syst Rehabil Eng 27:1360–1369PubMedCrossRef
go back to reference Kuhlman WN (1978) EEG feedback training: enhancement of somatosensory cortical activity. Electroencephalogr Clin Neurophysiol 45:290–294PubMedCrossRef Kuhlman WN (1978) EEG feedback training: enhancement of somatosensory cortical activity. Electroencephalogr Clin Neurophysiol 45:290–294PubMedCrossRef
go back to reference Lafleur K, Cassady K, Dou DA, Shades K, Rogin E, He B (2013) Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface. J Neural Eng 10:046003PubMedCrossRef Lafleur K, Cassady K, Dou DA, Shades K, Rogin E, He B (2013) Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface. J Neural Eng 10:046003PubMedCrossRef
go back to reference Lamti HA, Khelifa B, Hugel V (2019) Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment. Cogn Neurodyn 13:271–285PubMedPubMedCentralCrossRef Lamti HA, Khelifa B, Hugel V (2019) Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment. Cogn Neurodyn 13:271–285PubMedPubMedCentralCrossRef
go back to reference Lebedev MA, Nicolelis M (2006) Brain-machine interfaces: past, present and future. Trends Neurosci 29:536–546PubMedCrossRef Lebedev MA, Nicolelis M (2006) Brain-machine interfaces: past, present and future. Trends Neurosci 29:536–546PubMedCrossRef
go back to reference Lee MH, Williamson J, Won DO, Fazli S, Lee SW (2018) A high performance spelling system based on EEG-EOG signals with visual feedback. IEEE Trans Neural Syst Rehabil Eng 26:1443–1459PubMedCrossRef Lee MH, Williamson J, Won DO, Fazli S, Lee SW (2018) A high performance spelling system based on EEG-EOG signals with visual feedback. IEEE Trans Neural Syst Rehabil Eng 26:1443–1459PubMedCrossRef
go back to reference Li Y, Pan J, Wang F, Yu Z (2013) A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Trans Biomed Eng 60:3156–3166PubMedCrossRef Li Y, Pan J, Wang F, Yu Z (2013) A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Trans Biomed Eng 60:3156–3166PubMedCrossRef
go back to reference Lin Z, Zhang C, Wu W, Gao X (2006) Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng 53:2610–2614PubMedCrossRef Lin Z, Zhang C, Wu W, Gao X (2006) Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng 53:2610–2614PubMedCrossRef
go back to reference Ma T, Li H (2017) The hybrid BCI system for movement control by combining motor imagery and moving onset visual evoked potential. J Neural Eng 14:026015PubMedCrossRef Ma T, Li H (2017) The hybrid BCI system for movement control by combining motor imagery and moving onset visual evoked potential. J Neural Eng 14:026015PubMedCrossRef
go back to reference Middendorf M, McMillan G, Calhoun G, Jones KS (2000) Brain–computer interfaces based on the steady-state visual-evoked response. IEEE Trans Rehabil Eng 8:211–214PubMedCrossRef Middendorf M, McMillan G, Calhoun G, Jones KS (2000) Brain–computer interfaces based on the steady-state visual-evoked response. IEEE Trans Rehabil Eng 8:211–214PubMedCrossRef
go back to reference Pfurtscheller G, Neuper C (2001) Motor imagery and direct brain–computer communication. Proc IEEE 89:1123–1134CrossRef Pfurtscheller G, Neuper C (2001) Motor imagery and direct brain–computer communication. Proc IEEE 89:1123–1134CrossRef
go back to reference Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz GR (2010) Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng 18:409–414PubMedCrossRef Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz GR (2010) Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng 18:409–414PubMedCrossRef
go back to reference Rebsamen B, Guan C, Zhang H, Wang C, Burdet E (2010) A brain controlled wheelchair to navigate in familiar environments. IEEE Trans Neural Syst Rehabil Eng 18:590–598PubMedCrossRef Rebsamen B, Guan C, Zhang H, Wang C, Burdet E (2010) A brain controlled wheelchair to navigate in familiar environments. IEEE Trans Neural Syst Rehabil Eng 18:590–598PubMedCrossRef
go back to reference Thomas KP, Guan C, Lau CT, Vinod AP, Kai KA (2009) A new discriminative common spatial pattern method for motor imagery brain–computer interfaces. IEEE Trans Biomed Eng 56:2730–2733PubMedCrossRef Thomas KP, Guan C, Lau CT, Vinod AP, Kai KA (2009) A new discriminative common spatial pattern method for motor imagery brain–computer interfaces. IEEE Trans Biomed Eng 56:2730–2733PubMedCrossRef
go back to reference Xu M, Han J, Wang Y, Jung TP, Ming D (2020) Implementing over 100 command codes for a high-speed hybrid brain–computer interface using concurrent P300 and SSVEP features. IEEE Trans Biomed Eng 67:3073–3082PubMedCrossRef Xu M, Han J, Wang Y, Jung TP, Ming D (2020) Implementing over 100 command codes for a high-speed hybrid brain–computer interface using concurrent P300 and SSVEP features. IEEE Trans Biomed Eng 67:3073–3082PubMedCrossRef
go back to reference Xu L, Xu M, Jung TP, Ming D (2021) Review of brain encoding and decoding mechanisms for EEG-based brain–computer interface. Cogn Neurodyn 4:1–16 Xu L, Xu M, Jung TP, Ming D (2021) Review of brain encoding and decoding mechanisms for EEG-based brain–computer interface. Cogn Neurodyn 4:1–16
go back to reference Zhang H, Dong E, Zhu L (2020) Brain-controlled wheelchair system based on SSVEP. In: 2020 Chinese automation congress (CAC) Zhang H, Dong E, Zhu L (2020) Brain-controlled wheelchair system based on SSVEP. In: 2020 Chinese automation congress (CAC)
go back to reference Zuo C, Jin J, Yin E (2020) Novel hybrid brain–computer interface system based on motor imagery and P300. Cogn Neurodyn 14:253–265PubMedCrossRef Zuo C, Jin J, Yin E (2020) Novel hybrid brain–computer interface system based on motor imagery and P300. Cogn Neurodyn 14:253–265PubMedCrossRef
Metadata
Title
A multi-modal brain–computer interface based on threshold discrimination and its application in wheelchair control
Authors
Enzeng Dong
Haoran Zhang
Lin Zhu
Shengzhi Du
Jigang Tong
Publication date
24-01-2022
Publisher
Springer Netherlands
Published in
Cognitive Neurodynamics / Issue 5/2022
Print ISSN: 1871-4080
Electronic ISSN: 1871-4099
DOI
https://doi.org/10.1007/s11571-021-09779-7

Other articles of this Issue 5/2022

Cognitive Neurodynamics 5/2022 Go to the issue