Skip to main content
main-content
Top

Hint

Swipe to navigate through the articles of this issue

Published in: Neural Processing Letters 3/2022

06-05-2022

Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example

Authors: Feng Duan, Yizhi Lv, Zhe Sun, Junhua Li

Published in: Neural Processing Letters | Issue 3/2022

Login to get access
share
SHARE

Abstract

Neurophysiological signals are manifestations of the underlying brain activity, and they contain an abundance of neural information. The decoding and understanding of these signals is useful to develop robotic exoskeletons, benefitting device-aided motor rehabilitation. To this date, numerous efforts have been carried out to explore the relations between neurophysiological signals and locomotor capacity. Most of these studies focused on a single modality of neurophysiological signal and ignored its multiple modalities. In this study, the modalities from two kinds of biosensors were fused [electroencephalogram (EEG) and electromyogram (EMG)], and a novel deep learning model was proposed (multi-scale learning, MSL) to classify four walking patterns. The EEG and EMG data were collected during a walking experiment, where different walking conditions with and without exoskeleton-aided assistance were implemented (i.e. free-walking and exoskeleton-aided walking at zero, low, and high assistive forces). The performance achieved by the MSL model was compared to that of existing models, and the results show that multimodal MSL achieved the highest performance in terms of classification accuracy (89.33%). Moreover, the comparisons in our study show that an improved classification performance was obtained when a full 62-channel EEG setting was used compared to using a subset of 20 channels located on the sensorimotor region. This work contributes to the improvement of neurophysiological signal decoding and promotes the development of rehabilitation technologies as well as exoskeleton-aided applications.
Literature
1.
go back to reference Ren WJ, Han M (2019) Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine. Neural Process Lett 50(2):1281–1301 MathSciNetCrossRef Ren WJ, Han M (2019) Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine. Neural Process Lett 50(2):1281–1301 MathSciNetCrossRef
2.
go back to reference Xi XG, Yang C, Shi JH, Luo ZZ, Zhao YB (2019) Surface electromyography-based daily activity recognition using wavelet coherence coefficient and support vector machine. Neural Process Lett 50(3):2265–2280 CrossRef Xi XG, Yang C, Shi JH, Luo ZZ, Zhao YB (2019) Surface electromyography-based daily activity recognition using wavelet coherence coefficient and support vector machine. Neural Process Lett 50(3):2265–2280 CrossRef
3.
go back to reference Young AJ, Ferris DP (2017) State of the art and future directions for lower limb robotic exoskeletons. IEEE Trans Neural Syst Rehabil Eng 25(2):171–182 CrossRef Young AJ, Ferris DP (2017) State of the art and future directions for lower limb robotic exoskeletons. IEEE Trans Neural Syst Rehabil Eng 25(2):171–182 CrossRef
4.
go back to reference Gordleeva SY, Lobov SA, Grigorev NA, Savosenkov AO, Shamshin MO, Lukoyanov MV, Khoruzhko MA, Kazantsev VB (2020) Real-time EEG-EMG human-machine interface-based control system for a lower-limb exoskeleton. IEEE Access 8:84070–84081 CrossRef Gordleeva SY, Lobov SA, Grigorev NA, Savosenkov AO, Shamshin MO, Lukoyanov MV, Khoruzhko MA, Kazantsev VB (2020) Real-time EEG-EMG human-machine interface-based control system for a lower-limb exoskeleton. IEEE Access 8:84070–84081 CrossRef
5.
go back to reference Ma J, Zhang Y, Cichocki A, Matsuno F (2014) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62(3):876–889 CrossRef Ma J, Zhang Y, Cichocki A, Matsuno F (2014) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62(3):876–889 CrossRef
6.
go back to reference Rashed-Al-Mahfuz M, Moni MA, Uddin S, Alyami SA, Summers MA, Eapen V (2021) A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (eeg) data. IEEE J Transl Eng Health Med 9:2000112–2000112 Rashed-Al-Mahfuz M, Moni MA, Uddin S, Alyami SA, Summers MA, Eapen V (2021) A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (eeg) data. IEEE J Transl Eng Health Med 9:2000112–2000112
7.
go back to reference Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum Comput Interact 2013 Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum Comput Interact 2013
8.
go back to reference Giraudet L, Imbert J-P, Berenger M, Tremblay S, Causse M (2015) The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav Brain Res 294:246–253 CrossRef Giraudet L, Imbert J-P, Berenger M, Tremblay S, Causse M (2015) The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav Brain Res 294:246–253 CrossRef
9.
go back to reference da Silva-Sauer L, Valero-Aguayo L, de la Torre-Luque A, Ron-Angevin R, Varona-Moya S (2016) Concentration on performance with p300-based BCI systems: a matter of interface features. Appl Ergon 52:325–332 CrossRef da Silva-Sauer L, Valero-Aguayo L, de la Torre-Luque A, Ron-Angevin R, Varona-Moya S (2016) Concentration on performance with p300-based BCI systems: a matter of interface features. Appl Ergon 52:325–332 CrossRef
10.
go back to reference Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. GigaScience 6(7):gix034 CrossRef Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. GigaScience 6(7):gix034 CrossRef
11.
go back to reference Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446 CrossRef Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446 CrossRef
12.
go back to reference Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52 CrossRef Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52 CrossRef
13.
go back to reference Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314 CrossRef Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314 CrossRef
14.
go back to reference Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for EMG signal classification. J Expert Syst Appl 39(8):7420–7431 CrossRef Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for EMG signal classification. J Expert Syst Appl 39(8):7420–7431 CrossRef
15.
go back to reference Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y (2013) EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst Appl 40(12):4832–4840 CrossRef Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y (2013) EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst Appl 40(12):4832–4840 CrossRef
16.
go back to reference Wang H, Zhang L, Yao LX (2021) Application of genetic algorithm based support vector machine in selection of new EEG rhythms for drowsiness detection. Expert Syst Appl 171:7 Wang H, Zhang L, Yao LX (2021) Application of genetic algorithm based support vector machine in selection of new EEG rhythms for drowsiness detection. Expert Syst Appl 171:7
17.
go back to reference Griffin D, Lim J (1984) Signal estimation from modified short-time. IEEE Trans Acoust Speech Signal Process 32(2):236–243 CrossRef Griffin D, Lim J (1984) Signal estimation from modified short-time. IEEE Trans Acoust Speech Signal Process 32(2):236–243 CrossRef
18.
go back to reference Newson JJ, Thiagarajan TC (2019) EEG frequency bands in psychiatric disorders: a review of resting state studies. Front Hum Neurosci 12:24 CrossRef Newson JJ, Thiagarajan TC (2019) EEG frequency bands in psychiatric disorders: a review of resting state studies. Front Hum Neurosci 12:24 CrossRef
19.
go back to reference Li J, Dimitrakopoulos GN, Thangavel P, Chen G, Sun Y, Guo Z, Yu H, Thakor N, Bezerianos A (2019) What are spectral and spatial distributions of EEG-EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946 CrossRef Li J, Dimitrakopoulos GN, Thangavel P, Chen G, Sun Y, Guo Z, Yu H, Thakor N, Bezerianos A (2019) What are spectral and spatial distributions of EEG-EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946 CrossRef
20.
go back to reference Alharthi AS, Yunas SU, Ozanyan KB (2019) Deep learning for monitoring of human gait: a review. IEEE Sens J 19(21):9575–9591 CrossRef Alharthi AS, Yunas SU, Ozanyan KB (2019) Deep learning for monitoring of human gait: a review. IEEE Sens J 19(21):9575–9591 CrossRef
21.
go back to reference Zhang J, Wang B, Zhang C, Xiao Y, Wang MY (2019) An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand. Front Neurorobot 13:7 CrossRef Zhang J, Wang B, Zhang C, Xiao Y, Wang MY (2019) An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand. Front Neurorobot 13:7 CrossRef
22.
go back to reference Said AB, Mohamed A, Elfouly T, Harras K, Wang ZJ (2017) Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: 2017 IEEE wireless communications and networking conference (WCNC) Said AB, Mohamed A, Elfouly T, Harras K, Wang ZJ (2017) Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: 2017 IEEE wireless communications and networking conference (WCNC)
24.
go back to reference Zhang P, Wang X, Zhang W, Chen J (2019) Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 27(1):31–42 CrossRef Zhang P, Wang X, Zhang W, Chen J (2019) Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 27(1):31–42 CrossRef
25.
go back to reference Lamos M, Marecek R, Slavicek T, Mikl M, Rektor I, Jan J (2018) Spatial-temporal-spectral EEG patterns of BOLD functional network connectivity dynamics. J Neural Eng 15(3):036025 CrossRef Lamos M, Marecek R, Slavicek T, Mikl M, Rektor I, Jan J (2018) Spatial-temporal-spectral EEG patterns of BOLD functional network connectivity dynamics. J Neural Eng 15(3):036025 CrossRef
26.
go back to reference Burges Christopher JC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167 CrossRef Burges Christopher JC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167 CrossRef
28.
go back to reference Thomas C, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef Thomas C, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef
29.
go back to reference Wenyu L, Feng D, Sheng Shili XCG, Rensong L, Zhiwen Z, Xue J (2018) A human-vehicle collaborative simulated driving system based on hybrid brain-computer interfaces and computer vision. IEEE Trans Cognit Dev Syst 10(3):810–822 CrossRef Wenyu L, Feng D, Sheng Shili XCG, Rensong L, Zhiwen Z, Xue J (2018) A human-vehicle collaborative simulated driving system based on hybrid brain-computer interfaces and computer vision. IEEE Trans Cognit Dev Syst 10(3):810–822 CrossRef
30.
go back to reference Hu HF, Liao ZK, Xiao X (2019) Action recognition using multiple pooling strategies of CNN features. Neural Process Lett 50(1):379–396 CrossRef Hu HF, Liao ZK, Xiao X (2019) Action recognition using multiple pooling strategies of CNN features. Neural Process Lett 50(1):379–396 CrossRef
31.
32.
go back to reference Zhang R, Nie F, Li X, Wei X (2019) Feature selection with multi-view data: a survey. Inf Fusion 50:158–167 CrossRef Zhang R, Nie F, Li X, Wei X (2019) Feature selection with multi-view data: a survey. Inf Fusion 50:158–167 CrossRef
33.
go back to reference Wang G, Han H, Shan S, Chen X (2020) Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) Wang G, Han H, Shan S, Chen X (2020) Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)
34.
go back to reference Xie E, Zang Y, Shao S, Gang Yu, Yao C, Li G (2019) Scene text detection with supervised pyramid context network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9038–9045 Xie E, Zang Y, Shao S, Gang Yu, Yao C, Li G (2019) Scene text detection with supervised pyramid context network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9038–9045
35.
go back to reference Han ZY, Zhao J, Leung H, Ma A, Wang W (2021) A review of deep learning models for time series prediction. IEEE Sens J 21(6):7833–7848 CrossRef Han ZY, Zhao J, Leung H, Ma A, Wang W (2021) A review of deep learning models for time series prediction. IEEE Sens J 21(6):7833–7848 CrossRef
36.
go back to reference Alexander C, Yongtian H, Contreras-Vidal Jose L (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001 CrossRef Alexander C, Yongtian H, Contreras-Vidal Jose L (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001 CrossRef
37.
go back to reference Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2018) Deep learning in EMG-based gesture recognition. In: Proceedings of the 5th international conference on physiological computing systems, pp 107–114 Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2018) Deep learning in EMG-based gesture recognition. In: Proceedings of the 5th international conference on physiological computing systems, pp 107–114
38.
go back to reference Kuan GS, Abbass Hussein A, Chen TK, Abdullah A-M, Nitish T, Anastasios B, Junhua L (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26(9):1858–1867 CrossRef Kuan GS, Abbass Hussein A, Chen TK, Abdullah A-M, Nitish T, Anastasios B, Junhua L (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26(9):1858–1867 CrossRef
39.
go back to reference Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3):551 CrossRef Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3):551 CrossRef
40.
go back to reference Zhai X, Jelfs B, Chan RHM, Tin C (2017) Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network. Front Neurosci 11:379 CrossRef Zhai X, Jelfs B, Chan RHM, Tin C (2017) Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network. Front Neurosci 11:379 CrossRef
41.
go back to reference Neep H, Zhu CJ, Chung TA, Alex S (1997) Classification of EEG signals using the wavelet transform. Signal Process 59(1):61–72 CrossRef Neep H, Zhu CJ, Chung TA, Alex S (1997) Classification of EEG signals using the wavelet transform. Signal Process 59(1):61–72 CrossRef
42.
go back to reference Li J, Thakor N, Bezerianos A (2018) Unilateral exoskeleton imposes significantly different hemispherical effect in parietooccipital region, but not in other regions. Sci Rep 8:126 CrossRef Li J, Thakor N, Bezerianos A (2018) Unilateral exoskeleton imposes significantly different hemispherical effect in parietooccipital region, but not in other regions. Sci Rep 8:126 CrossRef
43.
go back to reference Li J, Chen G, Thangavel P, Yu H, Thakor N, Bezerianos A, Sun Y (2016) A robotic knee exoskeleton for walking assistance and connectivity topology exploration in EEG signal. In: 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob), proceedings of the IEEE RAS-EMBS international conference on biomedical robotics and biomechatronics, pp 1068–1073 Li J, Chen G, Thangavel P, Yu H, Thakor N, Bezerianos A, Sun Y (2016) A robotic knee exoskeleton for walking assistance and connectivity topology exploration in EEG signal. In: 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob), proceedings of the IEEE RAS-EMBS international conference on biomedical robotics and biomechatronics, pp 1068–1073
44.
go back to reference He P, Wilson G, Russell C (2004) Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 42(3):407–412 CrossRef He P, Wilson G, Russell C (2004) Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 42(3):407–412 CrossRef
45.
go back to reference Li J, Chen Y, Taya F, Lim J, Wong K, Sun Y, Bezerianos A (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681 CrossRef Li J, Chen Y, Taya F, Lim J, Wong K, Sun Y, Bezerianos A (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681 CrossRef
46.
go back to reference Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78 CrossRef Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78 CrossRef
47.
go back to reference Krizhevsky A, Sutskever I, Geoffrey HE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 875:84–90 CrossRef Krizhevsky A, Sutskever I, Geoffrey HE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 875:84–90 CrossRef
Metadata
Title
Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example
Authors
Feng Duan
Yizhi Lv
Zhe Sun
Junhua Li
Publication date
06-05-2022
Publisher
Springer US
Published in
Neural Processing Letters / Issue 3/2022
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI
https://doi.org/10.1007/s11063-021-10738-w

Other articles of this Issue 3/2022

Neural Processing Letters 3/2022 Go to the issue