Skip to main content
main-content

Tipp

Weitere Artikel dieser Ausgabe durch Wischen aufrufen

Erschienen in: Neural Processing Letters 3/2022

06.05.2022

Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example

verfasst von: Feng Duan, Yizhi Lv, Zhe Sun, Junhua Li

Erschienen in: Neural Processing Letters | Ausgabe 3/2022

Einloggen, um Zugang zu erhalten
share
TEILEN

Abstract

Neurophysiological signals are manifestations of the underlying brain activity, and they contain an abundance of neural information. The decoding and understanding of these signals is useful to develop robotic exoskeletons, benefitting device-aided motor rehabilitation. To this date, numerous efforts have been carried out to explore the relations between neurophysiological signals and locomotor capacity. Most of these studies focused on a single modality of neurophysiological signal and ignored its multiple modalities. In this study, the modalities from two kinds of biosensors were fused [electroencephalogram (EEG) and electromyogram (EMG)], and a novel deep learning model was proposed (multi-scale learning, MSL) to classify four walking patterns. The EEG and EMG data were collected during a walking experiment, where different walking conditions with and without exoskeleton-aided assistance were implemented (i.e. free-walking and exoskeleton-aided walking at zero, low, and high assistive forces). The performance achieved by the MSL model was compared to that of existing models, and the results show that multimodal MSL achieved the highest performance in terms of classification accuracy (89.33%). Moreover, the comparisons in our study show that an improved classification performance was obtained when a full 62-channel EEG setting was used compared to using a subset of 20 channels located on the sensorimotor region. This work contributes to the improvement of neurophysiological signal decoding and promotes the development of rehabilitation technologies as well as exoskeleton-aided applications.
Literatur
1.
Zurück zum Zitat Ren WJ, Han M (2019) Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine. Neural Process Lett 50(2):1281–1301 MathSciNetCrossRef Ren WJ, Han M (2019) Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine. Neural Process Lett 50(2):1281–1301 MathSciNetCrossRef
2.
Zurück zum Zitat Xi XG, Yang C, Shi JH, Luo ZZ, Zhao YB (2019) Surface electromyography-based daily activity recognition using wavelet coherence coefficient and support vector machine. Neural Process Lett 50(3):2265–2280 CrossRef Xi XG, Yang C, Shi JH, Luo ZZ, Zhao YB (2019) Surface electromyography-based daily activity recognition using wavelet coherence coefficient and support vector machine. Neural Process Lett 50(3):2265–2280 CrossRef
3.
Zurück zum Zitat Young AJ, Ferris DP (2017) State of the art and future directions for lower limb robotic exoskeletons. IEEE Trans Neural Syst Rehabil Eng 25(2):171–182 CrossRef Young AJ, Ferris DP (2017) State of the art and future directions for lower limb robotic exoskeletons. IEEE Trans Neural Syst Rehabil Eng 25(2):171–182 CrossRef
4.
Zurück zum Zitat Gordleeva SY, Lobov SA, Grigorev NA, Savosenkov AO, Shamshin MO, Lukoyanov MV, Khoruzhko MA, Kazantsev VB (2020) Real-time EEG-EMG human-machine interface-based control system for a lower-limb exoskeleton. IEEE Access 8:84070–84081 CrossRef Gordleeva SY, Lobov SA, Grigorev NA, Savosenkov AO, Shamshin MO, Lukoyanov MV, Khoruzhko MA, Kazantsev VB (2020) Real-time EEG-EMG human-machine interface-based control system for a lower-limb exoskeleton. IEEE Access 8:84070–84081 CrossRef
5.
Zurück zum Zitat Ma J, Zhang Y, Cichocki A, Matsuno F (2014) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62(3):876–889 CrossRef Ma J, Zhang Y, Cichocki A, Matsuno F (2014) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62(3):876–889 CrossRef
6.
Zurück zum Zitat Rashed-Al-Mahfuz M, Moni MA, Uddin S, Alyami SA, Summers MA, Eapen V (2021) A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (eeg) data. IEEE J Transl Eng Health Med 9:2000112–2000112 Rashed-Al-Mahfuz M, Moni MA, Uddin S, Alyami SA, Summers MA, Eapen V (2021) A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (eeg) data. IEEE J Transl Eng Health Med 9:2000112–2000112
7.
Zurück zum Zitat Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum Comput Interact 2013 Amiri S, Fazel-Rezai R, Asadpour V (2013) A review of hybrid brain-computer interface systems. Adv Hum Comput Interact 2013
8.
Zurück zum Zitat Giraudet L, Imbert J-P, Berenger M, Tremblay S, Causse M (2015) The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav Brain Res 294:246–253 CrossRef Giraudet L, Imbert J-P, Berenger M, Tremblay S, Causse M (2015) The neuroergonomic evaluation of human machine interface design in air traffic control using behavioral and EEG/ERP measures. Behav Brain Res 294:246–253 CrossRef
9.
Zurück zum Zitat da Silva-Sauer L, Valero-Aguayo L, de la Torre-Luque A, Ron-Angevin R, Varona-Moya S (2016) Concentration on performance with p300-based BCI systems: a matter of interface features. Appl Ergon 52:325–332 CrossRef da Silva-Sauer L, Valero-Aguayo L, de la Torre-Luque A, Ron-Angevin R, Varona-Moya S (2016) Concentration on performance with p300-based BCI systems: a matter of interface features. Appl Ergon 52:325–332 CrossRef
10.
Zurück zum Zitat Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. GigaScience 6(7):gix034 CrossRef Cho H, Ahn M, Ahn S, Kwon M, Jun SC (2017) EEG datasets for motor imagery brain-computer interface. GigaScience 6(7):gix034 CrossRef
11.
Zurück zum Zitat Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446 CrossRef Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446 CrossRef
12.
Zurück zum Zitat Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52 CrossRef Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52 CrossRef
13.
Zurück zum Zitat Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314 CrossRef Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314 CrossRef
14.
Zurück zum Zitat Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for EMG signal classification. J Expert Syst Appl 39(8):7420–7431 CrossRef Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for EMG signal classification. J Expert Syst Appl 39(8):7420–7431 CrossRef
15.
Zurück zum Zitat Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y (2013) EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst Appl 40(12):4832–4840 CrossRef Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y (2013) EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst Appl 40(12):4832–4840 CrossRef
16.
Zurück zum Zitat Wang H, Zhang L, Yao LX (2021) Application of genetic algorithm based support vector machine in selection of new EEG rhythms for drowsiness detection. Expert Syst Appl 171:7 Wang H, Zhang L, Yao LX (2021) Application of genetic algorithm based support vector machine in selection of new EEG rhythms for drowsiness detection. Expert Syst Appl 171:7
17.
Zurück zum Zitat Griffin D, Lim J (1984) Signal estimation from modified short-time. IEEE Trans Acoust Speech Signal Process 32(2):236–243 CrossRef Griffin D, Lim J (1984) Signal estimation from modified short-time. IEEE Trans Acoust Speech Signal Process 32(2):236–243 CrossRef
18.
Zurück zum Zitat Newson JJ, Thiagarajan TC (2019) EEG frequency bands in psychiatric disorders: a review of resting state studies. Front Hum Neurosci 12:24 CrossRef Newson JJ, Thiagarajan TC (2019) EEG frequency bands in psychiatric disorders: a review of resting state studies. Front Hum Neurosci 12:24 CrossRef
19.
Zurück zum Zitat Li J, Dimitrakopoulos GN, Thangavel P, Chen G, Sun Y, Guo Z, Yu H, Thakor N, Bezerianos A (2019) What are spectral and spatial distributions of EEG-EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946 CrossRef Li J, Dimitrakopoulos GN, Thangavel P, Chen G, Sun Y, Guo Z, Yu H, Thakor N, Bezerianos A (2019) What are spectral and spatial distributions of EEG-EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946 CrossRef
20.
Zurück zum Zitat Alharthi AS, Yunas SU, Ozanyan KB (2019) Deep learning for monitoring of human gait: a review. IEEE Sens J 19(21):9575–9591 CrossRef Alharthi AS, Yunas SU, Ozanyan KB (2019) Deep learning for monitoring of human gait: a review. IEEE Sens J 19(21):9575–9591 CrossRef
21.
Zurück zum Zitat Zhang J, Wang B, Zhang C, Xiao Y, Wang MY (2019) An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand. Front Neurorobot 13:7 CrossRef Zhang J, Wang B, Zhang C, Xiao Y, Wang MY (2019) An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand. Front Neurorobot 13:7 CrossRef
22.
Zurück zum Zitat Said AB, Mohamed A, Elfouly T, Harras K, Wang ZJ (2017) Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: 2017 IEEE wireless communications and networking conference (WCNC) Said AB, Mohamed A, Elfouly T, Harras K, Wang ZJ (2017) Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: 2017 IEEE wireless communications and networking conference (WCNC)
24.
Zurück zum Zitat Zhang P, Wang X, Zhang W, Chen J (2019) Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 27(1):31–42 CrossRef Zhang P, Wang X, Zhang W, Chen J (2019) Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 27(1):31–42 CrossRef
25.
Zurück zum Zitat Lamos M, Marecek R, Slavicek T, Mikl M, Rektor I, Jan J (2018) Spatial-temporal-spectral EEG patterns of BOLD functional network connectivity dynamics. J Neural Eng 15(3):036025 CrossRef Lamos M, Marecek R, Slavicek T, Mikl M, Rektor I, Jan J (2018) Spatial-temporal-spectral EEG patterns of BOLD functional network connectivity dynamics. J Neural Eng 15(3):036025 CrossRef
26.
Zurück zum Zitat Burges Christopher JC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167 CrossRef Burges Christopher JC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167 CrossRef
27.
28.
Zurück zum Zitat Thomas C, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef Thomas C, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef
29.
Zurück zum Zitat Wenyu L, Feng D, Sheng Shili XCG, Rensong L, Zhiwen Z, Xue J (2018) A human-vehicle collaborative simulated driving system based on hybrid brain-computer interfaces and computer vision. IEEE Trans Cognit Dev Syst 10(3):810–822 CrossRef Wenyu L, Feng D, Sheng Shili XCG, Rensong L, Zhiwen Z, Xue J (2018) A human-vehicle collaborative simulated driving system based on hybrid brain-computer interfaces and computer vision. IEEE Trans Cognit Dev Syst 10(3):810–822 CrossRef
30.
Zurück zum Zitat Hu HF, Liao ZK, Xiao X (2019) Action recognition using multiple pooling strategies of CNN features. Neural Process Lett 50(1):379–396 CrossRef Hu HF, Liao ZK, Xiao X (2019) Action recognition using multiple pooling strategies of CNN features. Neural Process Lett 50(1):379–396 CrossRef
31.
32.
Zurück zum Zitat Zhang R, Nie F, Li X, Wei X (2019) Feature selection with multi-view data: a survey. Inf Fusion 50:158–167 CrossRef Zhang R, Nie F, Li X, Wei X (2019) Feature selection with multi-view data: a survey. Inf Fusion 50:158–167 CrossRef
33.
Zurück zum Zitat Wang G, Han H, Shan S, Chen X (2020) Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) Wang G, Han H, Shan S, Chen X (2020) Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)
34.
Zurück zum Zitat Xie E, Zang Y, Shao S, Gang Yu, Yao C, Li G (2019) Scene text detection with supervised pyramid context network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9038–9045 Xie E, Zang Y, Shao S, Gang Yu, Yao C, Li G (2019) Scene text detection with supervised pyramid context network. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9038–9045
35.
Zurück zum Zitat Han ZY, Zhao J, Leung H, Ma A, Wang W (2021) A review of deep learning models for time series prediction. IEEE Sens J 21(6):7833–7848 CrossRef Han ZY, Zhao J, Leung H, Ma A, Wang W (2021) A review of deep learning models for time series prediction. IEEE Sens J 21(6):7833–7848 CrossRef
36.
Zurück zum Zitat Alexander C, Yongtian H, Contreras-Vidal Jose L (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001 CrossRef Alexander C, Yongtian H, Contreras-Vidal Jose L (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001 CrossRef
37.
Zurück zum Zitat Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2018) Deep learning in EMG-based gesture recognition. In: Proceedings of the 5th international conference on physiological computing systems, pp 107–114 Tsinganos P, Cornelis B, Cornelis J, Jansen B, Skodras A (2018) Deep learning in EMG-based gesture recognition. In: Proceedings of the 5th international conference on physiological computing systems, pp 107–114
38.
Zurück zum Zitat Kuan GS, Abbass Hussein A, Chen TK, Abdullah A-M, Nitish T, Anastasios B, Junhua L (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26(9):1858–1867 CrossRef Kuan GS, Abbass Hussein A, Chen TK, Abdullah A-M, Nitish T, Anastasios B, Junhua L (2018) Spatio-spectral representation learning for electroencephalographic gait-pattern classification. IEEE Trans Neural Syst Rehabil Eng 26(9):1858–1867 CrossRef
39.
Zurück zum Zitat Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3):551 CrossRef Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3):551 CrossRef
40.
Zurück zum Zitat Zhai X, Jelfs B, Chan RHM, Tin C (2017) Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network. Front Neurosci 11:379 CrossRef Zhai X, Jelfs B, Chan RHM, Tin C (2017) Self-recalibrating surface EMG pattern recognition for neuroprosthesis control based on convolutional neural network. Front Neurosci 11:379 CrossRef
41.
Zurück zum Zitat Neep H, Zhu CJ, Chung TA, Alex S (1997) Classification of EEG signals using the wavelet transform. Signal Process 59(1):61–72 CrossRef Neep H, Zhu CJ, Chung TA, Alex S (1997) Classification of EEG signals using the wavelet transform. Signal Process 59(1):61–72 CrossRef
42.
Zurück zum Zitat Li J, Thakor N, Bezerianos A (2018) Unilateral exoskeleton imposes significantly different hemispherical effect in parietooccipital region, but not in other regions. Sci Rep 8:126 CrossRef Li J, Thakor N, Bezerianos A (2018) Unilateral exoskeleton imposes significantly different hemispherical effect in parietooccipital region, but not in other regions. Sci Rep 8:126 CrossRef
43.
Zurück zum Zitat Li J, Chen G, Thangavel P, Yu H, Thakor N, Bezerianos A, Sun Y (2016) A robotic knee exoskeleton for walking assistance and connectivity topology exploration in EEG signal. In: 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob), proceedings of the IEEE RAS-EMBS international conference on biomedical robotics and biomechatronics, pp 1068–1073 Li J, Chen G, Thangavel P, Yu H, Thakor N, Bezerianos A, Sun Y (2016) A robotic knee exoskeleton for walking assistance and connectivity topology exploration in EEG signal. In: 6th IEEE international conference on biomedical robotics and biomechatronics (BioRob), proceedings of the IEEE RAS-EMBS international conference on biomedical robotics and biomechatronics, pp 1068–1073
44.
Zurück zum Zitat He P, Wilson G, Russell C (2004) Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 42(3):407–412 CrossRef He P, Wilson G, Russell C (2004) Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 42(3):407–412 CrossRef
45.
Zurück zum Zitat Li J, Chen Y, Taya F, Lim J, Wong K, Sun Y, Bezerianos A (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681 CrossRef Li J, Chen Y, Taya F, Lim J, Wong K, Sun Y, Bezerianos A (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681 CrossRef
46.
Zurück zum Zitat Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78 CrossRef Torrence C, Compo GP (1998) A practical guide to wavelet analysis. Bull Am Meteorol Soc 79(1):61–78 CrossRef
47.
Zurück zum Zitat Krizhevsky A, Sutskever I, Geoffrey HE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 875:84–90 CrossRef Krizhevsky A, Sutskever I, Geoffrey HE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 875:84–90 CrossRef
Metadaten
Titel
Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example
verfasst von
Feng Duan
Yizhi Lv
Zhe Sun
Junhua Li
Publikationsdatum
06.05.2022
Verlag
Springer US
Erschienen in
Neural Processing Letters / Ausgabe 3/2022
Print ISSN: 1370-4621
Elektronische ISSN: 1573-773X
DOI
https://doi.org/10.1007/s11063-021-10738-w

Weitere Artikel der Ausgabe 3/2022

Neural Processing Letters 3/2022 Zur Ausgabe