Skip to main content
Top

Hint

Swipe to navigate through the articles of this issue

Published in: Neural Processing Letters 5/2021

31-03-2021

Opportunistic Activity Recognition in IoT Sensor Ecosystems via Multimodal Transfer Learning

Authors: Oresti Banos, Alberto Calatroni, Miguel Damas, Hector Pomares, Daniel Roggen, Ignacio Rojas, Claudia Villalonga

Published in: Neural Processing Letters | Issue 5/2021

Login to get access
share
SHARE

Abstract

Recognizing human activities seamlessly and ubiquitously is now closer than ever given the myriad of sensors readily deployed on and around users. However, the training of recognition systems continues to be both time and resource-consuming, as datasets must be collected ad-hoc for each specific sensor setup a person may encounter in their daily life. This work presents an alternate approach based on transfer learning to opportunistically train new unseen or target sensor systems from existing or source sensor systems. The approach uses system identification techniques to learn a mapping function that automatically translates the signals from the source sensor domain to the target sensor domain, and vice versa. This can be done for sensor signals of the same or cross modality. Two transfer models are proposed to translate recognition systems based on either activity templates or activity models, depending on the characteristics of both source and target sensor systems. The proposed transfer methods are evaluated in a human–computer interaction scenario, where the transfer is performed in between wearable sensors placed at different body locations, and in between wearable sensors and an ambient depth camera sensor. Results show that a good transfer is possible with just a few seconds of data, irrespective of the direction of the transfer and for similar and cross sensor modalities.
Footnotes
Literature
1.
go back to reference Lin J, Wei Yu, Zhang N, Yang X, Zhang H, Zhao W (2017) A survey on internet of things: architecture, enabling technologies, security and privacy, and applications. IEEE Internet Things J 4(5):1125–1142 CrossRef Lin J, Wei Yu, Zhang N, Yang X, Zhang H, Zhao W (2017) A survey on internet of things: architecture, enabling technologies, security and privacy, and applications. IEEE Internet Things J 4(5):1125–1142 CrossRef
2.
go back to reference Chen L, Nugent CD (2019) Human activity recognition and behaviour analysis. Springer, Berlin CrossRef Chen L, Nugent CD (2019) Human activity recognition and behaviour analysis. Springer, Berlin CrossRef
3.
go back to reference Lukowicz P, Hanser F, Szubski C, Schobersberger W (2006) Detecting and interpreting muscle activity with wearable force sensors. In: Pervasive computing, pp 101–116 Lukowicz P, Hanser F, Szubski C, Schobersberger W (2006) Detecting and interpreting muscle activity with wearable force sensors. In: Pervasive computing, pp 101–116
4.
go back to reference Amft O (2010) A wearable earpad sensor for chewing monitoring. In: IEEE sensors conference, pp 222–227 Amft O (2010) A wearable earpad sensor for chewing monitoring. In: IEEE sensors conference, pp 222–227
5.
go back to reference Banos O, Damas M, Pomares H, Prieto A, Rojas I (2012) Daily living activity recognition based on statistical feature quality group selection. Expert Syst Appl 39(9):8013–8021 CrossRef Banos O, Damas M, Pomares H, Prieto A, Rojas I (2012) Daily living activity recognition based on statistical feature quality group selection. Expert Syst Appl 39(9):8013–8021 CrossRef
6.
go back to reference Wang F, Liu J (2010) Networked wireless sensor data collection: issues, challenges, and approaches. IEEE Commun Surv Tutor 13(4):673–687 CrossRef Wang F, Liu J (2010) Networked wireless sensor data collection: issues, challenges, and approaches. IEEE Commun Surv Tutor 13(4):673–687 CrossRef
7.
go back to reference Roggen D, Troester G, Lukowicz P, Ferscha L, Millan JR, Chavarriaga R (2013) Opportunistic human activity and context recognition. Computer 46(2):36–45 CrossRef Roggen D, Troester G, Lukowicz P, Ferscha L, Millan JR, Chavarriaga R (2013) Opportunistic human activity and context recognition. Computer 46(2):36–45 CrossRef
8.
go back to reference Guo B, Zhang D, Wang Z, Zhiwen Yu, Zhou X (2013) Opportunistic IoT: exploring the harmonious interaction between human and the internet of things. J Netw Comput Appl 36(6):1531–1539 CrossRef Guo B, Zhang D, Wang Z, Zhiwen Yu, Zhou X (2013) Opportunistic IoT: exploring the harmonious interaction between human and the internet of things. J Netw Comput Appl 36(6):1531–1539 CrossRef
9.
go back to reference Villalonga C, Pomares H, Rojas I, Banos O (2017) MIMU-wear: ontology-based sensor selection for real-world wearable activity recognition. Neurocomputing 250:76–100 CrossRef Villalonga C, Pomares H, Rojas I, Banos O (2017) MIMU-wear: ontology-based sensor selection for real-world wearable activity recognition. Neurocomputing 250:76–100 CrossRef
10.
go back to reference Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359 CrossRef Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359 CrossRef
11.
go back to reference Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: Proceedings of pervasive computing, pp 1–16 Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: Proceedings of pervasive computing, pp 1–16
12.
go back to reference Chavarriaga R, Bayati H, Del Millán J (2013) Unsupervised adaptation for acceleration-based activity recognition: robustness to sensor displacement and rotation. Pers Ubiquit Comput 17(3):479–490 CrossRef Chavarriaga R, Bayati H, Del Millán J (2013) Unsupervised adaptation for acceleration-based activity recognition: robustness to sensor displacement and rotation. Pers Ubiquit Comput 17(3):479–490 CrossRef
13.
go back to reference Foerster K, Roggen D, Troester G (2009) Unsupervised classifier self-calibration through repeated context occurences: is there robustness against sensor displacement to gain? In: International symposium on wearable computers, pp 77–84, Linz, Austria Foerster K, Roggen D, Troester G (2009) Unsupervised classifier self-calibration through repeated context occurences: is there robustness against sensor displacement to gain? In: International symposium on wearable computers, pp 77–84, Linz, Austria
14.
go back to reference Kunze K, Lukowicz P (2008) Dealing with sensor displacement in motion-based onbody activity recognition systems. In: International conference on ubiquitous computing, pp 20–29 Kunze K, Lukowicz P (2008) Dealing with sensor displacement in motion-based onbody activity recognition systems. In: International conference on ubiquitous computing, pp 20–29
15.
go back to reference Kunze K, Bahle G, Lukowicz P, Partridge K (2010) Can magnetic field sensors replace gyroscopes in wearable sensing applications? In: International symposium on wearable computers Kunze K, Bahle G, Lukowicz P, Partridge K (2010) Can magnetic field sensors replace gyroscopes in wearable sensing applications? In: International symposium on wearable computers
16.
go back to reference Kunze K, Lukowicz P (2014) Sensor placement variations in wearable activity recognition. IEEE Pervasive Comput 13(4):32–41 CrossRef Kunze K, Lukowicz P (2014) Sensor placement variations in wearable activity recognition. IEEE Pervasive Comput 13(4):32–41 CrossRef
17.
go back to reference Kunze K, Lukowicz P, Partridge K, Begole B (2009) Which way am i facing: inferring horizontal device orientation from an accelerometer signal. In: International symposium on wearable computers, pp 149–150 Kunze K, Lukowicz P, Partridge K, Begole B (2009) Which way am i facing: inferring horizontal device orientation from an accelerometer signal. In: International symposium on wearable computers, pp 149–150
18.
go back to reference Yurtman A, Barshan B (2017) Activity recognition invariant to sensor orientation with wearable motion sensors. Sensors 17(8):1838 CrossRef Yurtman A, Barshan B (2017) Activity recognition invariant to sensor orientation with wearable motion sensors. Sensors 17(8):1838 CrossRef
19.
go back to reference Yurtman A, Barshan B, Fidan B (2018) Activity recognition invariant to wearable sensor unit orientation using differential rotational transformations represented by quaternions. Sensors 18(8):2725 CrossRef Yurtman A, Barshan B, Fidan B (2018) Activity recognition invariant to wearable sensor unit orientation using differential rotational transformations represented by quaternions. Sensors 18(8):2725 CrossRef
20.
go back to reference Banos O, Damas M, Guillen A, Herrera L-J, Pomares H, Rojas I, Villalonga C (2015) Multi-sensor fusion based on asymmetric decision weighting for robust activity recognition. Neural Process Lett 42(1):5–26 CrossRef Banos O, Damas M, Guillen A, Herrera L-J, Pomares H, Rojas I, Villalonga C (2015) Multi-sensor fusion based on asymmetric decision weighting for robust activity recognition. Neural Process Lett 42(1):5–26 CrossRef
21.
go back to reference Banos O, Damas M, Pomares H, Rojas I (2012) On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 12(6):8039–8054 CrossRef Banos O, Damas M, Pomares H, Rojas I (2012) On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 12(6):8039–8054 CrossRef
22.
go back to reference Banos O, Toth MA, Damas M, Pomares H, Rojas I (2014) Dealing with the effects of sensor displacement in wearable activity recognition. Sensors 14(6):9995–10023 CrossRef Banos O, Toth MA, Damas M, Pomares H, Rojas I (2014) Dealing with the effects of sensor displacement in wearable activity recognition. Sensors 14(6):9995–10023 CrossRef
23.
go back to reference Lam A, Roy-Chowdhury AK, Shelton CR (2010) Interactive event search through transfer learning. In: Asian conference on computer vision. Springer, Berlin, pp 157–170 Lam A, Roy-Chowdhury AK, Shelton CR (2010) Interactive event search through transfer learning. In: Asian conference on computer vision. Springer, Berlin, pp 157–170
24.
go back to reference Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw 22(2):199–210 CrossRef Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw 22(2):199–210 CrossRef
25.
go back to reference Nater F, Tommasi T, Grabner H, Van Gool L, Caputo B (2011) Transferring activities: updating human behavior analysis. In: IEEE international conference on computer vision workshops, pp 1737–1744 Nater F, Tommasi T, Grabner H, Van Gool L, Caputo B (2011) Transferring activities: updating human behavior analysis. In: IEEE international conference on computer vision workshops, pp 1737–1744
26.
go back to reference Al-Halah Z, Rybok L, Stiefelhagen R (2016) Transfer metric learning for action similarity using high-level semantics. Pattern Recogn Lett 72:82–90 CrossRef Al-Halah Z, Rybok L, Stiefelhagen R (2016) Transfer metric learning for action similarity using high-level semantics. Pattern Recogn Lett 72:82–90 CrossRef
27.
go back to reference Zheng VW, Hu DH, Yang Q (2009) Cross-domain activity recognition. In: International conference on ubiquitous computing, pp 61–70 Zheng VW, Hu DH, Yang Q (2009) Cross-domain activity recognition. In: International conference on ubiquitous computing, pp 61–70
28.
go back to reference Hu DH, Zheng VW, Yang Q (2011) Cross-domain activity recognition via transfer learning. Pervasive Mob Comput 7(3):344–358 CrossRef Hu DH, Zheng VW, Yang Q (2011) Cross-domain activity recognition via transfer learning. Pervasive Mob Comput 7(3):344–358 CrossRef
29.
go back to reference Chiang Y, Hsu JY (2012) Knowledge transfer in activity recognition using sensor profile. In: International conference on ubiquitous intelligence computing and international conference on autonomic trusted computing, pp 180–187 Chiang Y, Hsu JY (2012) Knowledge transfer in activity recognition using sensor profile. In: International conference on ubiquitous intelligence computing and international conference on autonomic trusted computing, pp 180–187
30.
go back to reference Feuz K, Cook DJ (2014) Heterogeneous transfer learning for activity recognition using heuristic search techniques. Int J Pervasive Comput Commun 10(4):393–418 CrossRef Feuz K, Cook DJ (2014) Heterogeneous transfer learning for activity recognition using heuristic search techniques. Int J Pervasive Comput Commun 10(4):393–418 CrossRef
31.
go back to reference Ying JJ-C, Lin B-H, Tseng VS, Hsieh S-Y (2015) Transfer learning on high variety domains for activity recognition. In: ASE BigData and social informatics. ACM, pp 37:1–37:6 Ying JJ-C, Lin B-H, Tseng VS, Hsieh S-Y (2015) Transfer learning on high variety domains for activity recognition. In: ASE BigData and social informatics. ACM, pp 37:1–37:6
32.
go back to reference Feuz KD, Cook DJ (2015) Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (FSR). ACM Trans Intell Syst Technol 6(1):3:1–3:27 CrossRef Feuz KD, Cook DJ (2015) Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (FSR). ACM Trans Intell Syst Technol 6(1):3:1–3:27 CrossRef
33.
go back to reference Chen W-H, Cho P-C, Jiang Y-L (2017) Activity recognition using transfer learning. Sens Mater 29(7):897–904 Chen W-H, Cho P-C, Jiang Y-L (2017) Activity recognition using transfer learning. Sens Mater 29(7):897–904
34.
go back to reference Morales FJO, Roggen D (2016) Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In: Proceedings of the 2016 ACM international symposium on wearable computers. ACM, pp 92–99 Morales FJO, Roggen D (2016) Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In: Proceedings of the 2016 ACM international symposium on wearable computers. ACM, pp 92–99
35.
go back to reference Rokni SA, Ghasemzadeh H (2018) Autonomous training of activity recognition algorithms in mobile sensors: a transfer learning approach in context-invariant views. IEEE Trans Mob Comput 17(8):1764–1777 CrossRef Rokni SA, Ghasemzadeh H (2018) Autonomous training of activity recognition algorithms in mobile sensors: a transfer learning approach in context-invariant views. IEEE Trans Mob Comput 17(8):1764–1777 CrossRef
36.
go back to reference Calatroni A, Villalonga C, Roggen D, Troester G. (2009) Context cells: towards lifelong learning in activity recognition systems. In: European conference on smart sensing and context, pp 121–134 Calatroni A, Villalonga C, Roggen D, Troester G. (2009) Context cells: towards lifelong learning in activity recognition systems. In: European conference on smart sensing and context, pp 121–134
37.
go back to reference Calatroni A, Roggen D, Troester G (2011) Automatic transfer of activity recognition capabilities between body-worn motion sensors: training newcomers to recognize locomotion. In: International conference on networked sensing systems Calatroni A, Roggen D, Troester G (2011) Automatic transfer of activity recognition capabilities between body-worn motion sensors: training newcomers to recognize locomotion. In: International conference on networked sensing systems
38.
go back to reference Sjoeberg J, Zhang Q, Ljung L, Benveniste A, Delyon B, Glorennec P-Y, Hjalmarsson H, Juditsky A (1995) Nonlinear black-box modeling in system identification: a unified overview. Automatica 31(12):1691–1724 MathSciNetCrossRef Sjoeberg J, Zhang Q, Ljung L, Benveniste A, Delyon B, Glorennec P-Y, Hjalmarsson H, Juditsky A (1995) Nonlinear black-box modeling in system identification: a unified overview. Automatica 31(12):1691–1724 MathSciNetCrossRef
39.
go back to reference Hong X, Mitchell RJ, Chen S, Harris CJ, Li K, Irwin GW (2008) Model selection approaches for non-linear system identification: a review. Int J Syst Sci 39(10):925–946 MathSciNetCrossRef Hong X, Mitchell RJ, Chen S, Harris CJ, Li K, Irwin GW (2008) Model selection approaches for non-linear system identification: a review. Int J Syst Sci 39(10):925–946 MathSciNetCrossRef
40.
go back to reference Pota HR (1996) MIMO systems-transfer function to state-space. IEEE Trans Educ 39(1):97–99 CrossRef Pota HR (1996) MIMO systems-transfer function to state-space. IEEE Trans Educ 39(1):97–99 CrossRef
41.
go back to reference Berndt DJ, Clifford J (1994) Using dynamic time warping to find patterns in time series. In: AAAI—KDD workshop, pp 359–370 Berndt DJ, Clifford J (1994) Using dynamic time warping to find patterns in time series. In: AAAI—KDD workshop, pp 359–370
42.
go back to reference Ye L, Keogh E (2011) Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data Min Knowl Disc 22(1–2):149–182 MathSciNetCrossRef Ye L, Keogh E (2011) Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data Min Knowl Disc 22(1–2):149–182 MathSciNetCrossRef
44.
go back to reference Biswas KK, Basu SK (2011) Gesture recognition using microsoft kinect®. In: International conference on automation, robotics and applications. IEEE, pp 100–103 Biswas KK, Basu SK (2011) Gesture recognition using microsoft kinect®. In: International conference on automation, robotics and applications. IEEE, pp 100–103
45.
go back to reference Hongyong T, Youling Y (2012) Finger tracking and gesture recognition with kinect. In: International conference on computer and information technology, pp 214–218 Hongyong T, Youling Y (2012) Finger tracking and gesture recognition with kinect. In: International conference on computer and information technology, pp 214–218
46.
go back to reference Foerster K, Biasiucci A, Chavarriaga R, Millan JDR, Roggen D, Troester G (2010) On the use of brain decoded signals for online user adaptive gesture recognition systems. In: International conference on pervasive computing, pp 427–444 Foerster K, Biasiucci A, Chavarriaga R, Millan JDR, Roggen D, Troester G (2010) On the use of brain decoded signals for online user adaptive gesture recognition systems. In: International conference on pervasive computing, pp 427–444
47.
go back to reference Förster K, Monteleone S, Calatroni A, Roggen D, Tröster G (2010) Incremental KNN classifier exploiting correct—error teacher for activity recognition. In: International conference on machine learning and applications, pp 445–450 Förster K, Monteleone S, Calatroni A, Roggen D, Tröster G (2010) Incremental KNN classifier exploiting correct—error teacher for activity recognition. In: International conference on machine learning and applications, pp 445–450
48.
go back to reference Cover TM, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef Cover TM, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27 CrossRef
49.
go back to reference Yazdizadeh A, Khorasani K (2002) Adaptive time delay neural network structures for nonlinear system identification. Neurocomputing 47(1–4):207–240 CrossRef Yazdizadeh A, Khorasani K (2002) Adaptive time delay neural network structures for nonlinear system identification. Neurocomputing 47(1–4):207–240 CrossRef
50.
go back to reference Oussar Y, Dreyfus G (2001) How to be a gray box: dynamic semi-physical modeling. Neural Netw 14(9):1161–1172 CrossRef Oussar Y, Dreyfus G (2001) How to be a gray box: dynamic semi-physical modeling. Neural Netw 14(9):1161–1172 CrossRef
51.
go back to reference Rey VF, Hevesi P, Kovalenko O, Lukowicz P (2019) Let there be IMU data: generating training data for wearable, motion sensor based activity recognition from monocular RGB videos. In: Proceedings of the 2019 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2019 ACM international symposium on wearable computers, pp 699–708 Rey VF, Hevesi P, Kovalenko O, Lukowicz P (2019) Let there be IMU data: generating training data for wearable, motion sensor based activity recognition from monocular RGB videos. In: Proceedings of the 2019 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2019 ACM international symposium on wearable computers, pp 699–708
52.
go back to reference Al-Halah Z, Rybok L, Stiefelhagen R (2014) What to transfer? High-level semantics in transfer metric learning for action similarity. In: International conference on pattern recognition, pp 2775–2780 Al-Halah Z, Rybok L, Stiefelhagen R (2014) What to transfer? High-level semantics in transfer metric learning for action similarity. In: International conference on pattern recognition, pp 2775–2780
Metadata
Title
Opportunistic Activity Recognition in IoT Sensor Ecosystems via Multimodal Transfer Learning
Authors
Oresti Banos
Alberto Calatroni
Miguel Damas
Hector Pomares
Daniel Roggen
Ignacio Rojas
Claudia Villalonga
Publication date
31-03-2021
Publisher
Springer US
Published in
Neural Processing Letters / Issue 5/2021
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI
https://doi.org/10.1007/s11063-021-10468-z

Other articles of this Issue 5/2021

Neural Processing Letters 5/2021 Go to the issue