Skip to main content
SpringerLink
Log in

Literature Review on Transfer Learning for Human Activity Recognition Using Mobile and Wearable Devices with Environmental Technology

  • Survey Article
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Activity recognition systems utilise data from sensors in mobile, environmental and wearable devices, ubiquitously available to individuals. It is a growing research area within intelligent systems that aims to model and identify human physical, cognitive and social actions, patterns and skills. They typically rely on supervised machine-learning approaches, in which the cost of gathering and labelling data is high due to the diverse, interleaved and dynamic nature of human behaviour. Transfer learning is an approach in which previously learned knowledge is utilised to model a new but related setting. For instance, it can reuse existing knowledge to recognise activities performed by different types of users, using different sensor technologies and in different environmental conditions. As the adoption of Internet of Thing devices increases, mobile and wearable sensing is becoming pervasive, and more challenging behaviour recognition activities are being tackled. Yet, the availability of more data does not necessarily translate to better recognition models, if these data are not properly labelled. Thus, the importance of taking advantage of transfer learning to advance the field of activity recognition. This literature review summarises the transfer learning techniques and explores the benefits of combining mobile and wearable devices with environmental sensors in support of transfer learning. We also discuss the maturity of transfer learning by analysing the validation method used in the papers reviewed. Overall, 170 selected articles published between 2014 and 2019 were reviewed following the Okali and Schabram methodology. Findings show an increase of 41% of publications when comparing the output of 2019 against the average number of papers published in the previous 5 years (2014–2018). Inertial sensors such as accelerometers and gyroscopes, are the most frequently used. Feature and instance representation are mature techniques for transfer knowledge. Unsupervised learning across users is a typical application, and shallow techniques and active learning are areas of opportunity in transfer learning methodologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Accessed July 31st, 2019: www.techcrunch.com/2017/08/24/global-wearables-market-to-grow-17-in-2017-310m-devices-sold-30-5bn-revenue-gartner.

  2. Accessed July 31st, 2019: www.statista.com/statistics/302482/wearable-device-market-value/.

Abbreviations

ADL:

Activity of daily living

AE:

Across environment

AL:

Across location

AR:

Across sampling rate

ART:

Adaptive resonance theory

AS:

Across sensor

AT:

Across task

AU:

Across user

CNN:

Deep convolutional networks

DNN:

Deep neuronal network

HAR:

Human activity recognition

IL:

Inductive learning

IoT:

Internet of things

IS:

Information systems

MWE:

Mobile/wearable/environmental

T/L:

Teacher/learner

TL:

Transductive learning

UL:

Unsupervised learning

References

  1. Aggarwal JK, Xia L. Human activity recognition from 3D data: a review. Pattern Recognit Lett. 2014;. https://doi.org/10.1016/j.patrec.2014.04.011.

    Article  Google Scholar 

  2. Alam MAU, Roy N. Unseen activity recognitions: a hierarchical active transfer learning approach. Proc Int Conf Distrib Comput Syst. 2017;. https://doi.org/10.1109/ICDCS.2017.264.

    Article  Google Scholar 

  3. Appelboom G, Camacho E, Abraham ME, Bruce SS, Dumont ELP, Zacharia BE, D’Amico R, Slomian J, Reginster JY, Bruyère O, Sander CE. Smart wearable body sensors for patient self-assessment and monitoring. Arch Public Health. 2014;. https://doi.org/10.1186/2049-3258-72-28.

    Article  Google Scholar 

  4. Bashir SA, Petrovski A, Doolan D. UDetect: unsupervised concept change detection for mobile activity recognition. In: Proceedings of the 14th international conference on advances in mobile computing and multi media (MoMM ’16). New York: ACM; 2016. p. 20–27. https://doi.org/10.1145/3007120.3007144.

  5. Benedetti F, Carlino E, Piedimonte A. Increasing uncertainty in CNS clinical trials: the role of placebo, nocebo, and Hawthorne effects. Lancet Neurol. 2016;. https://doi.org/10.1016/S1474-4422(16)00066-1.

    Article  Google Scholar 

  6. Benini L, Farella E, Guiducci C. Wireless sensor networks: enabling technology for ambient intelligence. Microelectron J. 2006;37(12):1639–49. https://doi.org/10.1016/j.mejo.2006.04.021.

    Article  Google Scholar 

  7. Berlin E, Van Laerhoven K. Detecting leisure activities with dense motif discovery. In: UbiComp’12 - Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 2012. https://doi.org/10.1145/2370216.2370257.

  8. Bieber, G. et al. Sensor requirements for activity recognition on smart watches. In: ACM International Conference Proceeding Series. 2013. https://doi.org/10.1145/2504335.2504407.

  9. Boualia SN, Essoukri AN. Pose-based human activity recognition: a review. In: 2019 15th international wireless communications mobile computing conference (IWCMC). 2019. p. 1468–1475. https://doi.org/10.1109/IWCMC.2019.8766694.

  10. Campbell JP, Maxey VA, Watson WA. Hawthorne effect: implications for prehospital research. Ann Emerg Med. 1995;. https://doi.org/10.1016/S0196-0644(95)70009-9.

    Article  Google Scholar 

  11. Cao L, Liu Z, Huang TS. Cross-dataset action detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. 2010. https://doi.org/10.1109/CVPR.2010.5539875.

  12. Capponi A, Fiandrino C, Kantarci B, Foschini L, Kliazovich D, Bouvry P. A survey on mobile crowdsensing systems: challenges, solutions, and opportunities. IEEE Commun Surv Tutor. 2019;21(3):2419–65. https://doi.org/10.1109/COMST.2019.2914030.

    Article  Google Scholar 

  13. Carpenter GA, Grossberg S, Reynolds JH. ARTMAP: supervised real-time learning and classification of nonstationary data by a self-organizing neural network. Neural Netw. 1991;. https://doi.org/10.1016/0893-6080(91)90012-T.

    Article  Google Scholar 

  14. Chen L, Nugent CD, Wang H. A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng. 2012;. https://doi.org/10.1109/TKDE.2011.51.

    Article  Google Scholar 

  15. Chikhaoui B, Wang S, Pigot H. ADR-SPLDA: activity discovery and recognition by combining sequential patterns and latent Dirichlet allocation. Pervasive Mob Comput. 2012;8(6):845–62. https://doi.org/10.1016/j.pmcj.2012.08.004.

    Article  Google Scholar 

  16. Cook D, Feuz KD, Krishnan NC. Transfer learning for activity recognition: a survey. Knowl Inf Syst. 2013;36(3):537–56. https://doi.org/10.1007/s10115-013-0665-3.

    Article  Google Scholar 

  17. Dhillon JK, Chandni, Kushwaha AKS. A recent survey for human activity recognition based on deep learning approach. In: 2017 fourth international conference on image information processing (ICIIP). 2017. p. 1–6. https://doi.org/10.1109/ICIIP.2017.8313715.

  18. Dimitrievski A, Zdravevski E, Lameski P, Trajkovik V. A survey of ambient assisted living systems: challenges and opportunities. In: 2016 IEEE 12th international conference on intelligent computer communication and processing (ICCP). 2016. p. 49–53. https://doi.org/10.1109/ICCP.2016.7737121.

  19. Ding R, Li X, Nie L, Li J, Si X, Chu D, Liu G, Zhan D. Empirical study and improvement on deep transfer learning for human activity recognition. Sensors. 2019;19:1. https://doi.org/10.3390/s19010057.

    Article  Google Scholar 

  20. Du X, Farrahi K, Niranjan M. Transfer learning across human activities using a cascade neural network architecture. In: Proceedings of the 23rd international symposium on wearable computers (ISWC ’19). New York: Association for Computing Machinery; 2019. p. 35–44. https://doi.org/10.1145/3341163.3347730.

  21. Feuz KD, Cook DJ. Collegial activity learning between heterogeneous sensors. Knowl Inf Syst. 2017;. https://doi.org/10.1007/s10115-017-1043-3.

    Article  Google Scholar 

  22. Fortin-Simard D, Bilodeau JS, Bouchard K, Gaboury S, Bouchard B, Bouzouane A. Exploiting passive RFID technology for activity recognition in smart homes. IEEE Intell Syst. 2015;. https://doi.org/10.1109/MIS.2015.18.

    Article  Google Scholar 

  23. Francis L. Unsupervised learning. In: Predictive Modeling Applications in Actuarial Science: Volume I: Predictive Modeling Techniques. 2014. https://doi.org/10.1017/CBO9781139342674.012.

  24. Gama J, Žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A. A survey on concept drift adaptation. Comput Surv. 2014;. https://doi.org/10.1145/2523813.

    Article  MATH  Google Scholar 

  25. Gupta A, Priyadarshini J. Smart monitoring of automatic teller machine—a survey. In: Nath V, Mandal JK, editors. Proceedings of the third international conference on microelectronics, computing and communication systems. Singapore: Springer Singapore; 2019. p. 109–15.

  26. Hachiya H, Sugiyama M, Ueda N. Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition. Neurocomputing. 2012;. https://doi.org/10.1016/j.neucom.2011.09.016.

    Article  Google Scholar 

  27. Hu DH, Yang Q. Transfer learning for activity recognition via sensor mapping. In: IJCAI international joint conference on artificial intelligence. 2011. https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-328.

  28. Hu L, Chen Y, Wang S, Wang J, Shen J, Jiang X, Shen Z. Less annotation on personalized activity recognition using context data. In: Proceedings—13th IEEE international conference on ubiquitous intelligence and computing, 13th IEEE international conference on advanced and trusted computing, 16th IEEE international conference on scalable computing and communications, IEEE international conference on cloud and big data computing, IEEE international conference on internet of people and IEEE smart world congress and workshops, UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld 2016. 2017. https://doi.org/10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0066.

  29. Huynh T, Fritz M, Schiele B. Discovery of activity patterns using topic models. In: UbiComp 2008 - Proceedings of the 10th International Conference on Ubiquitous Computing. 2008. https://doi.org/10.1145/1409635.1409638.

  30. Jain AK, Murty MN, Flynn PJ. Data clustering: a review. Comput Surv. 1999. https://doi.org/10.1145/331499.331504.

  31. Jiang H, Cai C, Ma X, Yang Y, Liu J. Smart home based on WiFi sensing: a survey. IEEE Access. 2018;2018(6):13317–25. https://doi.org/10.1109/ACCESS.2018.2812887.

    Article  Google Scholar 

  32. Jobanputra C, Bavishi J, Doshi N. Human activity recognition: a survey. Proc Comput Sci. 2019;155(2019):698–703. https://doi.org/10.1016/j.procs.2019.08.100. The 16th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2019), The 14th International Conference on Future Networks and Communications (FNC-2019),The 9th International Conference on Sustainable Energy Information Technology.

  33. Kazemi L, Shahabi C. GeoCrowd: enabling query answering with spatial crowdsourcing. In: Proc. 20th ACM int. conf. adv. geogr. inf. syst. (SIGSPATIAL). 2012. https://doi.org/10.1145/2424321.2424346.

  34. Kwon Y, Kang K, Bae C. Unsupervised learning for human activity recognition using smartphone sensors. Expert Syst Appl. 2014;. https://doi.org/10.1016/j.eswa.2014.04.037.

    Article  Google Scholar 

  35. Lane ND, Miluzzo E, Lu H, Peebles D, Choudhury T, Campbell AT. A survey of mobile phone sensing. IEEE Commun Mag. 2010;48(9):140–50. https://doi.org/10.1109/MCOM.2010.5560598.

    Article  Google Scholar 

  36. Lara ÓD, Labrador MA. A survey on human activity recognition using wearable sensors. IEEE Commun Surv Tutor. 2013;. https://doi.org/10.1109/SURV.2012.110112.00192.

    Article  Google Scholar 

  37. Falk Leichsenring. Randomized controlled versus naturalistic studies: a new research agenda. Bull Menninger Clinic. 2004;. https://doi.org/10.1521/bumc.68.2.137.35952.

    Article  Google Scholar 

  38. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE. A survey of deep neural network architectures and their applications. Neurocomputing. 2017;. https://doi.org/10.1016/j.neucom.2016.12.038.

    Article  Google Scholar 

  39. Maekawa T, Nakai D, Ohara K, Namioka Y. Toward practical factory activity recognition: unsupervised understanding of repetitive assembly work in a factory. In: UbiComp 2016. 2016. https://doi.org/10.1145/2971648.2971721.

  40. Michalski RS. A theory and methodology of inductive learning. Artif Intell. 1983;. https://doi.org/10.1016/0004-3702(83)90016-4.

    Article  MathSciNet  Google Scholar 

  41. Morales FJO, Roggen D. Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In: Proceedings of the 2016 ACM international symposium on wearable computers (ISWC ’16). New York: Association for Computing Machinery; 2016. p. 92–9. https://doi.org/10.1145/2971763.2971764.

  42. Okoli C, Schabram K. A guide to conducting a systematic literature review of information systems research. SSRN Electron. J. 2012. https://doi.org/10.2139/ssrn.1954824.

    Article  Google Scholar 

  43. Pan SJ, Yang Q. A survey on transfer learning. 2010. https://doi.org/10.1109/TKDE.2009.191.

  44. Qiang Y, Pan SJ. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010;22(10):1345–59.

    Article  Google Scholar 

  45. Rawassizadeh R, Price BA, Petre M. Wearables: has the age of smartwatches finally arrived? Commun ACM. 2014;. https://doi.org/10.1145/2629633.

    Article  Google Scholar 

  46. Riboni D et al. Unsupervised recognition of interleaved activities of daily living through ontological and probabilistic reasoning. In: UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2016. https://doi.org/10.1145/2971648.2971691.

  47. Rohrbach M, Ebert S, Schiele B. Transfer learning in a transductive setting. In: Advances in Neural Information Processing Systems. 2013.

  48. Saeedi R, Ghasemzadeh H, Gebremedhin AH. Transfer learning algorithms for autonomous reconfiguration of wearable systems. In: Proceedings—2016 IEEE international conference on big data, big data 2016. 2016. https://doi.org/10.1109/BigData.2016.7840648.

  49. Sánchez D, Tentori M, Favela J. Activity recognition for the smart hospital. IEEE Intell Syst. 2008;. https://doi.org/10.1109/MIS.2008.18.

    Article  Google Scholar 

  50. Seiter J, Chiu WC, Fritz M, Amft O, Tröster G. Joint segmentation and activity discovery using semantic and temporal priors. In: 2015 IEEE international conference on pervasive computing and communications, PerCom 2015. 2015. https://doi.org/10.1109/PERCOM.2015.7146511.

  51. Sjøberg DIK, Hannay JE, Hansen O, Kampenes VB, Karahasanović A, Liborg NK, Rekdal AC. A survey of controlled experiments in software engineering. IEEE Trans Softw Eng. 2005;. https://doi.org/10.1109/TSE.2005.97.

    Article  Google Scholar 

  52. Stikic M, Larlus D, Schiele B. Multi-graph based semi-supervised learning for activity recognition. In: 2009 international symposium on wearable computers. 2009. p. 85–92. https://doi.org/10.1109/ISWC.2009.24.

  53. Stikic M, Van Laerhoven K, Schiele B. Exploring semi-supervised and active learning for activity recognition. In: Proceedings—international symposium on wearable computers, ISWC. 2008. https://doi.org/10.1109/ISWC.2008.4911590.

  54. Storm FA, Heller BW, Mazzà C. Step detection and activity recognition accuracy of seven physical activity monitors. PLoS One. 2015;. https://doi.org/10.1371/journal.pone.0118723.

    Article  Google Scholar 

  55. Su X, Tong H, Ji P. Activity recognition with smartphone sensors. Tsinghua Sci Technol. 2014;. https://doi.org/10.1109/TST.2014.6838194.

    Article  Google Scholar 

  56. Suh MK, Chen CA, Woodbridge J, Tu MK, Kim JI, Nahapetian A, Evangelista LS, Sarrafzadeh M. A remote patient monitoring system for congestive heart failure. J Med Syst. 2011;. https://doi.org/10.1007/s10916-011-9733-y.

    Article  Google Scholar 

  57. Sun S. A survey of multi-view machine learning. Neural Comput Appl. 2013;. https://doi.org/10.1007/s00521-013-1362-6.

    Article  Google Scholar 

  58. Taraldsen K, Chastin SFM, Riphagen II, Vereijken B, Helbostad JL. Physical activity monitoring by use of accelerometer-based body-worn sensors in older adults: a systematic literature review of current knowledge and applications. Maturitas. 2012;. https://doi.org/10.1016/j.maturitas.2011.11.003.

    Article  Google Scholar 

  59. Vahdatpour A, Amini N, Sarrafzadeh M, On-body device localization for health and medical monitoring applications. In: 2011 IEEE international conference on pervasive computing and communications, PerCom 2011. 2011. https://doi.org/10.1109/PERCOM.2011.5767593.

  60. van Kasteren T, Englebienne G, Kröse BJA. Recognizing activities in multiple contexts using transfer learning. In: AAAI AI in eldercare symposium. 2008.

  61. Wang J, Chen Y, Gu Y, Xiao Y, Pan H. SensoryGANs: an effective generative adversarial framework for sensor-based human activity recognition. In: Proceedings of the international joint conference on neural networks. 2018. https://doi.org/10.1109/IJCNN.2018.8489106.

  62. Yang CC, Hsu YL. A review of accelerometry-based wearable motion detectors for physical activity monitoring. Sensors. 2010;. https://doi.org/10.3390/s100807772.

    Article  Google Scholar 

  63. Yin XC, Yang C, Pei WY, Hao HW. Shallow classification or deep learning: an experimental study. In: Proceedings—international conference on pattern recognition. 2014. https://doi.org/10.1109/ICPR.2014.333.

  64. Zhu S, Xu J, Guo H, Liu Q, Wu S, Wang H. Indoor human activity recognition based on ambient radar with signal processing and machine learning. In: 2018 IEEE international conference on communications (ICC). 2018. p. 1–6. https://doi.org/10.1109/ICC.2018.8422107.

Download references

Acknowledgements

This project has received partial funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 734355.

Author information

Authors and Affiliations

  1. Ulster University, Newtownabbey, Belfast, Northern Ireland, UK

    Netzahualcoyotl Hernandez & Ian McChesney

  2. Raytelligence AB, Klammerdammsgatan, Halmstad, Sweden

    Jens Lundström

  3. CICESE, Ensenada, Baja California, Mexico

    Jesus Favela

  4. Hasso Plattner Institute, University of Potsdam, Potsdam, Germany

    Bert Arnrich

Authors
  1. Netzahualcoyotl Hernandez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jens Lundström
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jesus Favela
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ian McChesney
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bert Arnrich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Netzahualcoyotl Hernandez.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hernandez, N., Lundström, J., Favela, J. et al. Literature Review on Transfer Learning for Human Activity Recognition Using Mobile and Wearable Devices with Environmental Technology. SN COMPUT. SCI. 1, 66 (2020). https://doi.org/10.1007/s42979-020-0070-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-020-0070-4

Keywords

Search

Navigation