Abstract
Computer-assisted cognitive training can help patients affected by several illnesses alleviate their cognitive deficits or healthy people improve their mental performance. In most computer-based systems, training sessions consist of graded exercises, which should ideally be able to gradually improve the trainee’s cognitive functions. Indeed, adapting the difficulty of the exercises to how individuals perform in their execution is crucial to improve the effectiveness of cognitive training activities. In this article, we propose the use of reinforcement learning (RL) to learn how to automatically adapt the difficulty of computerized exercises for cognitive training. In our approach, trainees’ performance in performed exercises is used as a reward to learn a policy that changes over time the values of the parameters that determine exercise difficulty. We illustrate a method to be initially used to learn difficulty-variation policies tailored for specific categories of trainees, and then to refine these policies for single individuals. We present the results of two user studies that provide evidence for the effectiveness of our method: a first study, in which a student category policy obtained via RL was found to have better effects on the cognitive function than a standard baseline training that adopts a mechanism to vary the difficulty proposed by neuropsychologists, and a second study, demonstrating that adding an RL-based individual customization further improves the training process.
- [1] . 2021. Brainer | Riabilitazione Cognitiva. Retrieved November 17, 2021 from http://www.brainer.it.Google Scholar
- [2] . 2021. Cognitive Therapy Tools, Cognitive Rehabilitation. Retrieved November 17, 2021 from https://www.happyneuronpro.com.Google Scholar
- [3] . 2021. Cognitive Training Program: CogniPlus—SCHUHFRIED. Retrieved November 17, 2021 from https://www.schuhfried.com/cogniplus.Google Scholar
- [4] . 2021. Erica. Retrieved November 17, 2021 from http://www.erica.giuntios.it.Google Scholar
- [5] . 2021. NeuroTracker | #1 Cognitive Training System in the World. Retrieved November 17, 2021 from https://neurotracker.net.Google Scholar
- [6] . 2021. RehaCom—Computer-Based Cognitive Therapy. Retrieved November 17, 2021 from https://www.rehacom.co.uk.Google Scholar
- [7] . 2013. Computerized working memory training: Can it lead to gains in cognitive skills in students? Computers in Human Behavior 29, 3 (2013), 632–638. Google ScholarDigital Library
- [8] . 2013. Evaluation of a computer-based neuropsychological training in children with attention-deficit hyperactivity disorder (ADHD). NeuroRehabilitation 32, 3 (2013), 555–62. Google ScholarCross Ref
- [9] . 2016. Memory training and benefits for quality of life in the elderly: A case report. Dementia & Neuropsychologia 10, 2 (April–June 2016), 152–155. Google ScholarCross Ref
- [10] . 2013. Patient tailored virtual rehabilitation. In Converging Clinical and Engineering Research on Neurorehabilitation, , , and (Eds.). Springer, Berlin, Germany, 879–883.Google Scholar
- [11] . 2016. Preliminary study of a rehabilitation program based on attentional processes to treat auditory hallucinations. Cognitive Neuropsychiatry 21, 4 (
July 2016), 315–334.Google ScholarCross Ref - [12] . 2019. Survey on brain-computer interface: An emerging computational intelligence paradigm. ACM Computing Surveys 52, 1 (2019), 1–32.Google ScholarDigital Library
- [13] . 2015. Reinforcement learning for adaptive threshold control of restorative brain-computer interfaces: A Bayesian simulation. Frontiers in Neuroscience 9 (2015), 36. Google ScholarCross Ref
- [14] . 2017. Action-based cognitive remediation for individuals with serious mental illnesses: Effects of real-world simulations and goal setting on functional and vocational outcomes. Psychiatric Rehabilitation Journal 40, 1 (2017), 53–60.Google ScholarCross Ref
- [15] . 2017. New Frontiers in Neuropsychology. The Padua Rehabilitation Tool: A New Software for Rehabilitation Using Touch-Screen Technology. Ph.D. Dissertation. University of Padua.Google Scholar
- [16] . 2016. Padua rehabilitation tool: A pilot study on patients with dementia. In Revised Selected Papers of the 4th International Conference on Games and Learning Alliance—Volume 9599 (GALA 2015). Springer-Verlag, Berlin, Germany, 292–301. Google ScholarDigital Library
- [17] . 2013. Do neuropsychological and social cognition abilities in schizophrenia change after intensive cognitive training? A pilot study. Clinical Neuropsychiatry 10, 5 (2013), 202–212.Google Scholar
- [18] . 2010. Personalised context-aware ubiquitous learning system for supporting effective English vocabulary learning. Interactive Learning Environments 18, 4 (2010), 341–364.Google ScholarCross Ref
- [19] . 2013. Rasch Models in Health. Wiley Online Library.Google Scholar
- [20] . 2012. Evaluating the integration of fuzzy logic into the student model of a web-based learning environment. Expert Systems with Applications 39, 18 (2012), 13127–13134. Google ScholarDigital Library
- [21] . 2020. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine 109 (2020), 101964. Google ScholarCross Ref
- [22] . 2018. Intelligent tutoring systems for programming education: A systematic review. In Proceedings of the 20th Australasian Computing Education Conference (ACE’18). ACM, New York, NY, 53–62. Google ScholarDigital Library
- [23] . 2011. A randomized, controlled trial of computer-assisted cognitive remediation for schizophrenia. Schizophrenia Research 125, 2-3 (
Nov. 2011), 284–290.Google ScholarCross Ref - [24] . 2013. A randomized controlled pilot study assessing the feasibility of combined motor–cognitive training and its effect on gait characteristics in the elderly. Clinical Rehabilitation 27, 3 (2013), 215–225. Google ScholarCross Ref
- [25] . 2012. Reinforcement learning adaptive control for upper limb rehabilitation robot based on fuzzy neural network. In Proceedings of the 31st Chinese Control Conference. 5157–5161.Google Scholar
- [26] . 2017. Building adaptive tutoring model using artificial neural networks and reinforcement learning. In Proceedings of the 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT’17). 460–462. Google ScholarCross Ref
- [27] . 2020. Enhancing cognitive rehabilitation in multiple sclerosis with a disease-specific tool. Disability and Rehabilitation: Assistive Technology 2020 (2020), 1–14. Google ScholarCross Ref
- [28] . 2016. Affecting off-task behaviour: How affect-aware feedback can improve student learning. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK’16). ACM, New York, NY, 104–113. Google ScholarDigital Library
- [29] . 1977. Paced auditory serial-addition task: A measure of recovery from concussion. Perceptual and Motor Skills 44, 2 (1977), 367–373.Google ScholarCross Ref
- [30] . 2017. The transfer of skills from cognitive and physical training to activities of daily living: A randomised controlled study. European Journal of Ageing 14, 2 (
June 2017), 133–142. Google ScholarCross Ref - [31] . 2019. Domains of cognition and their assessment. Dialogues in Clinical Neuroscience 21, 3 (Sept. 2019), 227–237. Google Scholar
- [32] . 2015. A reinforcement learning approach to gait training improves retention. Frontiers in Human Neuroscience 9 (2015), 459. Google ScholarCross Ref
- [33] . 2018. The effects of perceptual-cognitive training with NeuroTracker on executive brain functions among elite athletes. Cogent Psychology 5, 1 (2018), Article 1544105. Google ScholarCross Ref
- [34] . 2014. The effectiveness of computerized cognitive rehabilitation training program in improving cognitive abilities of schizophrenia clients. Iran Journal of Psychiatry 9, 4 (
Oct. 2014), 209–215.Google Scholar - [35] . 2016. Computerized cognitive training and functional recovery in major depressive disorder: A meta-analysis. Journal of Affective Disorders 189, 1 (2016), 184–191.Google ScholarCross Ref
- [36] . 2018. Intelligent tutoring systems: A systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments 29, 1 (2018), 1–22. Google ScholarCross Ref
- [37] . 2001. Adaptability and flexibility of the human motor system: Implications for neurological rehabilitation. Neural Plasticity 8, 1–2 (2001), 131–140.Google ScholarCross Ref
- [38] . 2015. Reinforcement learning of self-regulated \( \beta \)-oscillations for motor restoration in chronic stroke. Frontiers in Human Neuroscience 9 (2015), 391.Google ScholarCross Ref
- [39] . 1990. Intelligent tutoring systems: An overview. Artificial Intelligence Review 4, 4 (1990), 251–277. Google ScholarCross Ref
- [40] . 2016. Adaptive vs. non-adaptive cognitive training by means of a personalized app: A randomized trial in people with multiple sclerosis. Journal of NeuroEngineering and Rehabilitation 13, 1 (Oct. 2016), 88. Google ScholarCross Ref
- [41] . 2011. Computer-based, personalized cognitive training versus classical computer games: A randomized double-blind prospective trial of cognitive stimulation. Neuroepidemiology 36, 2 (2011), 91–99. Google ScholarCross Ref
- [42] . 2005. Mild cognitive impairment as a clinical entity and treatment target. Archives of Neurology 62, 7 (2005), 1164–1166.Google ScholarCross Ref
- [43] . 2012. A computational model of use-dependent motor recovery following a stroke: Optimizing corticospinal activations via reinforcement learning can explain residual capacity and other strength recovery dynamics. Neural Networks 29–30 (2012), 60–69. Google ScholarDigital Library
- [44] . 2004. Adaptive cognitive orthotics: Combining reinforcement learning and constraint-based temporal reasoning. In Proceedings of the 21st International Conference on Machine Learning (ICML’04). ACM, New York, NY, 91. Google ScholarDigital Library
- [45] . 2016. Quantifying the User Experience: Practical Statistics for User Research (2nd ed.). Morgan Kaufmann, San Francisco, CA.Google Scholar
- [46] . 2015. Efficacy of neurocognitive remediation therapy during an acute depressive episode and following remission: Results from two randomised pilot studies. European Psychiatry 30, 1 (2015), 403.Google ScholarCross Ref
- [47] . 2016. Reinforcement learning: The sooner the better, or the later the better? In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (UMAP’16). ACM, New York, NY, 37–44. Google ScholarDigital Library
- [48] . 2010. Intelligent systems. In International Encyclopedia of Education (3rd ed.), , , and (Eds.). Elsevier, Oxford, UK, 75–80. Google ScholarCross Ref
- [49] . 2005. PASAT: Paced Auditory Serial Addition Task.
Technical Report . Department of General Psychology, University of Padova.Google Scholar - [50] . 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.Google ScholarDigital Library
- [51] . 2009. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 10 (
Dec. 2009), 1633–1685.Google ScholarDigital Library - [52] . 2006. A comprehensive review of the paced auditory serial addition test (PASAT). Archives of Clinical Neuropsychology 21, 1 (2006), 53–76. Google ScholarCross Ref
- [53] . 1995. Adaptive psychophysical procedures. Vision Research 35, 17 (1995), 2503–2522. Google ScholarCross Ref
- [54] . 2011. Training of attention functions in children with attention deficit hyperactivity disorder. ADHD Attention Deficit and Hyperactivity Disorders 3, 3 (Sept. 2011), 271–283. Google ScholarCross Ref
- [55] . 2020. Effectiveness of ReSET; a strategic executive treatment for executive dysfunctioning in patients with Parkinson’s disease. Neuropsychological Rehabilitation 30, 1 (2020), 67–84. Google ScholarCross Ref
- [56] . 2018. Exploring computerised cognitive training as a therapeutic intervention for people with Huntington’s disease (CogTrainHD): Protocol for a randomised feasibility study. Pilot and Feasibility Studies 4 (2018), 45. Google ScholarCross Ref
- [57] . 2017. Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research 19, 10 (Oct. 2017), e338. Google ScholarCross Ref
- [58] . 2020. Reinforcement learning in healthcare: A survey. arxiv:1908.08796[cs.LG].Google Scholar
- [59] . 2019. Hierarchical reinforcement learning for pedagogical policy induction. In Artificial Intelligence in Education, , , , , , and (Eds.). Springer International, Cham, Switzerland, 544–556.Google ScholarDigital Library
- [60] . 2018. Personalizing mobile fitness apps using reinforcement learning. CEUR Workshop Proceedings 2068 (March 2018). , http://ceur–ws.org/Vol–2068/humanize7.pdf https://pubmed.ncbi.nlm.nih.gov/32405286.Google Scholar
- [61] . 2014. Cognitive training in Parkinson disease: Cognition-specific vs nonspecific computer training. Neurology 82, 14 (
April 2014), 1219–1226. Google ScholarCross Ref
Index Terms
- Adaptive Cognitive Training with Reinforcement Learning
Recommendations
Inductive Semi-supervised Multi-Label Learning with Co-Training
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningIn multi-label learning, each training example is associated with multiple class labels and the task is to learn a mapping from the feature space to the power set of label space. It is generally demanding and time-consuming to obtain labels for training ...
Spectrum management of cognitive radio using multi-agent reinforcement learning
AAMAS '10: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Industry trackWireless cognitive radio (CR) is a newly emerging paradigm that attempts to opportunistically transmit in licensed frequencies, without affecting the pre-assigned users of these bands. To enable this functionality, such a radio must predict its ...
Reinforcement learning based sensing policy optimization for energy efficient cognitive radio networks
This paper introduces a machine learning based collaborative multi-band spectrum sensing policy for cognitive radios. The proposed sensing policy guides secondary users to focus the search of unused radio spectrum to those frequencies that persistently ...
Comments