HDL-PSR: Modelling Spatio-Temporal Features Using Hybrid Deep Learning Approach for Post-Stroke Rehabilitation

Physiotherapy exercises like extension, flexion, and rotation are an absolute necessity for patients of post stroke rehabilitation (PSR). A physiotherapist uses many techniques to restore movements needs in daily life including nerve re-education, task training, muscle strengthening and uses various assistive techniques. But, a physiotherapist guiding the physiotherapy exercises to a patient is a time-consuming, tedious and costly affair. In the paper, a novel automated system is designed for detecting and recognizing upper limb exercises using an RGB-Depth camera that could guide the patients to perform real-time physiotherapy exercises without human intervention. Hybrid deep learning (HDL) approaches are exploited for the highly accurate and robust system for recognizing physiotherapy exercises of the upper limb for PSR. As a baseline, a deep convolutional neural network (CNN) is designed that automatically extracts features from the pre-processed data and classifies the performed physiotherapy exercise. As the exercise is being performed, to extract and utilize temporal dependencies, architectures of recurrent neural network (RNN) are used. In the CNN-LSTM model, CNN derives useful features that are provided to LSTM thus increasing the accuracy of recognized exercises. To train faster, another hybrid deep learning model, CNN-GRU is implemented where a novel focal loss criterion is used to overcome the drawbacks of standard cross-entropy loss. Experimental evaluation is done using RGB-D data obtained from Microsoft Kinect v2 sensors. Dataset comprising of 10 different physiotherapy exercises were created. Experimental results have shown significant activity recognition accuracy with 98% and 99% for CNN and CNN-LSTM model respectively. CNN-GRU model is the best suitable architecture with 100% accuracy.