Application of Recurrent Convolutional Neural Networks for Mental Workload Assessment Using Functional Near-Infrared Spectroscopy

Mental workload assessment is a core element of designing complex high-precision human machine interfaces with industrial and medical applications, from aviation to robotic neurosurgery. High accuracy and continuous mental state decoding play an essential role for keeping the operator’s mental workload on a moderate level to prevent cognitive tunneling and improve the safety and performance of complex machine use. Monitoring brain activity using wearable and increasingly portable functional near-infrared spectroscopy (fNIRS) sensors enable measurement in realistic and real-world conditions. While a variety of machine learning techniques have been evaluated for this application, Recurrent Convolutional Neural Networks (R-CNN) have received only minimal attention. A significant advantage of R-CNN compared to other classification methods is that it can capture temporal and spatial patterns of brain activity simultaneously without requiring prior feature selection or computationally demanding preprocessing or denoising. This study represents an investigation on designing a hybrid deep learning architecture combining CNN and RNN (Long Short Term Memory-LSTM). The proposed architecture is evaluated for mental workload memory (n-Back) tasks from an open-source dataset. Proposed architecture demonstrates higher performance than both Fully Connected Deep Neural Network and Support Vector Machine methods, showing a high capacity for simultaneous spatio-temporal pattern recognition. We obtained improvements of 15% and 11% in average subject accuracy with deoxy-hemoglobin (deoxy-Hb) in R-CNN compared to SVM and DNN methods, respectively.