Robust Multimodal Cognitive Load Measurement

Mental workload is an increasingly important determinant of the performance of human-machine systems. As the power, complexity and ubiquity of both everyday and specialised computing solutions increase, the limiting factor of such systems performance is shifting from the ma-chine to the human. This dynamic is further exacerbated by the need for human decision making in environments characterized by the need to process and integrate an unprecedented amount of information. Furthermore, the rapid pace of industrial, commercial, and indeed every-day life frequently requires high levels of user performance under time-limited conditions and often in sub-optimal contexts characterised by stress and competing demands for attention. As such, the need to understand and design for the particular characteristics of the user has never been stronger.

Chapter 2 provides an overview of the latest developments in this body of work. Firstly, different approaches to the problem of measuring cognitive load are identified. The related work of each measurement approach is then discussed in more detail. These approaches include subjective (self-report) measures, physiological measures, performance measures, and behavioural measures. The related work on measuring different types of cognitive load (intrinsic load, extraneous load, and germane load) is also reviewed. This review also discussed how cognitive load may display variation according to factors such as gender, age, and information representational differences (e.g. static graphics versus animated graphics).

In Chap. 3, the theoretical aspects that result in the framework for Multimodal Cognitive Load Measures (MCLM) are discussed. This chapter brings together the critical elements of mental effort and performance studies from the domain of human factors and ergonomics. These elements are then linked to the extensive investigations of cognitive load measures in the context of CLT. Finally, multimodality and Load-Effort Homeostasis (LEH) models are presented as contributing to robustness within a real-time framework of MCLM.

This chapter investigates the measurement of cognitive load through eye tracking under the influence of varying luminance conditions. We demonstrate how reliable measurements can be achieved via this non-intrusive approach. We also discuss the characteristics of pupillary response and their association with different stages of cognitive processes when performing arithmetic tasks. The experimental results presented demonstrate the feasibility of a comparatively fine-grained method of cognitive load measurement in dynamic workplace environments.

This chapter focuses on the use of Galvanic Skin Response (GSR) for cognitive load measurement. GSR is a measure of conductivity of human skin, and provides an indication of changes within the human sympathetic nervous system.

This chapter discusses methods for cognitive load examination via language and shows that many features, some of which were originally designed to examine language complexity for learning analytics or comprehension, can be successfully applied to the cognitive load research. The linguistic approach to measure cognitive load can also be used as a post-hoc analysis technique for user interface evaluation and interaction design improvement using speech transcripts.

This chapter reviews cognitive load measurement methods via speech with typical experiments to induce different cognitive load levels via speech being introduced. Various speech features are then investigated, and a comparison of cognitive load level classification methods is conducted.

This chapter introduces methods to examine cognitive load via writing and pen-based features, including writing velocity, pen pressure, writing gestures and additional features derived from these signals. Based on the examination of different types of script, including text, digits and sketches, we show that cognitive load with behavioural features are affected by both text content and writing direction.

This chapter demonstrates the applicability of mouse interactivity based features as indicators of a user’s cognitive load. The chapter begins by introducing some basics of user mouse interaction. Then the temporal and spatial mouse features that are found to be viable indicators of cognitive load are introduced. The possibility of incorporating mouse interactivity features in multimodal cognitive load measures is also assessed in this chapter.

This chapter presents a model for multimodal cognitive load. The features extracted from speech, pen input and GSR in a user study are fused using the AdaBoost boosting algorithm to demonstrate the methods advantages.

This chapter investigates pupillary response and GSR as a cognitive load measure under the influence of such confounding factors. A video-based eye tracker is used to record pupillary response and GSR is recorded during arithmetic tasks under both luminance and emotional changes. The mean-difference feature and its extension (Haar-like features) are used to characterise physiological responses of cognitive load under these context effects. Boosting based feature selection and classification are employed that successfully classify workload even under the influence of those noisy factors.

This chapter investigates the effect of stress on cognitive load measurement using GSR as a physiological index of CL. The experiment utilises feelings of lack of control, task failure and social-evaluation to induce stress. The experiment described demonstrates that mean GSR can be used as an index of cognitive load during task execution, but that this relationship is obfuscated when test subjects experience fluctuating levels of stress. Alternate analysis methods are then presented that show how this confounding factor can be overcome.

This chapter investigates the relationship between trust perception and cognitive load. An experimental platform is designed and employed to collect multimodal data and different types of analyses are conducted.

This chapter presents a cognitive load adaptation model that dynamically adjusts workload during human-machine interaction, in order to keep the task demands at an appropriate level. Physiological signals such as GSR are collected to evaluate human workload in real-time and the task difficulty levels are adjusted in real-time to better fit the user.

In this chapter, we discuss how the efficacy of intelligent user interfaces would be greatly enhanced if a user’s cognitive load could be sensed in real time and adjustments made accordingly. Monitoring different data streams derived from the user can individually (or collectively) be processed to detect sudden shifts or gradual drifts in behavior. We present a reformulation of the problem of cognitive load change detection as the problem of concept shift/drift detection in data streams. The chapter extends the multimodal behavioral model to include mouse interactivity streams and discusses a modified sliding window implementation. In detailing an experiment utilising several variations of this model, the technical feasibility of learning from streams sheds light on the challenges presented by real time cognitive load measurement.

In this chapter, we present some typical application examples of cognitive load assessment and demonstrate the feasibility and applicability of multimodal cognitive load measurement approaches in various applications and instances of HCI.

Cognitive load is a significant factor in various application areas such as human-computer interaction (HCI), adaptive automation and training, traffic control, performance prediction, driving safety, and military command and control. Consequently, the investigation of cognitive load factors and cognitive load measurement is essential in order to improve human’s wellbeing and safety at work. As indicated in Part I, an individual human has limited cognitive resources. Both theories and models have been proposed to understand and measure cognitive load. Cognitive load theory models the interaction between limited working memory and the relatively unlimited long term memory during the learning process. It distinguishes between three types of cognitive load: intrinsic load, extraneous load, and germane load. The first type is associated with the nature of learning material, while the latter two are influenced by instructional design. Techniques used for cognitive load measurement can be divided into the following categories: subjective rating, performance measures, behavioral measures and physiological measures. The physiological approaches for cognitive load measurement are based on the assumption that any changes in human cognitive functioning are reflected in the human physiology, and have attracted increasing attention. Popular physiological measures used in cognitive load studies include brain wave, eye activity, respiration, heart rate, skin conductance, and speech, etc. Response-based behavioral features for cognitive load measurement are those that can be extracted from any user activity that is predominantly related to deliberate/voluntary task completion. This book focuses on the use of the following modalities for cognitive load measurement: eye, skin conductance, digital pen, speech, linguistic, mouse activity as well as fusions thereof.