Skip to main content
main-content

Über dieses Buch

This book explores robust multimodal cognitive load measurement with physiological and behavioural modalities, which involve the eye, Galvanic Skin Response, speech, language, pen input, mouse movement and multimodality fusions. Factors including stress, trust, and environmental factors such as illumination are discussed regarding their implications for cognitive load measurement. Furthermore, dynamic workload adjustment and real-time cognitive load measurement with data streaming are presented in order to make cognitive load measurement accessible by more widespread applications and users. Finally, application examples are reviewed demonstrating the feasibility of multimodal cognitive load measurement in practical applications.

This is the first book of its kind to systematically introduce various computational methods for automatic and real-time cognitive load measurement and by doing so moves the practical application of cognitive load measurement from the domain of the computer scientist and psychologist to more general end-users, ready for widespread implementation.

Robust Multimodal Cognitive Load Measurement is intended for researchers and practitioners involved with cognitive load studies and communities within the computer, cognitive, and social sciences. The book will especially benefit researchers in areas like behaviour analysis, social analytics, human-computer interaction (HCI), intelligent information processing, and decision support systems.

Inhaltsverzeichnis

Frontmatter

Preliminaries

Frontmatter

Chapter 1. Introduction

Abstract
Mental workload is an increasingly important determinant of the performance of human-machine systems. As the power, complexity and ubiquity of both everyday and specialised computing solutions increase, the limiting factor of such systems performance is shifting from the ma-chine to the human. This dynamic is further exacerbated by the need for human decision making in environments characterized by the need to process and integrate an unprecedented amount of information. Furthermore, the rapid pace of industrial, commercial, and indeed every-day life frequently requires high levels of user performance under time-limited conditions and often in sub-optimal contexts characterised by stress and competing demands for attention. As such, the need to understand and design for the particular characteristics of the user has never been stronger.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 2. The State-of-The-Art

Abstract
Chapter 2 provides an overview of the latest developments in this body of work. Firstly, different approaches to the problem of measuring cognitive load are identified. The related work of each measurement approach is then discussed in more detail. These approaches include subjective (self-report) measures, physiological measures, performance measures, and behavioural measures. The related work on measuring different types of cognitive load (intrinsic load, extraneous load, and germane load) is also reviewed. This review also discussed how cognitive load may display variation according to factors such as gender, age, and information representational differences (e.g. static graphics versus animated graphics).
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 3. Theoretical Aspects of Multimodal Cognitive Load Measures

Abstract
In Chap. 3, the theoretical aspects that result in the framework for Multimodal Cognitive Load Measures (MCLM) are discussed. This chapter brings together the critical elements of mental effort and performance studies from the domain of human factors and ergonomics. These elements are then linked to the extensive investigations of cognitive load measures in the context of CLT. Finally, multimodality and Load-Effort Homeostasis (LEH) models are presented as contributing to robustness within a real-time framework of MCLM.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Physiological Measurement

Frontmatter

Chapter 4. Eye-Based Measures

Abstract
This chapter investigates the measurement of cognitive load through eye tracking under the influence of varying luminance conditions. We demonstrate how reliable measurements can be achieved via this non-intrusive approach. We also discuss the characteristics of pupillary response and their association with different stages of cognitive processes when performing arithmetic tasks. The experimental results presented demonstrate the feasibility of a comparatively fine-grained method of cognitive load measurement in dynamic workplace environments.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 5. Galvanic Skin Response-Based Measures

Abstract
This chapter focuses on the use of Galvanic Skin Response (GSR) for cognitive load measurement. GSR is a measure of conductivity of human skin, and provides an indication of changes within the human sympathetic nervous system.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Behavioural Measurement

Frontmatter

Chapter 6. Linguistic Feature-Based Measures

Abstract
This chapter discusses methods for cognitive load examination via language and shows that many features, some of which were originally designed to examine language complexity for learning analytics or comprehension, can be successfully applied to the cognitive load research. The linguistic approach to measure cognitive load can also be used as a post-hoc analysis technique for user interface evaluation and interaction design improvement using speech transcripts.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 7. Speech Signal Based Measures

Abstract
This chapter reviews cognitive load measurement methods via speech with typical experiments to induce different cognitive load levels via speech being introduced. Various speech features are then investigated, and a comparison of cognitive load level classification methods is conducted.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 8. Pen Input Based Measures

Abstract
This chapter introduces methods to examine cognitive load via writing and pen-based features, including writing velocity, pen pressure, writing gestures and additional features derived from these signals. Based on the examination of different types of script, including text, digits and sketches, we show that cognitive load with behavioural features are affected by both text content and writing direction.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 9. Mouse Based Measures

Abstract
This chapter demonstrates the applicability of mouse interactivity based features as indicators of a user’s cognitive load. The chapter begins by introducing some basics of user mouse interaction. Then the temporal and spatial mouse features that are found to be viable indicators of cognitive load are introduced. The possibility of incorporating mouse interactivity features in multimodal cognitive load measures is also assessed in this chapter.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Multimodal Measures and Affecting Factors

Frontmatter

Chapter 10. Multimodal Measures and Data Fusion

Abstract
This chapter presents a model for multimodal cognitive load. The features extracted from speech, pen input and GSR in a user study are fused using the AdaBoost boosting algorithm to demonstrate the methods advantages.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 11. Emotion and Cognitive Load

Abstract
This chapter investigates pupillary response and GSR as a cognitive load measure under the influence of such confounding factors. A video-based eye tracker is used to record pupillary response and GSR is recorded during arithmetic tasks under both luminance and emotional changes. The mean-difference feature and its extension (Haar-like features) are used to characterise physiological responses of cognitive load under these context effects. Boosting based feature selection and classification are employed that successfully classify workload even under the influence of those noisy factors.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 12. Stress and Cognitive Load

Abstract
This chapter investigates the effect of stress on cognitive load measurement using GSR as a physiological index of CL. The experiment utilises feelings of lack of control, task failure and social-evaluation to induce stress. The experiment described demonstrates that mean GSR can be used as an index of cognitive load during task execution, but that this relationship is obfuscated when test subjects experience fluctuating levels of stress. Alternate analysis methods are then presented that show how this confounding factor can be overcome.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 13. Trust and Cognitive Load

Abstract
This chapter investigates the relationship between trust perception and cognitive load. An experimental platform is designed and employed to collect multimodal data and different types of analyses are conducted.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Making Cognitive Load Measurement Accessible

Frontmatter

Chapter 14. Dynamic Cognitive Load Adjustments in a Feedback Loop

Abstract
This chapter presents a cognitive load adaptation model that dynamically adjusts workload during human-machine interaction, in order to keep the task demands at an appropriate level. Physiological signals such as GSR are collected to evaluate human workload in real-time and the task difficulty levels are adjusted in real-time to better fit the user.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 15. Real-Time Cognitive Load Measurement: Data Streaming Approach

Abstract
In this chapter, we discuss how the efficacy of intelligent user interfaces would be greatly enhanced if a user’s cognitive load could be sensed in real time and adjustments made accordingly. Monitoring different data streams derived from the user can individually (or collectively) be processed to detect sudden shifts or gradual drifts in behavior. We present a reformulation of the problem of cognitive load change detection as the problem of concept shift/drift detection in data streams. The chapter extends the multimodal behavioral model to include mouse interactivity streams and discusses a modified sliding window implementation. In detailing an experiment utilising several variations of this model, the technical feasibility of learning from streams sheds light on the challenges presented by real time cognitive load measurement.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Chapter 16. Applications of Cognitive Load Measurement

Abstract
In this chapter, we present some typical application examples of cognitive load assessment and demonstrate the feasibility and applicability of multimodal cognitive load measurement approaches in various applications and instances of HCI.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway

Conclusions

Frontmatter

Chapter 17. Cognitive Load Measurement in Perspective

Abstract
Cognitive load is a significant factor in various application areas such as human-computer interaction (HCI), adaptive automation and training, traffic control, performance prediction, driving safety, and military command and control. Consequently, the investigation of cognitive load factors and cognitive load measurement is essential in order to improve human’s wellbeing and safety at work. As indicated in Part I, an individual human has limited cognitive resources. Both theories and models have been proposed to understand and measure cognitive load. Cognitive load theory models the interaction between limited working memory and the relatively unlimited long term memory during the learning process. It distinguishes between three types of cognitive load: intrinsic load, extraneous load, and germane load. The first type is associated with the nature of learning material, while the latter two are influenced by instructional design. Techniques used for cognitive load measurement can be divided into the following categories: subjective rating, performance measures, behavioral measures and physiological measures. The physiological approaches for cognitive load measurement are based on the assumption that any changes in human cognitive functioning are reflected in the human physiology, and have attracted increasing attention. Popular physiological measures used in cognitive load studies include brain wave, eye activity, respiration, heart rate, skin conductance, and speech, etc. Response-based behavioral features for cognitive load measurement are those that can be extracted from any user activity that is predominantly related to deliberate/voluntary task completion. This book focuses on the use of the following modalities for cognitive load measurement: eye, skin conductance, digital pen, speech, linguistic, mouse activity as well as fusions thereof.
Fang Chen, Jianlong Zhou, Yang Wang, Kun Yu, Syed Z. Arshad, Ahmad Khawaji, Dan Conway
Weitere Informationen