Skip to main content

2021 | Book

Advances in Neuroergonomics and Cognitive Engineering

Proceedings of the AHFE 2021 Virtual Conferences on Neuroergonomics and Cognitive Engineering, Industrial Cognitive Ergonomics and Engineering Psychology, and Cognitive Computing and Internet of Things, July 25-29, 2021, USA


About this book

This book offers a broad overview of the field of cognitive engineering and neuroergonomics, covering emerging practices and future trends toward the harmonious integration of human operators and computational systems. It gathers both theoretical and practice-oriented studies on mental workload and stress, activity theory, human reliability, error and risk. It covers applications in various field, and corresponding strategies to make assistive technologies more user-oriented. Further, the book describes key advances in our understanding of cognitive processes, including mechanisms of perception, memory, reasoning, and motor response, with a particular focus on their role in interactions between humans and other elements of computer-based systems. Gathering the proceedings of the AHFE 2021 Conferences on Neuroergonomics and Cognitive Engineering, Industrial Cognitive Ergonomics and Engineering Psychology, and Cognitive Computing and Internet of Things, held virtually on July 25-29, 2021, from USA, this book offers extensive information and a thought-provoking guide for researchers and practitioners in cognitive engineering, neuroergonomics and their applications.

Table of Contents


Modeling and Monitoring Humans for Operational Task Performance

Planned Investigations to Address Acute Central Nervous System Effects of Space Radiation Exposure with Human Performance Data

This work intends to generate evidence of acute, incremental human performance decrement similar to that due to space radiation and its impacts on the brain, to accompany ongoing human performance modeling work. The planned work will explore the boundaries of human behavioral and performance decrement after exposure to stress, which may be expected based in part on rodent responses found after exposure to ionizing radiation. The collection of evidence via simulation studies can characterize real human errors toward determining what stress levels lead to significantly-low levels of performance (below permissible outcome limits) which would imperil mission accomplishment. If mission-relevant animal-study-linked tasks are used, human and animal performance levels may be aligned to enable quantitative assignment of permissible exposure limits based on animal exposure studies. Ultimately, a transfer function between the performances of exposed rodents and humans under stress can be developed using shared impairment mechanisms.

Angela Harrivel, Steve Blattnig, Ryan Norman, Lisa Simonsen
Investigating the Modulation of Spatio-Temporal and Oscillatory Power Dynamics by Perceptible and Non-perceptible Rhythmic Light Stimulation

Several studies emphasize the great potential of rhythmic light stimulation to evoke steady-state visual evoked potentials (SSVEPs) measured via electroencephalographic (EEG) recordings as a safe method to modulate brain activity. In the current study, we investigated visual event-related potentials (ERPs) and oscillatory power evoked by perceptible (above a previously estimated individual threshold) and non-perceptible (below the individual threshold) frequency-modulated rhythmic light stimulation at 10 Hz via a light-emitting diode. Furthermore, we examined the effect of overt and covert attention by asking participants to (1) directly focus on the light source (overt attention condition) and (2) indirectly attend it (covert attention condition). Our results revealed entrainment effects reflected in both ERPs and oscillatory power in the EEG even for a stimulation intensity below the individual perceptibility threshold and without directly fixating the light source. This non-invasive stimulation method shows strong potential for naturalistic non-clinical applications.

Katharina Lingelbach, Isabel Schöllhorn, Alexander M. Dreyer, Frederik Diederichs, Michael Bui, Michael Weng, Jochem W. Rieger, Ina Petermann-Stock, Mathias Vukelić
Towards Artificial Social Intelligence: Inherent Features, Individual Differences, Mental Models, and Theory of Mind

We discuss the potential for near-future artificial social intelligences (ASIs) to exhibit functional artificial theory of mind in a video-game based teaming scenario. We also describe the impact of individual differences on mental model formation and mental state development. We focus on the possibility for an ASI to develop profiles of human agents by eliciting or observing a range of information which may influence their counterpart’s behavior during a novel task. We conclude with a discussion of the implications of this approach for integrating methods from the cognitive and social sciences in development of ASI.

Rhyse Bendell, Jessica Williams, Stephen M. Fiore, Florian Jentsch

Real-World Human State Assessment: Victories and Remaining Challenges

Can Situation Awareness Be Measured Physiologically?

To operate effectively across a variety of environments, personnel (e.g., air traffic controllers, pilots, truck drivers, emergency response crews) must develop and maintain situation awareness (SA), perceiving relevant elements in the environment, understanding their meaning, and projecting their status into the near future [1]. Although multiple SA assessment techniques have been developed, they require periodic interruptions of a task to query the individual regarding their knowledge of the situation. There has been a recent proliferation of more rugged and durable sensor devices (e.g., functional near infrared spectroscopy (fNIRS) sensors) that can be used while people take part in ecologically valid activities to assess changes in neurophysiology, physiology, and behavior that correlate with cognitive state (e.g., SA). In addition, recent advances in machine learning and modeling techniques can be used to interpret information about human states (e.g., SA) from noisy data acquired in such environments that previously was unusable. These advances provide opportunity to develop physiological measurement approaches that could provide a potential avenue for real time, continuous, and objective assessment of SA in many real-world settings. This class of measures could potentially provide a window into low SA states where an intervention may be necessary to ensure acceptable levels of performance. In this paper, we review potential SA metrics to assess their suitability for continuous real-world monitoring.

Bethany Bracken, Sean Tobyne, Aaron Winder, Nina Shamsi, Mica R. Endsley
Towards a Measure of Situation Awareness for Space Mission Schedulers

The success of space flight missions relies on support provided to astronauts through specialist knowledge of ground-based personnel. One of the many essential tasks that ground personnel provide is the scheduling of flight crew daily activities. Future long duration exploration missions will require astronauts to assume planning and scheduling responsibilities in order to facilitate increased autonomy from ground support. Although situation awareness is critical to the scheduling task, a sufficient measure for this domain has not been developed. This paper documents the approach and process by which the authors developed a framework for measuring situation awareness in space mission schedulers and presents the measure applications’ initial results.

Tamsyn Edwards, Summer L. Brandt, Jessica J. Marquez
Monitoring Human Performance on Future Deep Space Missions

NASA and the commercial spacecraft community are working diligently to put the first woman on the moon in the 2024 timeframe. At the same time, NASA researchers are thinking about how to solve the even larger challenges that future deep space missions will bring – primary among them is crew autonomy. Deep space mission crews will face communication delays and blackouts, and in those situations, the crew may not have access to Mission Control Center (MCC) experts. They will be dependent on each other and the available information onboard to stay alive, healthy, and achieve the mission. The only conceivable way to meet the challenges of Earth independence is to enable the crew to monitor their own health and performance. A number of technologies are presently being explored by NASA to enable crew self-monitoring. This paper highlights select projects, and provides broader discussion about the need for advanced monitoring technologies.

Kritina Holden, Brandin Munson, Jerri Stephenson
What Did Our Model Just Learn? Hard Lessons in Applying Deep Learning to Human Factors Data

Deep learning is revolutionizing all areas of data science, including human factors research. Much of human factors data, however, have fundamental idiosyncrasies that make applying deep learning challenging. Further, the complexity of deep learning can make finding errors challenging and deducing what was actually learned by the model near impossible. This paper provides two case-studies in which our research group faced and overcame such challenges. It examines the root causes of each issue and discusses how they may lead to common challenges. We describe how we discovered problems and describe how we rectified them. It is our hope, that by sharing our experiences with likely common challenges, we can help other researchers in avoiding similar pitfalls.

Brian Weigel, Kaleb Loar, Andrés Colón, Robert Wright
Addressing Two Central Issues of Team Interaction Dynamics: The Whole is Greater Than the Sum of Its Parts

In successful teams, each team member has a distinct taskwork based on individual task roles and responsibilities. Each team member effectively interacts with one another and the technology with which they must interact during the task. Both taskwork and teamwork have situational propensities and entail both ontic and epistemic aspects; thus, understanding how they affect teammates’ taskwork and teamwork becomes crucial to fathom the emergence of team coordination dynamics. This paper conceptually discusses the application of quantum cognition to team coordination; how this approach can improve the understanding of the notion of the whole. The open system model, which incorporates both classical and quantum probability descriptions of a system, is reviewed to describe both ontic and epistemic uncertainty. The open system model's contributions to the entropy of an entangled whole are discussed. Lastly, the conceptual differences between sensing and interaction and the experimental scenarios to study these differences are delineated.

Mustafa Canan, Mustafa Demir
Amyotrophic Lateral Sclerosis Disease Progression Presents Difficulties in Brain Computer Interface Use

Brain Computer Interface (BCI) systems potentially provide those with disabilities an alternative means to communicate and control their environment, increasing independence and quality of life. The goal of our study was to examine the effect of disease progression on the efficiency of communication using a P300 BCI speller. To address this, BCI data was analyzed from 19 people living with ALS at various stages of disease. Accuracy in word spelling showed significant correlation to ALSFRS-R score, a measure of disease severity and worsening function, with decreased accuracy as the disease progressed. This decrease in accuracy may be attributed to system components and methodology that lead to an increase in fatigue for people with less function, but alternatively may be due to the progressive changes in neural networks as the patient progresses. Adjustments to BCI systems, use of alternative event potentials, or alternative technologies may be necessary to optimize BCI use in ALS.

Emma Dryden, Mohammad Sahal, Sara Feldman, Hasan Ayaz, Terry Heiman-Patterson
Epileptic Seizure Detection Using Tunable Q-Factor Wavelet Transform and Machine Learning

Epileptic seizures constitute an important group of neurological disorders in brain that affect many people globally each year. Complexity of EEG signals due to their high-dimensional nature, as well as artifacts in data due to equipment flaws, pose significant challenges to physicians in diagnosing epileptic seizures directly and manually from EEG signals. In this paper, a method is proposed to combine signal processing and machine learning for diagnosing epileptic seizures and tested on the Bonn University database. We used Tunable Q-Factor wavelet transform (TQWT) method to transform signals. Subsequently, various statistical properties, frequency, chaotic and fractional were extracted from the TQWT sub-bands. Subset selection techniques were used in order to reduce the features of the methods used and their results were compared. Finally, a SVM method with different kernels were tested, where the empirical results show the high efficiency of the proposed method in the diagnosis of epileptic seizures.

Ala Tokhmpash, Sarah Hadipour, Bahram Shafai

Cognitive State Assessment

Augmented Reality Integrated Brain Computer Interface for Smart Home Control

Brain computer interface (BCI) technology is an alternate communication option for individuals with neuromuscular impairments. There are several challenges to optimal BCI use including positioning of the screen, ease of use, independence in access, and calibration. Our study is directed at the development of a practical, accessible, at-home use BCI system that addresses these obstacles. Our design utilizes an augmented reality (AR) head-mounted display as a solution to provide BCI stimuli and output. We used a battery-operated Bluetooth-connected 8-channel portable EEG system and a custom P300 selection matrix displaying icons corresponding to various home control actions. Finally, the BCI system is integrated with a built-in smart assistant (Google Assistant) which allows the user to control their environment. In this paper, we describe the engineering of this home-use BCI system designed specifically for people with Amyotrophic Lateral Sclerosis, a neuromuscular disease causing severe motor deficits and loss of mobility.

Mohammad Sahal, Emma Dryden, Mali Halac, Sara Feldman, Terry Heiman-Patterson, Hasan Ayaz
Expectations in Human-Robot Interaction

It is acknowledged that humans expect social robots to interact in a similar way as in human-human interaction. To create successful interactions between humans and social robots, it is envisioned that the social robot should be viewed as an interaction partner rather than an inanimate thing. This implies that the robot should act autonomously, being able to ‘perceive’ and ‘anticipate’ the human’s actions as well as its own actions ‘here and now’. Two crucial aspects that affect the quality of social human-robot interaction is the social robot’s physical embodiment and its performed behaviors. In any interaction, before, during or after, there are certain expectations of what the social robot is capable of. The role of expectations is a key research topic in the field of Human-Robot Interaction (HRI); if a social robot does not meet the expectations during interaction, the human (user) may shift from viewing the robot as an interaction partner to an inanimate thing. The aim of this work is to unravel the role and relevance of humans’ expectations of social robots and why it is important area of study in HRI research. Moreover, I argue that the field of HRI can greatly benefit from incorporating approaches and methods from the field of User Experience (UX) in its efforts to gain a deeper understanding of human users’ expectations of social robots and making sure that the matching of these expectations and reality is better aligned.

Julia Rosén
Application of Recurrent Convolutional Neural Networks for Mental Workload Assessment Using Functional Near-Infrared Spectroscopy

Mental workload assessment is a core element of designing complex high-precision human machine interfaces with industrial and medical applications, from aviation to robotic neurosurgery. High accuracy and continuous mental state decoding play an essential role for keeping the operator’s mental workload on a moderate level to prevent cognitive tunneling and improve the safety and performance of complex machine use. Monitoring brain activity using wearable and increasingly portable functional near-infrared spectroscopy (fNIRS) sensors enable measurement in realistic and real-world conditions. While a variety of machine learning techniques have been evaluated for this application, Recurrent Convolutional Neural Networks (R-CNN) have received only minimal attention. A significant advantage of R-CNN compared to other classification methods is that it can capture temporal and spatial patterns of brain activity simultaneously without requiring prior feature selection or computationally demanding preprocessing or denoising. This study represents an investigation on designing a hybrid deep learning architecture combining CNN and RNN (Long Short Term Memory-LSTM). The proposed architecture is evaluated for mental workload memory (n-Back) tasks from an open-source dataset. Proposed architecture demonstrates higher performance than both Fully Connected Deep Neural Network and Support Vector Machine methods, showing a high capacity for simultaneous spatio-temporal pattern recognition. We obtained improvements of 15% and 11% in average subject accuracy with deoxy-hemoglobin (deoxy-Hb) in R-CNN compared to SVM and DNN methods, respectively.

Marjan Saadati, Jill Nelson, Adrian Curtin, Lei Wang, Hasan Ayaz
Influence of Properties of the Nervous System on Cognitive Abilities

The data analysis has demonstrated that the general intellect (IQ) has the high relationship with strength and functional mobility of nervous processes, if those indices have been strengthened indices proposed by authors (namely, accounting noise immunity indicators, as well as the verbal and nonverbal intellect) in the 1st group of subjects. Thinking ability had the higher relationship (R = 0.80, p < 0.01). At the same time, students-researchers (military psychophysiologists should be researchers) demonstrated much more close dependence in those indices: the general intellect had extremely high relationship with extended FMNP indices (R = 0.82, p < 0.01), and thinking ability even higher (R = 0.95, p < 0.001). The possible reasons and ways of application are discussed.

Oleksandr Burov, Svitlana Lytvynova, Olga Pinchuk, Evgeniy Lavrov, Olga Siryk, Victoriya Logvinenko, Olena Hlazunova, Valentyna Korolchuk, Alexander Zolkin
Understanding Junior Design Students’ Emotion During the Creative Process

The discussion of the relationships between emotional concerns and design had to be conducted for over twenty years. However, most of the discussions still focus on how emotion would influence the feature of design outcomes or how to understand the users’ emotional concerns. Limited investigations are conducted for optimising the design process. Especially to the junior designers and junior design students, a niche need for helping them to manipulate their emotional concerns is discovered. This study aims to seek some methods to lead junior design students understanding their design processes which would be influenced by their emotional changes. It is expected that the proposed methods would help students to identify their emotional concerns and forest them to make decisions more effectively.

Amic G. Ho
Art Image Complexity Measurement Based on Visual Cognition: Evidence from Eye-Tracking Metrics

In order to obtain the physiological and psychological indicators of the visual complexity of art images from the perspective of visual cognition, this study explored the relationship between eye-tracking metrics and the psychological factors. The study invited 16 participants (8 females, age range 23.81 ± 0.98) to participate in the experiment. In this study, eye-tracking experiments and a questionnaire of psychological factors affecting visual complexity were conducted. The results show that there is a significant relationship between the fixation length, first fixation time and visual complexity. Image with the complexity score interval [74, 100] has a high mental workload on visual processing. There is a significant linear relationship between the fixation count and visual complexity. In addition, the analysis of the psychological scale shows that psychological factors have a positive significant correlation with visual complexity. The participants show sensitivity to the factor of color, texture, and cognitive on visual complexity, but were insensitive to shape factors.

Rui Hu, Minghan Weng, Liqun Zhang, Xiaodong Li

Neurobusiness Applications

Attentional and Emotional Engagement of Sustainability in Tourism Marketing: Electroencephalographic (EEG) and Peripheral Neuroscientific Approach

In recent years, sustainability has received significant interest from companies and structures that have adopted new marketing and business models to improve the living conditions of the environment and consumers. Given the growing interest and attention, the issue of sustainability has been investigated by different disciplines, such as neuroscience, which have proved useful for investigating the emotional responses and cognitive processes of individuals, allowing us to understand the relationship between consumers and sustainability better. In this regard, this paper offers an overview on the topic of sustainability in a specific sector, such as tourism marketing, on the use of neuroscience to investigate sustainability and on the application of a multimethodological paradigm, involving the use of electroencephalography (EEG) and biofeedback, to measure individuals’ electroencephalographic and autonomic activity during the exploration of a green hotel.

Michela Balconi, Federico Cassioli, Giulia Fronda
The Face of Bad Advertising: Assessing the Effects of Human Face Images in Advertisement Design Using Eye-Tracking

Visual messaging has been widely researched in psychology and communications specifically within the area of advertising [2, 3]. Such messaging research seeks to identify the characteristics and variables within an advertisement that contribute most to its effectiveness [4]. The present study aimed to investigate the relationship between the presence of a human face image in print advertising and viewer affinity for ad content. Thirty-three participants were instructed to preview 42 different personal injury law firm advertisements. Mean eye gaze fixation durations were recorded within both text and image areas of interest (AOIs). Additionally, self-reported advertisement ratings were used to stratify ads into low and high-affinity categories. Findings from the study indicate that there was less time spent engaging with text content in poorly rated advertisements when an image of a human face was present. Interestingly, this effect was not present in advertisements with favorable ratings, where longer fixation durations were dedicated to text AOIs as opposed to image AOIs regardless of the presence of a face image. These results suggest that negatively perceived human faces may impact the perception of an advertisement’s message and demonstrates that combined eye-tracking and self-reported measures can provide a comprehensive neuroergonomic assessment of advertisement preference and engagement in real-world environments.

Jan Watson, Hongjun Ye, Jintao Zhang, Yigit Topoglu, Rajneesh Suri, Hasan Ayaz
Interpersonal Synchrony Protocol for Cooperative Team Dynamics During Competitive E-Gaming

The performance of a team is tightly connected to how well its members communicate and collaborate while working towards shared objectives, a process known as group cognition. In competitive team sports, strategic and efficient coordination between team members often makes the difference between success and failure. Professional e-sport gaming, in particular, requires competitors to engage in quick decision making, strategic thinking, and fast reaction times in order to outmaneuver the opposing team. Interpersonal neural synchrony (INS) is one proposed mechanism by which team members may achieve mutual understanding necessary for successful teaming. However, it is unknown how such measures relate to team performance and how they are affected by gaming environments (in-person vs. remote gameplay). Here, we describe a study protocol aiming to investigate the relationship between inter-individual neurophysiological measures and successful teamwork of two-player teams in Overwatch, a competitive team-based first-person shooter (FPS). The objective of this study design is to relate behavioral and subjective measures of team performance with underlying neural and physiological activity during both in-person and remote sessions of cooperative gameplay.

Adrian Curtin, Jan Watson, Yigit Topoglu, Nicholas DeFilippis, Hongjun Ye, Rajneesh Suri, Hasan Ayaz
Assessing the Impact of Ad Characteristics on Consumer Behavior and Electrodermal Activity

The use of mobile, low-cost, and wearable sensors to measure physiological activity, continuously and non-intrusively during everyday tasks has the potential to improve our understanding of human perception, cognition, and behavior at large. One applied field with the practical use of such expansive assessment is consumer neuroergonomics. In this study, we demonstrated the use of ultramobile battery operated and wireless electrodermal activity (EDA) sensors to estimate emotional arousal during the viewing of various out-of-house, printed, and e-mail advertisements of law-firm services. A total of 33 participants’ EDA data and self-reported questionnaires were analyzed to assess the impact of different ad types on consumers in terms of their age, gender and prior experience with firms (client vs. non-client). EDA results confirmed as expected that e-mail ads were the least engaging to all participants. Furthermore, participants were engaged significantly less on ads that they are already familiar with, that is existing customers. Results indicate that the use of accessible and low-cost sensors like EDA could be widely deployed to enhance our understanding of customer behavior in order to optimize ad development and delivery.

Yigit Topoglu, Jan Watson, Jintao Zhang, Hongjun Ye, Rajneesh Suri, Hasan Ayaz

Systemic-Structural Activity Theory

Systemic-Structural Activity Theory and Artificial Intelligence

This paper is dedicated to the analysis of the issues the Artificial Intelligence (AI) developers are facing when they design the software. We suggest to use the numerous Systemic-Structural Activity Theory (SSAT) methods of analyzing and optimizing human performance in order to streamline the AI design process. The paper offers examples of the AI applications and describes some of the methods and application of the SSAT framework. AI replaces routine human tasks with the software and SSAT methodology allows to analyze human performance and build the human algorithm that can be replaced by AI. Utilization of SSAT will make software development much more efficient.

Inna S. Bedny, Waldemar Karwowski
Applying Web-based Application ExpressDecision2 in Patient-Centered Care

ExpressDecision2® (ED2®) is a web app designed to support the individual in quickly making difficult decisions under uncertainty, which are emotionally driven and typically solved by using rational intuition. ED2 is based on the self-regulation model of the thinking process developed within the framework of the systemic-structural activity theory. ED2 supports both decision-making and problem-solving processes. For problem-solving, ED2 supports goal-setting by implementing the principle of instrumental rationality. Paper demonstrates of ED2’s application in making a patient-centered and shared-with-doctor decision about the best treatment option for cholesterol reduction.

Alexander M. Yemelyanov, Rahul Sukumaran, Alina A. Yemelyanov
Neurocognitive Indicators of Insight According to P300 and Later Visual ERP Components

Event-related potentials (ERP) traditionally on 42 healthy subjects (men) aged 20–28 years were recorded. The visual images were a line of visual images with an incomplete set of signs, as well as images-illusions, which, with different perceptions, represent different images. The article discusses the results of original research in the context of the discussion of modern studies of the well-known psychological phenomenon of P300 evoked potentials. Thus, the main trends in the change in ERP depending on the variants of recognition of visual illusions and oddball images are as follows. Complete recognition of illusions, which corresponded to insight in the model of our experiment, was accompanied by an increase in the amplitude of the P300 wave in the temporal-parietal regions on the right. With correct identification of one of the dual images, the activation of the N450 component was recorded in the frontal regions on both sides. Negative recognition was characterized by generalized symmetric activation of all ERP components, more pronounced in the posterior half of the brain.

Sergey Lytaev
Self-regulation Approach for Setting Goals in Problem-Solving

In the self-regulation model of decision-making under uncertainty, the dynamic programming principle of optimality transforms into the principle of instrumental rationality, according to which the proximate goal should be reached from the perspective of attaining the distal goal. The solution of the problem is considered to be a result of multiple iterations in evaluating an alternative’s pros and cons from the perspective of the distal (long-term) goal. This paper demonstrates how the self-regulation approach for setting and resetting goals when making difficult decisions helps differentiate the alternatives and find the most suitable course of action by recognizing cons in proximal positive outcomes and pros in proximal negative outcomes, or by changing the new distal goal, which should be more long-term than the previous one.

Alexander M. Yemelyanov, Inna S. Bedny
Worker Engagement in Routinized Structured Activity Circumvention: Using SSAT to Understand the Significance of Involuntary Cognitive Intentionality

This study explored workers engagement in involuntary cognitive intentionality leading to their circumvention of structured and routinized activities at the workplace. Guided by Bedny and Karwowski’s postulation that activities of individuals are realized by goal-directed actions, informed either by mental or motor conscious processes, as objects of the cognitive psychology of skills and performances, qualitative data was collected from documented interactions between graduate students engaged in research work and their supervisor, and analyzed morphologically to understand the significance of workers involuntary cognitive intentionality in different work setting. It was found that individuals assigned consciously designed and assigned structured activity in work-settings can think that they know how to do such activities better, and even understand everything about how to do the activity properly in their minds. It is concluded that workers involuntary cognitive intentionality makes them circumvent consciously designed and assigned routinized structured activity, yielding outcomes that deviate from expectations.

Mohammed-Aminu Sanda

Brain-Machine Interface (BMI) and Neuroinformatics

Design for AI-Enhanced Operator Information Ergonomics in a Time-Critical Environment

Maintaining situational awareness in time-critical operation control is an omni-dimensional optimisation problem. For excellent situational awareness, complete information with sufficient time to process it is prerequisite. Making sound judgement with limited time the flight controllers suffer poor information ergonomics as demanding situations cause cognitive load as well as incoming information is constipated. In this normative paper, design principles and main functionalities are presented for an artificial intelligence powered and extended reality decision support information system.

Jussi Okkonen, Jaakko Hakulinen, Matti Jalava, Heikki Mansikka, Tuuli Keskinen, Markku Turunen
Feature Comparison of Emotion Estimation by EEG and Heart Rate Variability Indices and Accuracy Evaluation by Machine Learning

There has been a lot of attempts on estimating human emotions using physio-logical data, and it is expected to be applied to medical diagnosis. Recently, there is emotion estimation model using EEG and heart rate variability index-es as feature values, and applying deep learning to classify emotions with an accuracy of 61%. However, the accuracy may not be sufficient for applications such as medical diagnosis. In this study, we extracted and selected features of EEG and heart rate variability indexes in order to improve the accuracy. As a result, by using our proposed method to extract and select features, the accuracy of the model was increased to almost 100%.

Kei Suzuki, Ryota Matsubara, Tipporn Laohakangvalvit, Midori Sugaya
An Analysis of the Cognitive Process and Similarities of Complex Problem Solving Discussions

The concept of complex problem solving (CPS) as a new cognitive scientific area was initially introduced by Doner 30 years ago. It was not until recently that the verdicts of CPS studies were gradually being applied in business training . This research aims to convert lab findings into light-weighted cognitive-oriented guidelines for CPS. To achieve this purpose, we investigated the pattern of the underlying cognitive process in CPS discussions. This paper proposes a method that compares cognitive similarities based on the Kolmogorov–Smirnov test. Instead of comparing the semantic similarity, this method compares the proportion of conversation content. It serves the purpose of examining the similarities of thinking processes. From the correlation between the subjective problem conflict level and cognitive similarities, we found that a discussion process best practice can help build a more comprehensive problem understanding.

Yingting Chen, Taro Kanno, Kazuo Furuta
Control Room Operators’ Cognitive Strategies in Complex Troubleshooting

The aim of the present study was to investigate control room operators’ troubleshooting performance and adaptive readiness in nuclear domain with a new version of a virtual reality control room (VR CR) system. The VR CR was a high fidelity copy of a physical CR. The test session consisted of five simulated incidents. Two operator crews were recruited. In addition, one separate crew participated in the pilot test. This paper focuses on analysis of individual process-tracing and debriefing interviews, performed after each simulator run. Our results showed that all the failures were successfully found and perceived with one exception. The operators were also able to formulate hypotheses about the cause of the failure and make plans on how to test a hypothesis and evaluate the evidence. On the other hand, the crews experienced some problems in problem monitoring and process control due to the problems caused by the VR system.

Jari Laarni, Marja Liinasuo, Satu Pakarinen, Kristian Lukander, Tomi Passi
Cognitive Interventions Based on Technology: A Systematic Literature Review

This article reports a systematic review of research done about the positive impact of cognitive treatment for people with some type of brain disorder through the use of inclusive technologies. The article collected 21 publications of high impact magazines uncovering that countries such as Spain, France, and Russia have a significant number of contributions on this topic. Inclusive technological innovations which are intended for work on brain functions such as attention, memory, verbal fluency, problem-solving and behavior regulation have been mainly developed in devices such as Tablets and Robots. The data discussed highlights the need to continue in this line of research to determine the effect of this type of intervention, as well as the future projection of developing new technological devices facilitating human brain functions.

Carlos Ramos-Galarza, Omar Cóndor-Herrera, Hugo Arias-Flores, Janio Jadán-Guerrero, Mónica Bolaños-Pasquel, Priscila Cedillo
Exploring Relationships Between Distractibility and Eye Tracking During Online Learning

More than half of students think their attention is easily shifted when they’re learning online. Distractibility, to a certain extent caused by visual stimuli is the main impact to decrease their academic performance. In addition, eye-tracking technology has been widely applied to explore distractibility in many “look” tasks, such as reading, viewing advertisements, and watching online videos as well as measure the efficiency of visual cognition. Therefore, this paper aimed to discuss the relationship between distractibility with eye movement indices and academic performance. Fifty high school students (30 girls) were recruited to complete experiment that was divided into two groups, which are the experimental group with distractions and controls with no one. The result showed that three of traditional eye movement indices were significantly correlated with distractibility ( $$p < 0.05$$ p < 0.05 ). Then we introduced the network accessibility model and the gaze transformation entropy to create two composite indexes according to the complexity and directivity of distractibility characteristics. The result revealed that the two composite indexes are significantly correlated with distractibility ( $$p < 0.05$$ p < 0.05 ). Finally, we constructed the mapping model about eye movement metrics about distractibility and online learning performance with a machine learning algorithm. The result ration was $$R^{2} = 0.799,$$ R 2 = 0.799 , and the error was $$Re < 0.1$$ R e < 0.1 , which proved the model was feasible and accessible. The research from the perspective of distractibility can provide valuable support for physiological indicators testing tools of academic performance and highlights the applications of eye movement dynamics.

Shanshan Chen, Yiqian Zhao, Tianyu Wu, Yajun Li

Human–Machine interaction and Learning Systems

Designing Augmented Reality Learning Systems with Real-Time Tracking Sensors

This study aims to develop an advanced augmented reality (AR) system by integrating an AR system and a real-time tracking system. Lab experiments are critical parts of engineering education, and it is possible to revolutionize engineering labs to be self-paced learning by using AR technology. However, the current hand gesture interactions to communicate with AR devices have shown several limitations. Hence, in this study, we developed a location-based AR system by integrating a real-time tracking sensor and an AR device. This new AR learning system could reduce transition time between learning modules and improve learners’ interaction between the modules and an AR device.

Wenbin Guo, Jung Hyup Kim
Practical Evaluation of Impression and Aesthetics for Public Displays: A Case Study in Evaluation of Platform Display Design

In the present paper, we apply psychological approach for impression, perception and aesthetics evaluations of public display design considering its competing business context. In designing of approach, we developed questionnaires and two versions of data collection methods with application guidelines. The approach was developed for practitioners, thus it included deep consideration about managerial limitations (e.g., time, cost, skill/knowledge needed for implementation). The proposed approach, which was simple and could be carried out easily, was applied to our case study where three platform displays’ evaluations were carried out. Results of evaluations as well as implications are discussed.

Hirotaka Aoki
Location of the Shift Technical Advisor Role in Nuclear Power Plant Scenarios - Impact on Performance

This paper presents a simulator study investigating two research questions: Will physical location of a shift technical advisor (STA) affect 1) nuclear power plant crew performance? 2) the fulfilment of expected STA duties? Two of six planned crews participated before the travel restrictions from Covid-19. Each crew run through six complex scenarios lasting for 40–60 min. The location of the STA varied as follows: 1) seated right next to the shift supervisor at the same workstation 2) at a separate workstation in the same room as the crew, 3) at a separate workstation in a separate room. Process expert ratings and operator self-ratings indicated a better crew performance when STA was located in condition 1). Furthermore, process expert ratings indicated that the STA performed his expected duties better when located in condition 3), while the operators preferred the STA to be located in condition 1).

Magnhild Kaarstad, Espen Nystad, Robert McDonald
Psycho-Educational Intervention Program to Eradicate Sexual Harassment for University Students

The present research study allowed fulfilling the objective of proposing a psychoeducational intervention program, according to the conception of violence, to eradicate sexual harassment in students of the Faculty of Engineering of the Universidad Nacional de Frontera in Sullana, Peru. The research is non-experimental and has a descriptive-correlational-propositional design. The sample consisted of 182 students, obtained by non-probabilistic convenience sampling. The survey technique was applied through two instruments: Inventory of conception of violence and the scale of perception against sexual harassment. The results indicate that there is a positive correlation, with strong strength and significance level (p  < .05), between the variable conception of violence of the program proposal and the perception of sexual harassment. In addition, approval was found with significant percentages in the following beliefs: “Sexual harassment occurs because one allows it”, “Most reports of sexual harassment are false”, and others.

Angélica Atoche, Ernesto Hernández, Victor Horna, Edwin Garcia, Lucia Pantoja
Does Being Human Cause Human Errors? Consideration of Human-Centred Design in Ship Bridge Design

75%–96% shipping accidents involved human-error in 2018 [1]. One of the most critical reasons was poor design of controls and lack of proper procedures [2]. It is unclear if ship-bridge design entirely considers user experience (UX). The crews are working in an increasingly time/resource pressured industry. Especially when hazardous scenarios occur, the increased amount of information being processed and decreased available time for decision-making together make it error-prone, adding an extra complexity to UX. Investigating the extent to which design influences human-error and distracts people from task’s reality, this paper evaluates the overlooked aspects of ship-bridge design to understand if it distorts user’s reality in hazardous situations and increases high cognitive loads with complex interfaces. Considering human-centred marine design (HCMD) that deals with end-user, addresses issues by adopting human factors/ergonomics (HF/E) introduced in industrial design, and applying product semantics/semiotics to the bridge design, reducing high cognitive loads with ergonomic interfaces.

Fang Bin Guo, Zaili Yang, Eddie Blanco Davis, Abdul Khalique, Alan Bury
Reflections of the Different Reasons for not Teleworking

The objective of this paper was to analyze the reflections of the different reasons for not teleworking on important aspects such as organizational commitment, intention to leave, job satisfaction and burnout. Most existing literature does not look at the different reasons for not teleworking. Four companies with different teleworking practices were selected and two phases survey were carried out. In the first phase, an interview was conducted with the human resources area to obtain information about the telework practice. In the second phase, questionnaires were emailed to all their employees to collect their perceptions: 632 questionnaires were answered, 249 from non-teleworkers. Strong evidence was found to the relationship between employees who were not allowed to telework and greater burnout and intention to leave and lower organizational commitment and satisfaction. The paper shows the importance of a good eligibility process and training for managers.

Simone Castro, Fernando Ferraz, Claudio Mahler, Isaac Santos
Psychological Impact on Design: Empirical Case Studies in City Regeneration of Post-industrial Sites

Industry restructuring are pressures faced by cities in both the East and West. Some successful Chinese examples do not consider the distinctive characteristics of local and demand for experience from people. Nowadays, people expect more than just function in their chosen environment but benefit from interaction, connection and engaging the senses. Good Human-Centred Emotional Design (HCED) generates not only happiness and but also a sense of security and safety. Empirical case study was used as a means of human-centred research, to explore the contribution of psychological factors in post-industrial sites regeneration. Applying Norman’s concept of three levels of design as a tool, this paper will examine whether the designs offered at current renovation projects satisfy visitors’ expectation, in terms of visceral appearance, experience of interaction with environment and finally the psychological satisfaction of people. Thereby, to clarify how can architectural/environmental psychology and emotional design theory enhance the design.

Xiaochun Zhan, Fangbin Guo, Stephen Fairclough, Denise Lee
Effect of Color Weight Balance on Visual Aesthetics Based on Gray-Scale Algorithm

This paper analyzes and studies the color weight balance value based on the weighting rule of gray-scale algorithm, explores the combination of color beauty measurement and planar visual aesthetics evaluation, discusses the effect of color weight balance on the visual aesthetics, and evaluates the matching experiment through image processing. The data fits the curve of the relationship between the balance of color weight and visual aesthetics, and it provides a basic reference for the quantitative evaluation of the aesthetic quality of human-computer interaction interfaces. In the end, it is proved that the color weight balance has positive effect on visual aesthetics and it is also affected by other surrounding conditions.

Tangling He, Jianrun Zhang
Improving Physical Activity with a Data-Analyzing Smart Insole that Assesses Root Causes of Chronic Pain and Physical Inactivity

Chronic lower body pain is a pervasive problem in the United States and globally. People with flat feet are more likely to experience plantar fasciitis, Achilles tendonitis, calluses, corns, blisters, bunions, hammertoes, shin splints, among many other ailments. If untreated, these issues may develop into more serious injuries and chronic pain. Physical inactivity due to chronic pain predisposes to a cluster of metabolic diseases. Low-fit individuals especially benefit from increasing physical activity. LAAF - Live Active and Agony Free - addresses the important role of physical activity in reducing mortality risk for individuals with chronic pain by enabling a healthy and active lifestyle. LAAF insoles provide comfort and support while analyzing gait, improving posture and alignment, and tracking fitness metrics. Data from the insoles is compared to normative gait data using algorithms to create user-specific reports in real-time. The LAAF mobile app connects users to caregivers, articles, and personalized exercises. Monitoring of these parameters is essential in the management of chronic pain and sedentary lifestyle. Management and prevention of chronic foot pain using current standards of care methods remains a challenge. LAAF Inc. has developed a smart wearable device with a mobile app to monitor and manage chronic pain as an effective strategy in reducing foot pain and increasing physical activity.

Meher Khan, Zain Hussain, Faasel Khan

Cognitive Neuroscience, Health Care and Artificial Intelligence (AI) Systems

Sense of Agency in Human-Machine Interaction

Although being in control is an important aspect of human-machine interaction, little is known about the combined effect of automation and mental workload on the sense of agency. In this study, participants were asked to reproduce the time interval between a keypress and an acoustic tone presented with different time delays (1250 to 2250 ms). Automation had three levels from the human being in complete control, an intermediate condition, to the machine being fully automatic. Mental workload was manipulated with a secondary memory task with two levels. Results showed a gradual loss of sense of agency with increasing automation intervention. Mental workload was found to affect only the intermediate automation condition. Further, we found an Intentional Binding effect for delays longer than 1750 ms in this intermediate condition. These findings demonstrate the existence of a residual sense of agency, which has important implications for the future design of hybrid, semi-autonomous systems.

Debora Zanatto, Mark Chattington, Jan Noyes
The Differences in Information Transmission Efficiency - A Comparison of Analog and Digital Media

In Japan, many people are still not familiar with reading documents digitally, although digital books have been widely promoted recently. To disclose the obstacles preventing digital reading, a fundamental experiment investigating the difference in information-transmission efficiency between analog and digital media was performed. In this investigation, the participants were asked to read documents in two different forms, in a printed form (analog media) and a displayed form (digital media). In addition to the analog and digital comparison, the experiment analyzed another aspect: the differences among young, middle-aged, and senior generations. Herein, the results of the investigation and the implications of the experiment results were discussed.

Jun Iio, Tatsuya Sashizawa, Kenta Kawamoto, Minami Higuchi
Using Virtual Reality in the Treatment of Social Anxiety Disorder: Technological Proposal

The treatments for anxiety that have proven efficacy are based on cognitive behavioral therapy, this time we will focus on systematic desensitization, which aims to expose the subject successively to the element that generates the maladaptive response, with this purpose a technological application is proposed for the treatment of this disorder from virtual reality. To accomplish this goal, an electronic helmet where different scenarios will be projected is going to be used. The person who is wearing it will face the distressing stimuli, allowing the treatment of his/her anxiety. With this application and device, it will be possible to safeguard the physical and psychic integrity of the subjects that demand this type of attention.

Carlos Ramos-Galarza, Pamela Acosta-Rodas, Jaime Moscoso-Salazar, Omar Condor-Herrera, Jorge Cruz-Cárdenas
Product Design for Yangliuqing Woodblock New Year Paintings Based on Eye Movement Experiment

The Yangliuqing woodblock New Year painting in Tianjin is one of the four major New Year paintings [The four major New Year paintings include woodblock New Year paintings in Mianzhu, Yangliuqing in Tianjin, Yangjiabu in Shandong, and Taohuawu in Jiangsu. Woodblock New Year paintings are color-overprinted woodblock New Year decorations that change every year. People put them indoors to wish good luck in the New Year.] in China and has been included in the protection list of “Intangible Cultural Heritage”. However, most of products about Yangliuqing only contain the visual elements directly and are designed for the purpose of decoration. In addition, the traditional paintings can no long reach the modern aesthetic standard. In order to solve this problem, the research aims at collecting and analyzing the eye movement data of products about Yangliuqing. Then, with factor analysis method, the visual image elements of the products were achieved by calculation and filter, finally actualizing the improvement and design for products after element deduction and recombination. The scheme effectively improved the problems existing in the actual design and application of traditional prints, which has certain reference significance for the extraction of visual images and innovative applications in traditional Chinese painting.

Shangshi Pan, Beibei Dong, RongRong Fu
Human-Computer Interaction (HCI) Approach for the Optimal Generation and Selection of Batches Destination Options in Steel Making Factories

The objectives of the destination options generation of the semi-products batches coming from the continuous casting installations for the production of finished steel profiles in lamination workshops are close to the minimum of the excess of mechanical properties respecting its normed values in the steel industry. The estimation of mechanical properties of the batches starting from its chemical composition and traverse surface of the billets and finished profiles is done by radial based neural networks, starting from the available mechanical properties data obtained from the quality control of the workshops. The systemic analysis of the production function in steel factories allows to formulate the conceptual optimization model, that breaks down in sub-models of the batches destination options generation, as a discrete stochastic optimization and the selection of the batches to be to satisfy the sales demand sub-tasks. Solutions outlines of the generation and selection stages are also presented.

Denis-Joaquín Zambrano-Ortiz, José Arzola-Ruiz, Rosa-Mariuxi Litardo-Velásquez, Umer Ashger
Advantage Design of Small Commodities Under Cultural Transfer

Yiwu, China, is the world’s largest consumer goods distribution center, and more than 2.1 million kinds of small commodities are sold all over the world. Cultural transfer refers to the process in which ideas, experience, skills and other cultural characteristics are transferred from one place to another. Due to the dual attributes of functional demand and cultural integration of products in international trade, the process of product trade is bound to be accompanied by the result of cultural transfer. In view of the urgent need for Yiwu small commodities to gain competitive advantages in international trade, according to the path of cultural transfer, using the idea, principle and technology of advantage design to put forward the advantages design strategies, such as the traceability design under the positive cultural transfer, the fusion design under the negative cultural transfer, and the system design under the two-way cultural transfer. And according to the market-oriented, forward-looking, dynamic and procedural attributes of design, the advantage design method based on the stage demand of the whole process in the future is constructed. At the same time, case studies of small commodities in international trade show that the design strategy has good application value. The research results of this paper have certain guiding significance and practical value for the future international trade of consumer goods to obtain competitive advantage, and will further improve the design level of China’s small commodities.

Li-xia Hua, Jian-ping Yang, Jun-nan Ye, Yi-xiang Wu, Shan-wei Zhang

Cognitive Living Spaces Using IoT Devices

Cognitive Living Spaces by Using IoT Devices and Ambient Biosensor Technologies

In the near future, our deeply connected and fully digitized physical facilities require cognitive processes that are embodied in a regular sensing of living environments. Pervasive sensor networks will enable the development of more efficient technologies that will integrate Artificial Intelligence based services better into psychosocial and human ecological contexts.These innovative mixed and integrated biophysical and digital living spaces enable to use them more efficiently, conveniently, and, furthermore, to interpret these new intelligent environments in a completely different way. Based on the growing insight into relations between human beings and their surroundings, residents get an overview of their building ecosphere. They obtain the potential to develop strategies in context with intelligent buildings that are enabled to assist in changing behaviors in everyday life.We study human factors, in particular, the potential of change in human behavior in such fully integrated services, strongly related to our living and work place in the case of home office. After using these sentient cognitive environments, we explore novel ways of interacting with living and work spaces by offering opportunities to give intelligent environments a virtual voice and representation via the digital data space. During the COVID-19 pandemic, people should pay more attention to their immediate living and working environment: it becomes more important to monitor the quality of the air to breathe, surroundings are made visible to get to know them better, cope better, enjoy them, etc.The presented work provides in this context quantitative data from novel low-cost biosensors, such as for measuring carbon dioxide concentration distribution, highlighting the presence and attention of residents and their change in behavior within a sample living space, and also provides conclusions towards novel research pathways for integrating cognitive processes into a network of IoT devices and ambient biosensor technologies.

Zeiner Herwig, Lucas Paletta, Julia Aldrian, Roland Unterberger
Human-Centric Emergent Configurations: Supporting the User Through Self-configuring IoT Systems

The Internet of Things (IoT) is revolutionizing our environments with novel types of services and applications by exploiting the large number of diverse connected things. One of the main challenges in the IoT is to engineer systems to support human users to achieve their goals in dynamic and uncertain environments. For instance, the mobility of both users and devices makes it infeasible to always foresee the available things in the users’ current environments. Moreover, users’ activities and/or goals might change suddenly. To support users in such environments, we developed an initial approach that exploits the notion of Emergent Configurations (ECs) and mixed initiative techniques to engineer self-configuring IoT systems. An EC is a goal-driven IoT system composed of a dynamic set of temporarily connecting and cooperating things. ECs are more flexible and usable than IoT systems whose constituents and interfaces are fully specified at design time.

Fahed Alkhabbas, Romina Spalazzese, Paul Davidsson
Playful Screening of Executive Functions Using Augmented Reality and Gaze Based Assessment

Augmented Reality (AR) technologies have recently been explored for application in dementia care, including cognitive training and screening, navigational assistance to find their way around, assistance to identify friends and family members, and assistance with activities of daily living. The presented work proposes the use of mobile and playful assessment of executive functions in the home environment by AR technology. We implemented a gamified version of the neuropsychological test ‘Tower of London’ (Shallice, 1982 [1]). It is played with poles of various size and colored balls, starting each exercise with a certain start and desired goal configuration of balls on poles. The novel AR-based technology enables hand-based interaction with artificial objects in the field of view and monitoring of the user’s gaze behavior via eye tracking towards the artificial objects of interest. A pilot study was performed with 12 healthy people that played the game with various degrees of difficulty concerning its planning depth. The results of the study demonstrate statistically significant correlation between the eye tracking features and the planning ability as well as cognitive flexibility measured by standardized psychological tests. This study provides first indication that executive functions can be estimated from playful AR-based interaction. Future work will focus on adjusting the game for persons with dementia, such as, by decreasing the difficulty level appropriately.

Martin Pszeida, Amir Dini, Sandra Schüssler, Claudia Voithofer, Jean-Philippe Andreu, Philipp Hafner, Lucas Paletta
Advanced Cyber and Physical Situation Awareness in Urban Smart Spaces

The ever-growing adoption of big data technologies, smart sensing, data science and artificial intelligence is enabling the development of new intelligent urban spaces with real-time monitoring and advanced cyber-physical situational awareness capabilities. In the S4AllCities international research project, the advancement of cyber-physical situational awareness will be experimented for achieving safer smart city spaces in Europe and beyond. The deployment of digital twins will lead to understanding real-time situation awareness and risks of potential physical and/or cyber-attacks on urban critical infrastructure specifically. The critical extraction of knowledge using digital twins, which ingest, process and fuse observation data and information, prior to machine reasoning is performed in S4AllCities. In this paper, a cyber behavior detection module, which identifies unusualness in cyber traffic networks is described. Also, a physical behaviour detection module is introduced. The two modules function within the so-called Malicious Attacks Information Detection System (MAIDS) digital twin.

Zoheir Sabeur, Constantinos Marios Angelopoulos, Liam Collick, Natalia Chechina, Deniz Cetinkaya, Alessandro Bruno
Design of an IoT Architecture in Livestock Environments for the Treatment of Information for the Benefit of Cattle

Internet of things (IoT) is the interconnection of one or more technological devices with any other around it. The use of IoT in the livestock* sector helps in an advanced, simple and practical way by giving farmers the possibility of generating comments when making decisions, optimizing the growth and welfare of the animals, thus improving the agricultural production matrix. Several IoT architecture models allow the implementation of IoT on a large scale, two designs of architecture models will be denoted specifying and suggesting their use in livestock in our country. Having online information on animals is essential since the state of health, geo position, or location among others is revolutionizing the livestock business, thus turning the IoT market into a great ally of farmers, providing a future of great opportunities and improvement mentions. Basic criteria are detailed to lighten decision-making when choosing an IoT technology to be implemented. These criteria are developed after carrying out an analysis of documents, magazines, works, publications among others, identifying the needs in the development of livestock in rural areas of the province of Guayas.

Miguel Angel Quiroz Martinez, David Manuel Rodriguez Zapata, Monica Daniela Gomez Rios, Maikel Yelandi Leyva Vazquez

Intelligent Computing and Cognitive Computing in Healthcare

Requirements Analysis on Emotional Preferences for Leisure Activities in Virtual Reality for Female Nursing Home Residents – A Mixed Method Approach

Insufficient tailoring of individual needs of nursing home residents can lead to standardized (leisure) activities. Virtual Reality (VR) provides the potential to offer a multimodal experience of individualized activities to residents. The aim of the VR4Care study is to explore attitudes, knowledge, and expectations regarding the use of VR glasses for (leisure) activities in female residents living in rural nursing homes. A mixed method approach in two phases (phase 1: interviews with a complementary standardized questionnaire; phase 2: measurement of affective states) was used. In total, 20 participants were interviewed regarding nine categories. The results show that the most described VR-scenarios for (leisure) activities include arts and crafts (e.g., sewing and knitting), followed by gardening (e.g., farmhouse garden), washing and ironing clothes and dancing (e.g., Austrian folk dances). This study serves as an important basis for scenario development and for testing the use of biosensor measurements in phase 2.

Alfred Haeussl, Sandra Schuessler, Lucas Paletta, Hermine Fuerli, Beatrix Koch, Thomas Binder, Michael Schneeberger, Jean-Philippe Andreu, Sybille Reidl, Sarah Beranek, Robert Hartmann, Martin Sighart
Virtual Reality-Based Sensory Triggers and Gaze-Based Estimation for Mental Health Care

Recent studies underline the importance of the cognitive reserve for mental health, especially in dementia care, which is supported by stress reduction, joyful experience and meditation. Mindfulness training has previously been successfully applied to dementia and indicates a lasting positive effect on cognitive reserve, well-being and motivation [1.Clin. Psychol. Rev. 31:449–464]. We investigated the potential of unobtrusive technology for the measurement of eye movements in Virtual Reality (VR)-based mindfulness training. The objective of this research is to develop software estimators for cognitive assessment and mindfulness trait in order to apply VR technology in the future as a screening instrument, monitoring tool and thereby serving for decision support in mental health care. Eye movement analysis within a pilot study demonstrated significantly different results for persons with Alzheimer’s dementia and healthy controls. These results indicate that significant conclusions are drawn on relevant mental health parameters even within a very short eye movement measurement period applying few minutes of observation of carefully selected video-based stimuli.

Lucas Paletta, Martin Pszeida, Sandra Schüssler, Jean-Philippe Andreu, Amir Dini, Elke Zweytik, Josef Steiner, Andrea Grabher, Julia Lodron
Towards Decision Support with Assessment of Neuropsychological Profiles in Alzheimer's Dementia Using Playful Tablet-Based Multimodal Activation

In dementia care there is a lack of knowledge in the context of interventions considering the evolvement of individual cognitive impairments over the course of time. Dementia principally affects distinctive neuroanatomic networks associated with complex cognitive domains, i.e., the neuropsychological profile. The PLAYTIME app - a suite of serious games - was played on a Tablet-PC combining two approaches. One component represented a serious game with integrated eye movement analysis. This functionality is outlined in the MIRA (Mobile Instrumental Review of Attention) framework, a toolbox of attention-based games. The toolbox is used to evaluate executive functions, such as, the inhibitory functionality of controlled eye movements. Another component is represented by the multimodal activation (MMA) app with various cognitively challenging gamified exercises. Both MIRA and MMA showed different profiles of correlation between their play scores and several MoCA (Montreal Cognitive Assessment) sub-scores. The score captured from MIRA as well as from MMA enables to estimate Alzheimer’s mental state and establish neuropsychological profiles to identify individual cognitive deficits.

Lucas Paletta, Martin Pszeida, Maria Fellner, Silvia Russegger, Amir Dini, Sandra Draxler, Thomas Orgel, Anna Jos, Eva Schuster, Josef Steiner
THERADIA: Digital Therapies Augmented by Artificial Intelligence

Digital plays a key role in the transformation of medicine. Beyond the simple computerisation of healthcare systems, many non-drug treatments are now possible thanks to digital technology. Thus, interactive stimulation exercises can be offered to people suffering from cognitive disorders, such as developmental disorders, neurodegenerative diseases, stroke or traumas. The efficiency of these new treatments, which are still primarily offered face-to-face by therapists, can be greatly improved if patients can pursue them at home. However, patients are left to their own devices which can be problematic. We introduce THERADIA, a 5-year project that aims to develop an empathic virtual agent that accompanies patients while receiving digital therapies at home, and that provides feedback to therapists and caregivers. We detail the architecture of our agent as well as the framework of our Wizard-of-Oz protocol, designed to collect a large corpus of interactions between people and our virtual assistant in order to train our models and improve our dialogues.

Franck Tarpin-Bernard, Joan Fruitet, Jean-Philippe Vigne, Patrick Constant, Hanna Chainay, Olivier Koenig, Fabien Ringeval, Béatrice Bouchot, Gérard Bailly, François Portet, Sina Alisamir, Yongxin Zhou, Jean Serre, Vincent Delerue, Hippolyte Fournier, Kévin Berenger, Isabella Zsoldos, Olivier Perrotin, Frédéric Elisei, Martin Lenglet, Charles Puaux, Léo Pacheco, Mélodie Fouillen, Didier Ghenassia

Cognitive Assessment and Physical Strain of First Responders and Action Forces

Electrotactile Stimulation, A New Feedback Channel for First Responders

This paper presents the early results of research aiming to develop a novel system for unobtrusive and intuitive electrotactile feedback for first responders. The system leverages the multi-pad stimulation technology based on spatiotemporal modulation of the stimuli. Two-point discrimination threshold mapping was performed in potential electrode placement locations, defined from the usability perspective by the first responders in initial co-development sessions. Based on these results a custom electrode design was proposed and validated in six healthy volunteers. Psychometric testing was conducted to determine spatial discrimination between stimuli produced by the multi-pad electrode. The average success rate of 80% indicates that the proposed approach is feasible.

Matija Štrbac, Milica Isaković, Jovana Malešević, Gorana Marković, Strahinja Došen, Nikola Jorgovanović, Goran Bijelić, Milos Kostić
Multisensory Wearable Vital Monitoring System for Military Training, Exercise and Deployment

Military organizations have extensive technological solutions to precisely monitor machines and operating equipment. In recent decades, extensive research and development projects have been launched focusing on the physiological monitoring of soldiers, with new opportunities arising from innovative developments in the field of biosensors. This paper describes the main objectives of the VitalMonitor project, which is carried out in the frame of the Austrian Defence Research Program FORTE (FORTE - Austrian Defence Research Program; ). The project focuses on the development of a real-time monitoring system for situation-dependent physiological load on soldiers based on innovative body worn biosensors integrated into clothing or equipment. Intelligent sensor fusion and data analysis methods enable an overview of the actual physical stress situation in military training, exercises or missions. The analysis of scenario-based physiological requirements will be the basis for the optimization of physical resilience as well as operational readiness and will eventually reduce the risk for dangerous situations caused by physical exhaustion.

Alexander Almer, Anna Weber, Lucas Paletta, Michael Schneeberger, Stefan Ladstätter, Dietmar Wallner, Günter Grabher, Peter Süss, Philip Klöckl, Patrick Fuchshofer, Thomas Hölzl

Multimodal Measurements, Artificial Intelligence and Mental Structure

A Database for Cognitive Workload Classification Using Electrocardiogram and Respiration Signal

Cognitive workload is a critical factor in determining the level of attentional effort exerted by users. Understanding and classifying cognitive workload is challenging as individuals exert varying levels of mental effort to meet the task's underlying demands. Twenty-six participants (12M, 14F, Mean = 22.68 ± 5.10) were exposed to two different tasks designed to induce low and high cognitive workloads. Subjective and objective measures were collected to create a novel, validated multimodal dataset for cognitive workload classification. Participants’ perceived workload was collected using the NASA-TLX. Electrocardiogram (ECG) and Respiration (RR) data were collected to extract the Heart Rate Variability and Respiration Rate Variability features. Four machine learning algorithms were utilized to classify cognitive workload levels where AdaBoost classifier achieved the highest Leave-One-Subject-Out Cross-Validation accuracy, and F1-Score of 80.2%, 80.3% respectively. This is the first publicly available dataset with ECG, RR and subjective responses for cognitive workload classification.

Apostolos Kalatzis, Ashish Teotia, Vishnunarayan Girishan Prabhu, Laura Stanley
Using BERT Model for Intent Classification in Human-Computer Dialogue Systems to Reduce Data Volume Requirement

User-intent classification is a sub-task in natural language understanding of human-computer dialogue systems. To reduce the data volume requirement of deep learning for intent classification, this paper proposes a transfer learning method for Chinese user-intent classification task, which is based on the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. First, a simulation experiment on 31 Chinese participants was implemented to collect first-handed Chinese human-computer conversation data. Then, the data was augmented through back-translation and randomly split into the training dataset, validation dataset and test dataset. Next, the BERT model was fine-tuned into a Chinese user-intent classifier. As a result, the predicting accuracy of the BERT classifier reaches 99.95%, 98.39% and 99.89% on the training dataset, validation dataset and test dataset. The result suggests that the application of BERT transfer learning has reduced the data volume requirement for Chinese intent classification task to a satiable level.

Hao Liu, Huaming Peng
Human-Machine Learning with Mental Map

Many cognitive processes can be represented as a graph that is visual and computational. Graph search used to be a classic AI method. Here we present a dynamic graph, called “Mental Map” with a set of timestamps, nodes, edges, attributes, and operators for extracting experts’ knowledge and incorporating other AI models such as Association Rule Learning and Decision Tree. Mental Map is written in Javascript and it can run on any platform that has a web browser. Three case studies are presented: suspicious behavior detection, email phishing, and malware detection in embedded systems. The tool can be used interactively and automatically. Through human-machine collaborative learning, Mental Map provides more explainability and flexibility than prevailing semantic webs and machine learning algorithms.

Yang Cai
Advances in Neuroergonomics and Cognitive Engineering
Dr. Hasan Ayaz
Umer Asgher
Lucas Paletta
Copyright Year
Electronic ISBN
Print ISBN