2025 | Buch

Kapitel lesen Erstes Kapitel lesen

Multimodal Interaction Technologies for Training Affective Social Skills

herausgegeben von: Satoshi Nakamura

Verlag: Springer Nature Singapore

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik"

Inhaltsverzeichnis

Über dieses Buch

Dieses Buch konzentriert sich darauf, wie interaktive, multimodale Technologien wie virtuelle Agenten in Training und Behandlung (soziales Kompetenztraining, kognitive Verhaltenstherapie) eingesetzt werden können. Menschen mit sozio-affektiven Defiziten haben Schwierigkeiten, ihr Sozialverhalten zu kontrollieren und leiden auch unter der Interpretation des Sozialverhaltens anderer. Verhaltenstrainings wie Sozialkompetenztraining werden im klinischen Umfeld eingesetzt. Die Patienten werden von einem Coach geschult, um soziale Interaktion zu erleben und sozialen Stress abzubauen. Zusätzlich zum Verhaltenstraining ist die kognitive Verhaltenstherapie auch nützlich, um ein besseres Verständnis zu entwickeln und sozial-affektive Interaktion zu trainieren. All diese Methoden sind effektiv, aber teuer und schwer zugänglich. Dieses Buch beschreibt, wie multimodale interaktive Technologien im Gesundheitswesen eingesetzt werden können, um sozial-affektive Interaktionen zu messen und zu trainieren. Sensortechnologie analysiert das Verhalten und den Blick der Nutzer, und verschiedene Methoden des maschinellen Lernens können für Vorhersageaufgaben eingesetzt werden. Dieses Buch konzentriert sich auf die Analyse menschlichen Verhaltens und die Implementierung von Trainingsmethoden (z.B. durch virtuelle Agenten, virtuelle Realität, Dialogmodellierung, personalisiertes Feedback und Bewertungen). Zu den Zielgruppen zählen Depressionen, Schizophrenie, Störungen des Autismus-Spektrums und eine viel größere Gruppe sozialer pathologischer Phänomene.

Mit KI übersetzt

Inhaltsverzeichnis

Frontmatter

Introduces the Social Interaction Theory

Frontmatter

Introduction

Abstract

Social skills refer to the ability to manage verbal and nonverbal behaviors during interactions with one or more individuals. People who struggle with social interactions often find it challenging to accurately grasp their own social behaviors and understand how others perceive them. Social Skills Training (SST), typically conducted by clinical psychologists or psychiatrists, aims to enhance social interactions and reduce social stress. In addition to traditional SST, we also explore cognitive-behavioral therapy (CBT), a psychiatric approach that improves cognitive abilities and alleviates symptoms such as anxiety and depression stemming from a lack of social skills. To implement these methods in practice, we design conversational virtual agents capable of taking on various roles and scenarios. This research project, initiated in October 2019, is part of a Japan-France international joint ANR-CREST research initiative.

Satoshi Nakamura

Social Skills Theoretical Framework

Abstract

This chapter examines various psychological approaches that have sought to understand and conceptualize social competence and social skills. We explore several theoretical models that attempt to define the key notions surrounding this competence, highlighting the distinction between social skills, which are observable and acquired, and social competence, which is more global and multidimensional. These models provide frameworks for analyzing social adaptation, performance in specific tasks, and social skills, while considering contextual influences and the evaluation by others in assessing social effectiveness.

Jennifer Hamet Bagnou, Elise Prigent, Jean-Claude Martin, Céline Clavel

Theory of Social Skills Training (SST)

Abstract

Social skills training (SST) is a structured approach to improving how individuals interact with others. It focuses on developing specific skills that help people navigate interpersonal situations more effectively. SST is used for all ages, from children to adults, and is also used for people with psychiatric disorders, such as autism spectrum disorder (ASD) and schizophrenia, who have difficulty building interpersonal relationships. Previous studies have reported that SST is useful in improving social skills, including those of people with psychiatric disorders. However, SST requires multiple facilitators with specific expertise, making access to it difficult. To overcome such disadvantages, this study investigates the usefulness of automated SST-based technology. This chapter describes conventional SST for children and adults, as well as the type of SST used in this research.

Mitsuhiro Uratani, Tsubasa Morimoto

Theory of Cognitive Behavioral Therapy

Abstract

Cognitive Behavioral Therapy (CBT) is now being applied not only to depression and anxiety disorders but also to a wide range of psychiatric disorders. In Japan, in 2010, cognitive therapy and cognitive behavioral therapy, which are implemented by skilled physicians based on manuals for mood disorders, became eligible for medical reimbursement. Furthermore, in 2016, the scope was expanded to include anxiety disorders such as obsessive–compulsive disorder, social anxiety disorder, panic disorder, and post-traumatic stress disorder. In the United States, psychotherapy became mandatory during resident training in 2001, and subsequent revisions identified three specific types of psychotherapy: supportive psychotherapy, psychodynamic psychotherapy, and cognitive behavioral therapy. Today, CBT is not just a therapeutic approach one should be familiar with; it has become a practice that requires the ability to implement or appropriately refer alongside pharmacotherapy.

Takashi Kudo

Introduces the Measurement of Social Skills

Frontmatter

Social Performance Rating

Abstract

Behavioral observation scales are important for understanding and assessing social skills. In this chapter, we explore tools for assessing social skills. It highlights the limitations of existing methods, such as self-assessments and observation tools, which often suffer from response bias. The chapter introduces and validates two scales: the SSC (Social Skill for Collaboration) for measuring collaborative problem-solving skills and the SPRS (Social Performance Rating Scale) for communication skills.

Jennifer Hamet Bagnou, Céline Clavel, Jean-Claude Martin, Elise Prigent

Eye-Gaze Analysis

Abstract

This chapter explains eye movements during collaborative problem-solving. Collaborative problem-solving means resolving difficulties and working toward a goal during social collaboration with others. Social life is filled with collaborative problem-solving tasks, such as group work in schools or workplaces. Understanding behavioral signals is one important step in improving performance in collaborative problem-solving. We focus on eye movements as an example in this section. We show the analyses of eye movements related to social performance and discuss future directions for investigating eye movements to improve collaborative problem-solving.

Kana Miyamoto, Kosuke Okazaki

Multimodal Behavior Modeling

Abstract

This chapter explains the complexities of multimodal behavior modeling. While humans recognize social intentions unconsciously, the automation of such recognition involves significant challenges, particularly in spontaneous interactions. We explore the variety of multimodal behavior modeling through two specific examples—ASD severity estimation and social skill estimation—to showcase its potential. Using datasets from clinical populations and machine learning techniques, the chapter demonstrates the potential analytic power of automatic behavior modeling with multimodal features like text, audio, and visual cues. We also discuss expected challenges in this field in the era of LLMs, such as the need for sequential modeling and the limited amount of data. Accordingly, we hope to show the future direction of multimodal behavior modeling.

Takeshi Saga, Hiroki Tanaka

Introduces the Virtual Agent

Frontmatter

Multimodal Adaptive Behavior Generation During Human–Agent Interaction

Abstract

Interaction consists of verbal exchange as well as multimodal social signals. Multimodality is essential in communicating fruitfully by sending implicit and explicit information. When conversing, multimodal signals are transferred back and forth between interlocutors. Via this transfer, interlocutors constantly adapt their behaviors to those of their interlocutors. Virtual agents, which look and act like humans, should also consider the multimodal and adaptive aspect of the interaction they build with the users. In this chapter, we elaborate on adaptive behavior evaluation methods, multimodal adaptive behavior generation modeling, and the effect of a virtual agent providing real-time adaptive behaviors for cognitive behavioral therapy (CBT).

Jieyeon Woo, Catherine Achard, Catherine Pelachaud

Virtual Reality and Augmented Reality

Abstract

Public speaking skills, the ability to address large audiences and interact with individuals, are crucial in social life. This chapter presents two studies on applying virtual reality (VR) technologies to public speaking training and support. The first study investigates the use of VR for reflection on public speaking training. Commonly, individuals record and review their presentations to identify areas for improvement. However, reviewing one’s own videos can cause discomfort and hinder objective self-assessment. To address this issue, the study confirmed the effectiveness of employing an avatar with a different appearance and using VR to review presentations from the audience’s perspective [38]. The second study examines the provision of real-time feedback in a VR environment. In VR, comprehensive data, including body movements, gaze, speech, and slide manipulation, can be utilized to detect issues and provide immediate feedback. However, due to the cognitive demands of the presentation, users may not always accept feedback. To alleviate this problem, the possibility of estimating feedback timing that is inappropriate for the user from observable information was investigated [8].

Yuichiro Fujimoto

Introduces Interaction and Pedagogical Feedback

Frontmatter

Interaction in Social Skills Training

Abstract

Social communication is a crucial factor influencing human social life. Quantifying the degree of difficulty an individual faces in social communication is necessary to understand developmental and neurological disorders and to create systems for automatic screening as well as assistive methods such as social skills training (SST). As discussed elsewhere in this book, SST by experienced human trainers is a well-established method. In this chapter, we describe interaction in social skills training. Specifically, we present an analysis of the human social skills training dataset, the development of virtual SST, and the acceptability of this approach.

Hiroki Tanaka, Satoshi Nakamura

Interaction in Cognitive Behavioral Therapy

Abstract

Cognitive Behavioral Therapy (CBT) is a widely recognized and practical approach for treating various mental health issues, including depression, anxiety, and stress-related disorders. By focusing on identifying and restructuring negative thought patterns and behaviors, CBT empowers individuals to develop healthier coping mechanisms. Traditionally administered through face-to-face sessions with trained therapists, CBT has proven efficacy but faces barriers that limit its accessibility, such as limited availability of professionals, high costs, and geographical constraints. In this chapter, we address these limitations by presenting an adaptive Embodied Conversational Agent (ECA) that adjusts its questioning strategy based on the user’s real-time psychological distress. By integrating a distress detection model analyzing users’ speech transcriptions, the ECA personalizes the therapeutic interaction to ensure that interventions are neither overwhelming nor insufficient. We conducted an experiment with 49 participants, comparing an adaptive condition—where the number of Socratic questions was modified according to detected distress—with a random condition. Results demonstrated that the adaptive approach yielded superior outcomes in reducing psychological distress and negative moods. These findings highlight the potential of ECAs to offer CBT sessions for a broader population.

Kazuhiro Shidara, Takashi Kudo

Introduces Evaluation

Frontmatter

Validation of Developed Automated SST System

Abstract

This study aims to validate an automated social skills training (SST) system designed to assist individuals with distinct social communication challenges. Specifically, it targets those diagnosed with schizophrenia, who often struggle with social withdrawal, flat affect, and difficulty in interpreting social cues, as well as individuals with autism spectrum disorder, who typically face challenges in understanding nonverbal communication, maintaining reciprocal conversations, and adapting to social norms. The system is intended to help these individuals objectively assess their social interactions, learn appropriate social cues, and develop strategies to improve their communication skills in various social contexts. We conducted two key verifications: (1) comparing the improvement in social skills between a group of healthy adults trained with our SST system and a wait-list control group of healthy adults who received no intervention during the study period and (2) assessing the effectiveness of our SST system compared to traditional human–human SST. This chapter presents an analysis of the first verification, which has been completed, and outlines the experimental design for the second, which is currently in progress. Our findings demonstrate the effectiveness of the automated SST system over a four-week training period, with significant improvements in general self-efficacy, reduction in state anxiety, and enhanced speech clarity in the trained group.

Masato Honda, Kosuke Okazaki, Hiroki Tanaka

Titel: Multimodal Interaction Technologies for Training Affective Social Skills
herausgegeben von: Satoshi Nakamura
Verlag: Springer Nature Singapore
Electronic ISBN: 978-981-9613-13-7
Print ISBN: 978-981-9613-12-0
DOI: https://doi.org/10.1007/978-981-96-1313-7