Companion Technology

Frontmatter

Chapter 1. An Introduction to Companion-Technology

Abstract

Companion-technology enables a new generation of intelligent systems. These Companion-systems smartly adapt their functionality to a user’s individual requirements. They comply with his or her abilities, preferences, and current needs and adjust their behavior as soon as critical changes of the environment or changes of the user’s emotional state or disposition are observed. Companion-systems are distinguished by characteristics such as competence, individuality, adaptability, availability, cooperativeness, and trustworthiness. These characteristics are realized by integrating the technical functionality of systems with a combination of cognitive processes. Companion-systems are able to perceive the user and the environment; they reason about the current situation, exploit background knowledge, and provide and pursue appropriate plans of action; and they enter into a dialog with the user where they select the most suitable modes of interaction in terms of media, modalities and dialog strategies. This chapter introduces the essence of Companion-technology and sheds light on the huge range of its prospective applications.

Susanne Biundo, Andreas Wendemuth

Chapter 2. Multi-level Knowledge Processing in Cognitive Technical Systems

Abstract

Companion-Systems are composed of different modules that have to share a single, sound estimate of the current situation. While the long-term decision-making of automated planning requires knowledge about the user’s goals, short-term decisions, like choosing among modes of user-interaction, depend on properties such as lighting conditions. In addition to the diverse scopes of the involved models, a large portion of the information required within such a system cannot be directly observed, but has to be inferred from background knowledge and sensory data—sometimes via a cascade of abstraction layers, and often resulting in uncertain predictions. In this contribution, we interpret an existing cognitive technical system under the assumption that it solves a factored, partially observable Markov decision process. Our interpretation heavily draws from the concepts of probabilistic graphical models and hierarchical reinforcement learning, and fosters a view that cleanly separates between inference and decision making. The results are discussed and compared to those of existing approaches from other application domains.

Thomas Geier, Susanne Biundo

Chapter 3. Model-Based Frameworks for User Adapted Information Exploration: An Overview

Abstract

The target group of search engine users in the Internet is very wide and heterogeneous. The users differ in background, knowledge, experience, etc. That is why, in order to find relevant information, such search systems not only have to retrieve web documents related to the search query but also have to consider and adapt to the user’s interests, skills, preferences and context. In addition, numerous user studies have revealed that the search process itself can be very complex, in particular if the user is not providing well-defined queries to find a specific piece of information, but is exploring the information space. This is very often the case if the user is not completely familiar with the search topic and is trying to get an overview of or learn about the topic at hand. Especially in this scenario, user- and task-specific adaptations might lead to a significant increase in retrieval performance and user experience. In order to analyze and characterize the complexity of the search process, different models for information(-seeking) behavior and information activities have been developed. In this chapter, we discuss selected models, with a focus on models that have been designed to cover the needs of individual users. Furthermore, an aggregated framework is proposed to address different levels of information(-seeking) behavior and to motivate approaches for adaptive search systems. To enable Companion-Systems to support users during information exploration, the proposed models provide solid and suitable frameworks to allow cooperative and competent assistance.

Michael Kotzyba, Tatiana Gossen, Sebastian Stober, Andreas Nürnberger

Chapter 4. Modeling Aspects in Human-Computer Interaction: Adaptivity, User Characteristics and Evaluation

Abstract

During system interaction, the user’s emotions and intentions shall be adequately determined and predicted to recognize tendencies in his or her interests and dispositions. This allows for the design of an evolving search user interface (ESUI) which adapts to changes in the user’s emotional reaction and the users’ needs and claims.

Here, we concentrate on the front end of the search engine and present two prototypes, one which can be customised to the user’s needs and one that takes the user’s age as a parameter to roughly approximate the user’s skill space and for subsequent system adaptation. Further, backend algorithms to detect the user’s abilities are required in order to have an adaptive system.

To develop an ESUI, user studies with users of gradually different skills have been conducted with groups of young users. In order to adapt the interaction dialog, we propose monitoring the user’s emotional state. This enables monitoring early detection of the user’s problems in interacting with the system, and allows us to adapt the dialog to get the user on the right path. Therefore, we investigate methods to detect changes in the user’s emotional state.

We furthermore propose a user mood modeling from a technical perspective based on a mechanical spring model in PAD-space, which is able to incorporate several psychological observations. This implementation has the advantage of only three internal parameters and one user-specific parameter-pair.

We present a technical implementation of that model in our system and evaluate the principal function of the proposed model on two different databases. Especially on the EmoRecWoz corpus, we were able to show that the generated mood course matched the experimental setting.

By utilizing the user-specific parameter-pair the personality trait extraversion was modeled. This trait is supposed to regulate the individual emotional experiences.

Technically, we present an implementable feature-based, dimensional model for emotion analysis which is able to track and predict the temporal development of emotional reactions in an evolving search user interface, and which is adjustable based on mood and personality traits.

Tatiana Gossen, Ingo Siegert, Andreas Nürnberger, Kim Hartmann, Michael Kotzyba, Andreas Wendemuth

Chapter 5. User-Centered Planning

Abstract

User-centered planning capabilities are core elements of Companion-Technology. They are used to implement the functional behavior of technical systems in a way that makes those systems Companion-able—able to serve users individually, to respect their actual requirements and needs, and to flexibly adapt to changes in their situation and environment. This chapter presents various techniques we have developed and integrated to realize user-centered planning. They are based on a hybrid planning approach that combines key principles also humans rely on when making plans: stepwise refining complex tasks into executable courses of action and considering causal relationships between actions. Since the generated plans impose only a partial order on actions, they allow for a highly flexible execution order as well. Planning for Companion-Systems may serve different purposes, depending on the application for which the system is created. Sometimes, plans are just like control programs and executed automatically in order to elicit the desired system behavior; but sometimes they are made for humans. In the latter case, plans have to be adequately presented and the definite execution order of actions has to coincide with the user’s requirements and expectations. Furthermore, the system should be able to smoothly cope with execution errors. To this end, the plan generation capabilities are complemented by mechanisms for plan presentation, execution monitoring, and plan repair.

Pascal Bercher, Daniel Höller, Gregor Behnke, Susanne Biundo

Chapter 6. Addressing Uncertainty in Hierarchical User-Centered Planning

Abstract

Companion-Systems need to reason about dynamic properties of their users, e.g., their emotional state, and the current state of the environment. The values of these properties are often not directly accessible; hence information on them must be pieced together from indirect, noisy or partial observations. To ensure probability-based treatment of partial observability on the planning level, planning problems can be modeled as Partially Observable Markov Decision Processes (POMDPs).

While POMDPs can model relevant planning problems, it is algorithmically difficult to solve them. A starting point for mitigating this is that many domains exhibit hierarchical structures where plans consist of a number of higher-level activities, each of which can be implemented in different ways that are known a priori. We show how to make use of such structures in POMDPs using the Partially Observable HTN (POHTN) planning approach by developing a Partially Observable HTN (POHTN) action hierarchy for an example domain derived from an existing deterministic demonstration domain.

We then apply Monte-Carlo Tree Search to POHTNs for generating plans and evaluate both the developed domain and the POHTN approach empirically.

Felix Richter, Susanne Biundo

Chapter 7. To Plan for the User Is to Plan with the User: Integrating User Interaction into the Planning Process

Abstract

Settings where systems and users work together to solve problems collaboratively are among the most challenging applications of Companion-Technology. So far we have seen how planning technology can be exploited to realize Companion-Systems that adapt flexibly to changes in the user’s situation and environment and provide detailed help for users to realize their goals. However, such systems lack the capability to generate their plans in cooperation with the user. In this chapter we go one step further and describe how to involve the user directly into the planning process. This enables users to integrate their wishes and preferences into plans and helps the system to produce individual plans, which in turn let the Companion-System gain acceptance and trust from the user.

Such a Companion-System must be able to manage diverse interactions with a human user. A so-called mixed-initiative planning system integrates several Companion-Technologies which are described in this chapter. For example, a—not yet final—plan, including its flaws and solutions, must be presented to the user to provide a basis for her or his decision. We describe how a dialog manager can be constructed such that it can handle all communication with a user. Naturally, the dialog manager and the planner must use coherent models. We show how an ontology can be exploited to achieve such models. Finally, we show how the causal information included in plans can be used to answer the questions a user might have about a plan.

The given capabilities of a system to integrate user decisions and to explain its own decisions to the user in an appropriate way are essential for systems that interact with human users.

Gregor Behnke, Florian Nielsen, Marvin Schiller, Denis Ponomaryov, Pascal Bercher, Birte Glimm, Wolfgang Minker, Susanne Biundo

Chapter 8. Neurobiological Fundamentals of Strategy Change: A Core Competence of Companion-Systems

Abstract

Companion-Systems interact with users via flexible, goal-directed dialogs. During dialogs both, user and Companion-System, can identify and communicate their goals iteratively. In that sense, they can be conceptualized as communication partners, equipped with a processing scheme producing actions as outputs in consequence of (1) inputs from the other communication partner and (2) internally represented goals. A quite general core competence of communication partners is the capability for strategy change, defined as the modification of action planning under the boundary condition of maintaining a constant goal. Interestingly, the biological fundamentals for this capability are largely unknown. Here we describe a research program that employs an animal model for strategy change to (1) investigate its underlying neuronal mechanisms and (2) describe these mechanisms in an algorithmic syntax, suitable for implementation in technical Companion-Systems. It is crucial for this research program that investigated scenarios be sufficiently complex to contain all relevant aspects of strategy change, but at the same time simple enough to allow for a detailed neurophysiological analysis only obtainable in animal models. To this end, two forms of strategy change are considered in detail: Strategy change caused by modified feature selection, and strategy change caused by modified action assignment.

Andreas L. Schulz, Marie L. Woldeit, Frank W. Ohl

Chapter 9. Assistive and Adaptive Dialog Management

Abstract

One of the most important challenges in the field of human-computer interaction is maintaining and enhancing the willingness of the user to interact with the technical system. This willingness to cooperate provides a solid basis which is required for a collaborative human-computer dialog. For the dialog management this means that a Companion-System adapts the course and content of human-computer dialogs to the user and assists during the interaction through individualized help and explanation. In this chapter we elucidate our dialog management approach, which provides user- and situation-adaptive dialogs, and our explanation management approach, which enables the system to provide assistance and clarification for the user during run-time.

Florian Nielsen, Wolfgang Minker

Chapter 10. Management of Multimodal User Interaction in Companion-Systems

Abstract

While interacting, human beings continuously adapt their way of communication to their surroundings and their communication partner. Although present context-aware ubiquitous systems gather a lot of information to maximize their functionality, they predominantly offer rather static ways to communicate. In order to fulfill the user’s communication needs and demands, ubiquitous sensors’ varied information could be used to dynamically adapt the user interface. Considering such an adaptive user interface management as a major and relevant component for a Companion-Technology, we also have to cope with emotional and dispositional user input as a source of implicit user requests and demands. In this chapter we demonstrate how multimodal fusion based on evidential reasoning and probabilistic fission with adaptive reasoning can act together to form a highly adaptive and model-driven interactive system component for multimodal interaction. The presented interaction management (IM) can handle uncertain or ambiguous data throughout the complete interaction cycle with a user. In addition, we present the IM’s architecture and its model-driven concept. Finally, we discuss its role within the framework of the other constituents of a Companion-Technology.

Felix Schüssel, Frank Honold, Nikola Bubalo, Michael Weber, Anke Huckauf

Chapter 11. Interaction with Adaptive and Ubiquitous User Interfaces

Abstract

Current user interfaces such as public displays, smartphones and tablets strive to provide a constant flow of information. Although they all can be regarded as a first step towards Mark Weiser’s vision of ubiquitous computing they are still not able to fully achieve the ubiquity and omnipresence Weiser envisioned. In order to achieve this goal these devices must be able to blend in with their environment and be constantly available. Since this scenario is technically challenging, researchers simulated this behavior by using projector-camera systems. This technology opens the possibility of investigating the interaction of users with always available and adaptive information interfaces. These are both important properties of a Companion-technology. Such a Companion system will be able to provide users with information how, where and when they are desired. In this chapter we describe in detail the design and development of three projector-camera systems(UbiBeam, SpiderLight and SmarTVision). Based on insights from prior user studies, we implemented these systems as a mobile, nomadic and home deployed projector-camera system which can transform every plain surface into an interactive user interface. Finally we discuss the future possibilities for Companion-systems in combination with a projector-camera system to enable fully adaptive and ubiquitous user interface.

Jan Gugenheimer, Christian Winkler, Dennis Wolf, Enrico Rukzio

Chapter 12. Interaction History in Adaptive Multimodal Interaction

Abstract

Modern Companion-Technologies provide multimodal and adaptive interaction possibilities. However, it is still unclear which user characteristics should be used in which manner to optimally support the interaction. An important aspect is that users themselves learn and adapt their behavior and preferences based on their own experiences. In other words, certain characteristics of user behavior are slowly but continuously changed and updated by the users themselves over multiple encounters with the Companion-Technology. Thus, a biological adaptive multimodal system observes and interacts with an electronic one, and vice versa. Consequently, such a user-centered interaction history is essential and should be integrated in the prediction of user behavior. Doing so enables the Companion to achieve more robust predictions of user behavior, which in turn leads to better fusion decisions and more efficient customization of the UI. We present the development of an experimental paradigm based on visual search tasks. The setup allows the induction of various user experiences as well as the testing of their effects on user behavior and preferences during multimodal interaction.

Nikola Bubalo, Felix Schüssel, Frank Honold, Michael Weber, Anke Huckauf

Chapter 13. LAST MINUTE: An Empirical Experiment in User-Companion Interaction and Its Evaluation

Abstract

The LAST MINUTE Corpus (LMC) is a unique resource for research on issues of Companion-technology. LMC not only comprises 57.5 h of multimodal recordings (audio, video, psycho-biological data) from interactions between users—133 subjects in sum, balanced in age and gender—and a WoZ-simulated speech-based interactive dialogue system. LMC also includes full verbatim transcripts of all these dialogues, sociodemographic and psychometric data of all subjects as well as material from 73 in-depth user interviews focusing the user’s individual experience of the interaction. In this chapter the experimental design and data collection of the LMC are shortly introduced. On this basis, exemplifying results from semantic analyses of the dialogue transcripts as well as from qualitative analyses of the interview material are presented. These illustrate LMC’s potential for investigations from numerous research perspectives.

Jörg Frommer, Dietmar Rösner, Rico Andrich, Rafael Friesen, Stephan Günther, Matthias Haase, Julia Krüger

Chapter 14. The LAST MINUTE Corpus as a Research Resource: From Signal Processing to Behavioral Analyses in User-Companion Interactions

Abstract

The LAST MINUTE Corpus (LMC) is one of the rare examples of a corpus with naturalistic human-computer interactions. It offers richly annotated data from N _total = 130 experiments in a number of modalities. In this paper we present results from various investigations with data from the LMC using several primary modalities, e.g. transcripts, audio, questionnaire data.

We showed that sociodemographics (age, gender) have an influence on the global dialog success. Furthermore, distinct behavior during the initial phase of the experiment can be used to predict global dialog success during problem solving. Also, the influence of interventions on the dialog course was evaluated.

Additionally, the importance of discourse particles as prosodic markers could be shown. Especially during critical dialog situations, the use of these markers is increasing. These markers are furthermore influenced by user characteristics.

Thus, to enable future Companion-Systems to react appropriately to the user, these systems have to observe and monitor acoustic and dialogic markers and have to take into account the user’s characteristics, such as age, gender and personality traits.

Dietmar Rösner, Jörg Frommer, Andreas Wendemuth, Thomas Bauer, Stephan Günther, Matthias Haase, Ingo Siegert

Chapter 15. Environment Adaption for Companion-Systems

Abstract

One of the key characteristics of a Companion-System is the adaptation of its functionality to the user’s preferences and the environment. On the one hand, a dynamic environment model facilitates the adaption of output modalities in human computer interaction (HCI) to the current situation. On the other hand, continuous tracking of users in the proximity of the system allows for resuming a previously interrupted interaction. Thus, an environment perception system based on a robust multi-object tracking algorithm is required to provide these functionalities. In typical Companion-System applications, persons in the proximity are closely spaced, which leads to statistical dependencies in their behavior. The multi-object Bayes filter allows for modeling these statistical dependencies by representing the multi-object state using random finite sets. Based on the social force model and the knowledge base of the companion system, an approach to modeling object interactions is presented. In this work, the interaction model is incorporated into the prediction step of the sequential Monte Carlo (SMC) of the multi-object Bayes filter. Further, an alternative implementation of the multi-object Bayes filter based on labeled random finite sets is outlined.

Stephan Reuter, Alexander Scheel, Thomas Geier, Klaus Dietmayer

Chapter 16. Non-intrusive Gesture Recognition in Real Companion Environments

Abstract

Automatic gesture recognition pushes Human-Computer Interaction (HCI) closer to human-human interaction. Although gesture recognition technologies have been successfully applied to real-world applications, there are still several problems that need to be addressed for wider application of HCI systems: Firstly, gesture-recognition systems require a robust tracking of relevant body parts, which is challenging, since the human body is capable of an enormous range of poses. Therefore, a pose estimation approach that identifies body parts based on geodetic distances is proposed. Further, the generation of synthetic data, which is essential for training and evaluation purposes, is presented. A second problem is that gestures are spatio-temporal patterns that can vary in shape, trajectory or duration, even for the same person. Static patterns are recognized using geometrical and statistical features which are invariant to translation, rotation and scaling. Moreover, stochastical models like Hidden Markov Models and Conditional Random Fields applied to quantized trajectories are employed to classify dynamic patterns. Lastly, a non-gesture model-based spotting approach is proposed that separates meaningful gestures from random hand movements (spotting).

Sebastian Handrich, Omer Rashid, Ayoub Al-Hamadi

Chapter 17. Analysis of Articulated Motion for Social Signal Processing

Abstract

Companion technologies aim at developing sustained long-term relationships by employing non-verbal communication (NVC) skills. Visual NVC signals can be conveyed over a variety of non-verbal channels, such as facial expressions, gestures, or spatio-temporal behavior. It remains a challenge to equip technical systems with human-like abilities to reliably and effortlessly detect and analyze such social signals. In this proposal, we focus our investigation on the modeling of visual mechanisms for the processing and analysis of human-articulated motion and posture information from spatially intermediate to remote distances. From a modeling perspective, we investigate how visual features and their integration over several stages in a processing hierarchy take part in the establishment of articulated motion representations. We build upon known structures and mechanisms in cortical networks of primates and emphasize how generic processing principles might realize the building blocks for such network-based distributed processing through learning. We demonstrate how feature representations in segregated pathways and their convergence lead to integrated form and motion representations using artificially generated articulated motion sequences.

Georg Layher, Michael Glodek, Heiko Neumann

Chapter 18. Automated Analysis of Head Pose, Facial Expression and Affect

Abstract

Automated analysis of facial expressions is a well-investigated research area in the field of computer vision, with impending applications such as human-computer interaction (HCI). The conducted work proposes new methods for the automated evaluation of facial expression in image sequences of color and depth data. In particular, we present the main components of our system, i.e. accurate estimation of the observed person’s head pose, followed by facial feature extraction and, third, by classification. Through the application of dimensional affect models, we overcome the use of strict categories, i.e. basic emotions, which are focused on by most state-of-the-art facial expression recognition techniques. This is of importance as in most HCI applications classical basic emotions are only occurring sparsely, and hence are often inadequate to guide the dialog with the user. To resolve this issue we suggest the mapping to the so-called “Circumplex model of affect”, which enables us to determine the current affective state of the user, which can then be used in the interaction. Especially, the output of the proposed machine vision-based recognition method gives insight to the observed person’s arousal and valence states. In this chapter, we give comprehensive information on the approach and experimental evaluation.

Robert Niese, Ayoub Al-Hamadi, Heiko Neumann

Chapter 19. Multimodal Affect Recognition in the Context of Human-Computer Interaction for Companion-Systems

Abstract

In general, humans interact with each other using multiple modalities. The main channels are speech, facial expressions, and gesture. But also bio-physiological data such as biopotentials can convey valuable information which can be used to interpret the communication in a dedicated way. A Companion-System can use these modalities to perform an efficient human-computer interaction (HCI). To do so, the multiple sources need to be analyzed and combined in technical systems. However, so far only few studies have been published dealing with the fusion of three or even more such modalities. This chapter addresses the necessary processing steps in the development of a multimodal system applying fusion approaches.

ATLAS and ikannotate are presented which are designed for the pre-analyzing of multimodal data streams and the labeling of relevant parts. ATLAS allows us to display raw data, extracted features and even outputs of pre-trained classifier modules. Further, the tool integrates annotation, transcription and an active learning module. Ikannotate can be directly used for transcription and guided step-wise emotional annotation of multimodal data. The tool includes the three mainly used annotation paradigms, namely the basic emotions, the Geneva emotion wheel and the self-assessment manikins (SAMs). Furthermore, annotators using ikannotate can assign an uncertainty to samples.

Classifier architectures need to realize a fusion system in which the multiple modalities are combined. A large number of machine learning approaches were evaluated, such as data, feature, score and decision-level fusion schemes, but also temporal fusion architectures and partially supervised learning.

The proposed methods are evaluated on either multimodal benchmark corpora or on the datasets of the Transregional Collaborative Research Centre SFB/TRR 62, i.e. Last Minute Corpus and the EmoRec Dataset. Furthermore, we present results which were achieved in international challenges.

Friedhelm Schwenker, Ronald Böck, Martin Schels, Sascha Meudt, Ingo Siegert, Michael Glodek, Markus Kächele, Miriam Schmidt-Wack, Patrick Thiam, Andreas Wendemuth, Gerald Krell

Chapter 20. Emotion Recognition from Speech

Abstract

Spoken language is one of the main interaction patterns in human-human as well as in natural, companion-like human-machine interactions. Speech conveys content, but also emotions and interaction patterns determining the nature and quality of the user’s relationship to his counterpart. Hence, we consider emotion recognition from speech in the wider sense of application in Companion-systems. This requires a dedicated annotation process to label emotions and to describe their temporal evolution in view of a proper regulation and control of a system’s reaction. This problem is peculiar for naturalistic interactions, where the emotional labels are no longer a priori given. This calls for generating and measuring of a reliable ground truth, where the measurement is closely related to the usage of appropriate emotional features and classification techniques. Further, acted and naturalistic spoken data has to be available in operational form (corpora) for the development of emotion classification; we address the difficulties arising from the variety of these data sources. Speaker clustering and speaker adaptation will as well improve the emotional modeling. Additionally, a combination of the acoustical affective evaluation and the interpretation of non-verbal interaction patterns will lead to a better understanding of and reaction to user-specific emotional behavior.

Andreas Wendemuth, Bogdan Vlasenko, Ingo Siegert, Ronald Böck, Friedhelm Schwenker, Günther Palm

Chapter 21. Modeling Emotions in Simulated Computer-Mediated Human-Human Interactions in a Virtual Game Environment

Abstract

Emotions form a major part of humans’ day-to-day lives, especially in the areas of communication and interaction with others. They modify our gesture or facial expression and therefore serve as an additional communication channel. Furthermore, they have an impact on decision-making. This has two possible implications for computer science in the field of human-computer-interaction. First, computers should be able to adequately recognize and model human emotions if they genuinely want to help users in applied fields of human-human interactions. Second, a reliable and valid computer model of users’ emotions is the basis of effective implementations for human-computer interaction, with the computer thus being able to adapt to users’ emotions flexibly in any given application.

From an empirical point of view, though, computerized recognition of human emotions still lacks substantial reliability and validity. In our opinion there are two main reasons for this shortcoming. First, triggers of emotional responses, i.e. eliciting situations, are typically complex in nature and thus difficult to predict or even assess once apparent. Second, the emotional response itself is a complex reaction involving subjects’ individual learning history, appraisal, preparedness, bodily reactions, and so forth. Both factors make it difficult for any algorithm to recognize real-life emotions.

In a venture to approach this problem, the main goal of our study is to test an implementation of a computer model (COMPLEX) that predicts user emotions in a simulated human-human interaction. The prediction is supported by an elaborate appraisal model of emotions and the assessment of user bodily reactions, facial expression and speech. This article will give an overview of the theoretical background, the practical implementation of our new approach and first results of an empirical validation.

Andreas Scheck, Holger Hoffmann, Harald C. Traue, Henrik Kessler

Chapter 22. Companion-Systems: A Reference Architecture

Abstract

Companion-Technology for cognitive technical systems consists of a multitude of components that implement different properties. A primary point is the architecture which is responsible for the interoperability of all components. It defines the capabilities of the systems crucially. For research concerning the requirements and effects of the architecture, several demonstration scenarios were developed. Each of these demonstration scenarios focuses on some aspects of a Companion-System. For the implementation a middleware concept was used, having the capability to realize the major part of the Companion-Systems. Currently the system architecture takes up only a minor property in projects which are working on related research topics. For the description of an architecture representing the major part of possible Companion-Systems, the demonstration scenarios are studied with regard to their system structure and the constituting components. A monolithic architecture enables a simple system design and fast direct connections between the components, such as: sensors with their processing and fusion components, knowledge bases, planning components, dialog systems and interaction components. Herein, only a limited number of possible Companion-Systems can be represented. In a principled approach, a dynamic architecture, capable of including new components during run time, is able to represent almost all Companion-Systems. Furthermore, an approach for enhancing the architecture is introduced.

Thilo Hörnle, Michael Tornow, Frank Honold, Reinhard Schwegler, Ralph Heinemann, Susanne Biundo, Andreas Wendemuth

Chapter 23. Investigation of an Augmented Reality-Based Machine Operator Assistance-System

Abstract

In this work we propose three applications towards an augmented reality-based machine operator assistance system. The application context is worker training in motor vehicle production. The assistance system visualizes information relevant to any particular procedure directly at the workplace. Mobile display devices in combination with augmented reality (AR) technologies present situational information. Head-mounted displays (HMD) can be used in industrial environments when workers need to have both hands free. Such systems augment the user’s field of view with visual information relevant to a particular job. The potentials of HMDs are well known and their capabilities have been demonstrated in different application scenarios. Nonetheless, many systems are not user-friendly and may lead to rejection or prejudice among users. The need for research on user-related aspects as well as methods of intuitive user interaction arose early but has not been met until now. Therefore, a robust prototypical system was developed, modified and validated. We present image-based methods for robust recognition of static and dynamic hand gestures in real time. These methods are used for intuitive interaction with the mobile assistance system. The selection of gestures (e.g., static vs. dynamic) and devices is based on psychological findings and ensured by experimental studies.

Frerk Saxen, Anne Köpsel, Simon Adler, Rüdiger Mecke, Ayoub Al-Hamadi, Johannes Tümler, Anke Huckauf

Chapter 24. Advanced User Assistance for Setting Up a Home Theater

Abstract

In many situations of daily life, such as in educational, work-related, or social contexts, one can observe an increasing demand for intelligent assistance systems. In this chapter, we show how such assistance can be provided in a wide range of application scenarios—based on the integration of user-centered planning with advanced dialog and interaction management capabilities. Our approach is demonstrated by a system that assists a user in the task of setting up a complex home theater. The theater consists of several hi-fi devices that need to be connected with each other using the available cables and adapters. In particular for technically inexperienced users, the task is quite challenging due to the high number of different ports of the devices and because the used cables might not be known to the user. Support is provided by presenting a detailed sequence of instructions that solves the task.

Pascal Bercher, Felix Richter, Thilo Hörnle, Thomas Geier, Daniel Höller, Gregor Behnke, Florian Nielsen, Frank Honold, Felix Schüssel, Stephan Reuter, Wolfgang Minker, Michael Weber, Klaus Dietmayer, Susanne Biundo

Chapter 25. Multi-modal Information Processing inCompanion-Systems: A Ticket Purchase System

Abstract

We demonstrate a successful multimodal dynamic human-computer interaction (HCI) in which the system adapts to the current situation and the user’s state is provided using the scenario of purchasing a train ticket. This scenario demonstrates that Companion Systems are facing the challenge of analyzing and interpreting explicit and implicit observations obtained from sensors under changing environmental conditions. In a dedicated experimental setup, a wide range of sensors was used to capture the situative context and the user, comprising video and audio capturing devices, laser scanners, a touch screen, and a depth sensor. Explicit signals describe a user’s direct interaction with the system, such as interaction gestures, speech and touch input. Implicit signals are not directly addressed to the system; they comprise the user’s situative context, his or her gesture, speech, body pose, facial expressions and prosody. Both multimodally fused explicit signals and interpreted information from implicit signals steer the application component, which was kept deliberately robust. The application offers stepwise dialogs gathering the most relevant information for purchasing a train ticket, where the dialog steps are sensitive and adaptable within the processing time to the interpreted signals and data. We further highlight the system’s potential for a fast-track ticket purchase when several pieces of information indicate a hurried user.

A video of the complete scenario in German language is available at: http://​www.​uni-ulm.​de/​en/​in/​sfb-transregio-62/​pr-and-press/​videos.​html

Ingo Siegert, Felix Schüssel, Miriam Schmidt, Stephan Reuter, Sascha Meudt, Georg Layher, Gerald Krell, Thilo Hörnle, Sebastian Handrich, Ayoub Al-Hamadi, Klaus Dietmayer, Heiko Neumann, Günther Palm, Friedhelm Schwenker, Andreas Wendemuth