From Brows to Trust

The construction of embodied conversational agents is an ambitious, complex, and essentially interdisciplinary process. This is inevitable given the depth, sophistication, and many modalities of the products we seek to create. Other chapters in this book address methods for evaluating ECAs as artefacts or according to their usability. In this chapter, we offer a complementary perspective: grounding the evaluation of ECAs in the context of the different disciplines that have merged to create the research community constructing them.

Different research areas have different goals and criteria for success, and without understanding what these are and how they relate, we cannot intelligently recognise what contributions other groups are making, a necessary requirement for integrating work done on one aspect of ECAs with work on another. Our goal is to help our community ultimately to create the high-quality interdisciplinary products necessary for this discipline to mature.

One would like to rely on design guidelines for embodied conversational agents (ECAs), grounded on evaluation studies. How to define the physical and mental characteristics of an ECA, optimal for an envisioned application? What will be the added value of using an ECA? Although there have been studies addressing such issues, we are still far from getting a complete picture. This is not only due to the still relatively little experience with applications of ECAs, but also to the diversity in terms and experimental settings used. The lack of a common, established framework makes it di cult to compare ECAs, interpret evaluation results and judge their scope and relevance. In this chapter we propose a common taxonomy of the relevant design and evaluation aspects of ECAs. We refer to recent works to elicit evaluation concepts and discuss measurement issues.

The objective of this chapter is to identify the common knowledge and practice in research methodology and to apply it to the field of software evaluation, especially of embodied conversational agents. Relevant issues discussed are: how to formulate a good research question, what research strategy to use, which data collection methods are most appropriate and how to select the right participants. Reliability and validity of the data sets are dealt with and finally the chapter concludes with a list of guidelines that one should keep in mind when setting up and conducting empirical evaluation studies on embodied conversational agents.

An increasing number of dialogue systems are deployed to provide public services in our everyday lives. They are becoming more service-minded and several of them provide different channels for interaction. The rationale is to make automatic services available in new environments and more attractive to use. From a developer perspective, this affects the complexity of the requirements elicitation activity, as new combinations and variations in end-user interaction need to be considered. The aim of our investigation is to propose new parameters and metrics to evaluate multimodal dialogue systems endowed with embodied conversational agents (ECAs). These new metrics focus on the users, rather than on the system. Our assumption is that the intentional use of prosodic variation and the production of communicative non-verbal behaviour by users can give an indication of their attitude towards the system and might also help to evaluate the users’ overall experience of the interaction. To test our hypothesis we carried out analyses on di erent Swedish corpora of interactions between users and multimodal dialogue systems. We analysed the prosodic variation in the way the users ended their interactions with the system and we observed the production of non-verbal communicative expressions by users. Our study supports the idea that the observation of users’ prosodic variation and production of communicative non-verbal behaviour during the interaction with dialogue systems could be used as an indication of whether or not the users are satisfied with the system performance.

One obvious challenge for affective interfaces is to find ways of checking whether the expressed emotions are understood by users, and whether the system can interpret user emotions correctly. Even more challenging is whether the overall usage scenarios are achieving their purpose of being e.g. engaging, fun, believable, or creating a relationship with the user, and how much of this can be attributed to the emotion modeling and expression. We propose a two-tiered design and evaluation model. We exemplify this model through studies of three di erent a ective interfaces: the Agneta & Frida system, the Influencing Machine, and SenToy & FantasyA.

Traditionally, an optimal embodied conversational agent (ECA) has the same capabilities and appearance as an actual person. This chapter proposes a ‘user as assessor’ approach to evaluating ECAs that focuses on how ECAs manifest human capabilities independent of actual capabilities that an ECA may possess. Literatures on humans as producers of behavior and humans as interpreters of behavior are lever-aged to draw implications for how ECAs should behave to seem most realistic to their human assessors. To illustrate the approach, we answer the question, ”what will convince a user that an ECA is paying attention to him or her, whether the ECA truly is paying attention or not?” ‘Apparent attention’ is conceptualized in terms of two basic dimensions — selectivity and breadth — and their indicators and impacts. Using the proposed approach, the chapter provides guidelines for how agents, conversational agents, and ECAs can effectively exhibit attention.

In a seminal paper, Ekman (1979) remarks that brows can play an accentuation role (e.g., to signal focus). However, the literature about eyebrows is inconclusive about their exact role and as a consequence there is no agreement among developers of embodied conversational agents about their precise timing and placement. In addition, it is unclear whether eyebrow movements perform the same role in different languages. In this chapter, an analysis-by-synthesis technique is used to find out what the role of eyebrow movements is for the perception of focus and to see whether this role is the same across different languages. Three experiments are performed, both for Dutch and Italian, investigating where subjects prefer eyebrow movements, whether brows influence the perceived prominence of words and whether they are used in a functional way when subjects interpret utterances. The results for Dutch and Italian are indeed different, but it is argued that these differences can be reduced to prosodic differences between the two languages. The advantages and potential limitations of studies via analysis-by-synthesis are discussed, and an approach to compensate for the limitations is offered.

Individuality of Embodied Conversational Agents (ECAs) may depend on both the look of the agent and the way it combines different modalities such as speech and gesture. In this chapter, we describe a study in which male and female users had to listen to three short technical presentations made by ECAs. Three multimodal strategies of ECAs for using arm gestures with speech were compared: redundancy, complementarity, and speech-specialization. These strategies were randomly attributed to different-looking 2D ECAs, in order to test independently the effects of multimodal strategy and ECA’s appearance. The variables we examined were subjective impressions and recall performance. Multimodal strategies proved to influence subjective ratings of quality of explanation, in particular for male users. On the other hand, appearance affected likeability, but also recall performance. These results stress the importance of both multimodal strategy and appearance to ensure pleasantness and effectiveness of presentation ECAs.

A strong debate has ensued in the computing community about whether Embodied Conversational Agents (ECAs) are beneficial and whether we should pursue this direction in interface design. Proponents cite the naturalness and power of ECAs as strengths, and detractors feel that ECAs disempower, mislead, and confuse users. As this debate rages on, relatively little systematic empirical evaluation on ECAs is actually being performed, and the results from this research have been contradictory or equivocal. We propose a framework for evaluating ECAs that can systematize the research. The framework emphasizes features of the agent, the user, and the task the user is performing. Our goal is to be able to make informed, scientific judgments about the utility of ECAs in user interfaces. If intelligent agents can be built, are there tasks or applications for which an ECA is appropriate? Are there characteristics (in appearance, in personality, etc.) the ECA should have? What types of users will be more productive and happy by interacting with an ECA? Our initial experiment within this framework manipulated the ECA’s appearance (realistic human versus iconic object) and the objectivity of the user’s task (editing a document versus deciding what to pack on a trip). We found that the perception of the ECA was strongly influenced by the task while features of the ECA that we manipulated had littlee ect.

Conversational interfaces that incorporate animated characters potentially are well suited for educational software, since they can engage children as active learners and support question asking skills. In the present research, a simulation study was conducted in which twenty-four 7-to-10-year-old children used speech and pen input to converse directly with animated fish as they learned about marine biology. The animated fish responded with TTS voices crafted to sound either extroverted or introverted in accordance with the speech signal literature. During these interactions, children became highly engaged, asking an average of 152 questions during a 45-minute session. Self-report measures further confirmed that children liked “talking to the animals and that the TTS and interface were easy to understand and use. The auditory embodiment of animated characters as TTS output also had a significant selective impact on children’s engagement in asking science questions. Specifically, children asked +16% more science questions when conversing with animated characters embodying an extrovert TTS voice that resembled the speech of a master teacher (e.g., higher volume and pitch, wider pitch range), rather than an introvert TTS voice, although no differential impact was found on social questions. These findings reveal that conversational interfaces can be designed that effectively stimulate children during learning activities, thereby supporting the goals of next-generation educational software.

This chapter describes an experimental approach to the evaluation of embodied conversational agents (ECAs) within eCommerce contexts and exemplifies the approach with three case studies. Results are presented from three experiments into the usability of eCommerce applications employing 3D ECAs within the domain of eRetail and eBanking.

The findings described here confirm user preferences for applications in which the agent acts as a conversational partner compared with a non-visual telephone application using speech recognition. Further, data in this chapter confirm the positive role of ECAs in interfaces and the benefits to that role of adding other modalities such as text output.

In this chapter we describe a commercial platform for the development of net environments, virtual spaces inhabited by avatars which have been created and are subsequently visited and instructed by users via the Internet. The platform allows extensive data collection. Which data are collected and the scope of analysis will be explained on a theoretical level and by examples from data sets gathered from different applications.