Quantifying Quality Aspects of Multimodal Interactive Systems

For many years the prototypical interactive system has been the personal computer (PC) that offers a screen, a keyboard and a mouse as an interface for the human–machine interaction. The human face-to-face communication on the other hand involves spoken language, gestures and facial expressions on the transmitting side, and listening as well as lip- and expression reading on the perceiving side. With the advances in computation, and the consequent pervasion of computer systems in our daily life, an interest has arisen in designing more ‘natural’ interfaces to the computer. An obvious approach is to develop interactive systems that mirror human–human interaction (Krämer and Bente 2002). Such interfaces should recognize spoken language, gestures and facial expressions. Ideally, the system would also respond in a similar way, making a body or at least a face necessary. This approach has led, for example, to the developement of so-called Embodied Conversational Agents (ECA), aiming to eliminate the need of learning special strategies for human–computer interaction (Xiao et al. 2002).

As seen in the previous chapter, multimodal systems are well-established—at least in the research community studying HCI. But during the development process and—at the latest–once the system is built, methods for quantitative assessment are needed. In fact, as systems are usually meant to fullfill certain needs of, assist or even replace the human, the user’s perspective needs to be considered from the very start of conceiving a system. This approach of taking into account the potential user group and its characteristics has culminated in the user-centered or participatory design approach (Schuler and Namioka 1993), namely involving the user in the design process. Some accepted methods for user-centered design, such as the cognitive walkthrough, are discussed below, please refer to Vredenburg et al. (2002) for a survey of user-centered design practice.

As a precondition for the work presented a multimodal system is needed to enable the realization of extended studies to address the research questions motivated in Chap. 1. It has become apparent in the previous chapters that quite a number of multimodal systems have been designed, mostly for a similar purpose and only recently for consumers. The nature of the studies conducted did not allow the use of an off-the-shelf product, as these systems usually only permit so-called black box for an explanation of black box and glass box evaluations. Thus, an existing system accessible at the home institution was extended. In this chapter the domain as well as the different input and output modalities chosen are motivated in the light of existing research, state of the art and the research questions addressed. Furthermore, the different versions of the multimodal system used for the underlying studies are described.

After having laid the theoretical and practical foundations—in terms of the multimodal system INSPIRE_Me described in Chap. 3—it is now time to address the research questions that are the core of this book.

In Chap. 3.4 it has been stated that ‘often, multimodal systems are multimodal in input only, at best presenting multimedia output’. At the same time there is a rising interest in real multimodal output. While there are several possible combinations of output modalities, in this work, two types are discussed in detail: output via a talking head (see Chap. 3.4.1) and the combination of the talking head with graphics.

In correspondence to the previous chapter on the quantification of output quality in this chapter the assessment of input quality will be discussed with a focus on spoken, touch and gestural input.

After discussing the quantification of multimodal interaction, input quality and output quality, in this chapter the results are consolidated to give an idea of the full picture. Based on the taxonomy introduced in Chap. 2 an attempt is made to quantify the quality aspects of an entire multimodal interactive system. To this end the chapter will first give a short introduction by taking up a few concepts described in Chap. 2. The research question is motivated shortly, the methodology described and results illustrated. After discussing these in the light of findings presented in the previous chapters conclusions are drawn and a short summary is given. The content of this chapter has been partially published in Kühnel et al. (2011a).

To conclude, the results presented in the previous chapters are summarized. Furthermore, potential criticism is addressed and future work outlined.