Designing Interactive Speech Systems

When we use a computer system to perform a certain task, the computer system acts both as a tool and as a partner in communication. It would be stretching the sense of the term “communication” beyond reasonable limits to say that one communicates with a spade when using it. The computer is different to the spade in important ways. The user must input information in some form in order to make the system execute. Similarly, to inform the user of its state, processes and their results, the computer must output information to the user. The information which is being exchanged between user and system during task performance can be represented in different forms, or modalities, using a variety of different input/output devices. For a wide range of tasks, the system can achieve task adequacy as a tool by exchanging information with its users in ways that are completely different to those of human-human communication, such as through a keyboard and mouse as input devices, the screen as an output device and typed command notation as the key modality for representing input information. With or without the inclusion of typed command input notation, this form of interaction is called the graphical user interface (GUI) paradigm.

In the following chapters we describe the development and evaluation of interaction model and dialogue component aspects of advanced interactive speech systems in accordance with the idea of a rationalized development process presented in Section 1.2. Ideally, development and evaluation would be exhaustively presented on the basis of a consolidated and transparent version of a theory of spoken interaction such as the one presented in Chapter 2. For the time being, we can offer only a less comprehensive and more fragmented view. Advanced interactive speech system development has so far taken place mainly in research projects and a complete best practice methodology which can support, improve, make more efficient and help standardize the development and evaluation of advanced interactive speech systems is still far away. The methodology should specialize software engineering best practice to the particular purposes of advanced interactive speech system engineering by specifying in detail the methods (procedures, guidelines, heuristics), concepts and tools to be used in developing and evaluating advanced interactive speech systems as well as providing guidelines on when and how to use each method, set of concepts or tool.

Interaction model analysis and design is a core issue in the development of advanced interactive speech systems (cf. Section 5.4). It starts at a low speed in the survey phase, is the focal point in the analysis and design phase, and continues during subsequent phases alternating with evaluation (Figure 3.1). Revisions always require analysis, and some form of re-design may be needed as late as in the acceptance test phase. In the analysis and design phase the aim is to develop the interaction model to such a level of formal detail that it can serve as a basis for implementation. The design specification initiated during the survey (Section 3.2) serves as a basis for establishing an interaction model for the system to be developed. The design specification is iteratively extended in the analysis and design phase because many new questions typically arise during interaction model design. These questions must be addressed, new design decisions made and conflicts arising from design decision making resolved, often through trade-offs among conflicting constraints. Results in terms of new design goals, constraints and modifications are added to the design specification and the development of the interaction model is continued on this evolving basis.

When the first interaction model has been designed (cf. Section 4.1) interactive speech system development may either go through a phase of Wizard of Oz (WOZ) simulations as will be described in this chapter, or go straight to implementation (Chapter 6) following the implement-test-revise approach (Figure 3.1). Today’s research on interaction model design for advanced interactive speech systems often includes the WOZ experimental prototyping method. In WOZ a human (the wizard) simulates whole or part of the interaction model of the system to be developed, carrying out spoken interactions with users who are made to believe that they are interacting with a real system. WOZ is a relatively costly development method because: (1) the wizard needs a significant amount of training and support; (2) involving experimental subjects, WOZ experiments require careful planning and preparation and take time to run; and (3) experimental results have to be transcribed and analysed, which takes time and requires skill to benefit further system development. On the other hand, by producing data on the interaction between a (fully or partially) simulated system and its users, WOZ provides the basis for early tests of the system and its feasibility, as well as of the coverage and adequacy of requirements prior to implementation. The use of WOZ has so far been justified through the comparatively higher cost of having to revise an already implemented interactive speech system whose interaction model turned out to be seriously flawed, or of having to discard a system which users will not use. As recognition and parsing techniques continue to improve and the body of standard software grows, implement-test-revise methods are likely to gain ground in the design of advanced interactive speech systems.

The implementational issues of interactive speech systems, neither in principle nor in practice, differ much from those of any other software system. Nevertheless, this chapter illustrates some important implementational issues raised in particular by the nature of the dialogue control layer.

A corpus is a collection or body of linguistic data, organized in a manner that will facilitate investigation of, and reference to, the data. By today’s standards, corpora are in machine-readable form. Dictionary publishers maintain corpora of citations and word uses, and researchers collect huge (millions of words) corpora of texts of all kinds for many different purposes. Corpus linguistics is both a well-established discipline and an active research area (McEnery and Wilson, 1996). A growing subdiscipline focuses on spoken language (Leech et al, 1995). Spoken corpora are collections of usually transcribed spoken language such as monologues, interviews, conversations or task-oriented dialogues. This chapter focuses on the transcription, markup and coding of spoken dialogue corpora, emphasizing the representations, procedures and tools that are relevant to the design of interactive speech systems.

System evaluation is a highly important discipline which is tightly interwoven with system development. Evaluation is constantly needed throughout development to measure progress towards the goals that the system has to meet. Interactive speech system evaluation today is as much of an art and craft as it is an exact science with established standards and procedures of good engineering practice. In particular, little is still known about interaction model evaluation, including evaluation of dialogue components and integrated interactive speech systems. There is not even consensus on terminology. Following Hirschmann and Thompson (1996) (see also Gibbon et al, 1997) we will distinguish between three types of evaluation which, although they are clearly not orthogonal, seem to cover the relevant aspects of evaluation, and subsume the scopes of other commonly used terms and distinctions. Each of these three types of evaluation may be used at any stage of system development:

The advanced interactive speech system technologies that we have been discussing up to this point are more or less at the level of the Danish Dialogue System. It is an interesting, even if somewhat vague, question of how far it will be possible to advance towards fully natural interactive speech systems on the basis of these technologies. What are the issues ahead that will require significant changes of approach? In this chapter, we discuss two such issues. The first issue we have chosen to term the “heterogeneous task” which appears to demand a significant increase in systems’ language processing skills and in the theoretical underpinnings of these skills (Section 9.2). The second issue is multimodality. In the future, spoken human-system interaction no doubt will become much more similar to natural human-human spoken interaction than is currently the case. However, as long as the interaction is purely spoken, and hence unimodal, it remains far from the ideal of fully natural human-human communication presented in Section 1.1. Section 9.3 presents a range of multimodal systems which actually or potentially incorporate advanced interactive speech technologies, and discusses ways to develop a systematic understanding of such Advanced Multimodal Interactive Speech Systems (AMISSs).