Conceptual frameworks for multimodal social signal processing

This special issue is about a research area which is developing rapidly. Pentland [4] gave it a name which has become widely used, ‘Social Signal Processing’ (SSP for short), and his phrase provides the title of a European project, SSPnet, which has a brief to consolidate the area.¹ The challenge that Pentland highlighted was understanding the nonlinguistic signals that serve as the basis for “subconscious discussions between humans about relationships, resources, risks, and rewards”. He identified it as an area where computational research had made interesting progress, and could usefully make more.

If effective progress is to be made, one of the requirements is to develop some consensus on a variety of issues that are basic to the area—obviously the topics to be covered, but also terminology, the literature that people in the field are expected to know, the simplifications that are considered acceptable, and so on. That kind of statement might look routine, but in the context of technology dealing with human thoughts and feelings, there is always a grim precedent to consider. Technologies that were supposed to detect lying fell short of any reasonable standards of reliability, and yet they convinced both the public and (for a while) the law [1, 3]. It is not a mistake that should be recycled. More than anything else, that example defines the problem that faces technology moving onto grounds that have traditionally belonged to human judgment: sophisticated technology plus naivete about human beings is a recipe for disaster.

Efforts since Pentland’s paper have made it clear that it is not easy to achieve well-grounded consensus for the new area. Some of the reasons are superficial, such as newcomers assuming that the name defines the field rather than being a useful label for an existing (and expanding) body of work. Others are deeper, such as the fact that there are notoriously intractable divisions in the existing literature on social phenomena (e.g. [2]). Those divisions reflect the uncomfortable reality that social phenomena defy any single, coherent analysis, and it would be naïve to expect that the new field could transcend them. What it can do is to find a way of living with them.

The aim of this special issue is to reflect the kinds of conceptual framework that are emerging in the new field. It accepts that part and parcel of the task is to acknowledge tensions. Because the area is clearly difficult, it takes a twofold approach. The traditional, and much weightier strand, consists of papers that address important parts of the conceptual framework, and that to a greater or lesser extent reflect specific viewpoints. The less conventional strand consists of a statement developed within SSPnet, which forms part of this editorial. It has become known as the Declaration of Belfast.

The papers reflect quite diverse positions. There is perhaps a default position, which is shared (to different extents) by three of the papers. But even within those, there are differences of emphasis; and beyond them, there are quite different perspectives to consider.

In the centre, there are striking overlaps between Brunet and Cowie and Scherer et al. Both papers understand the challenge in terms of states to be detected, and signals that carry information about them. The states may be states of an individual, or of a set of people who are interacting, or (Brunet and Cowie say) of an organisation. Both stress that the signals are not neatly packaged: a major part of the challenge is to pull the relevant information from a multimodal flux extended over time.

Scherer et al. develop that general framework in one direction. They offer a succinct list of subject states to be identified (using terms like interested, surprised, stressed, accepting, etc.), and sources of information about them (involving talk style, revealing events, the focus of the speaker, and the dialog role)—these, for them, provide the social signals. They look in some detail at technologies that may be relevant , and develop their ideas through a detailed study of particular data.

Brunet and Cowie move in another direction from the common ground. Their emphasis is on the psychological complexities that have to be reckoned with. They highlight the enormous range of states and signals that may be relevant to social interactions; the different kinds of control that humans may exert over the production of signals, and the different kinds of inference that they may employ; and also the contextual and cultural issues that bear on the generation & interpretation of social signals. They do not argue that systems should reproduce the complexities of human processing, but that system developers should be alert to it.

In both Scherer et al. and Brunet and Cowie, the states most often discussed are socially significant states of an individual. However, Brunet and Cowie acknowledge in principle that some significant states are intrinsically concerned with relationships between interactants. Janssen looks in depth at a key kind of relational state, which is empathy. He argues that in fact, empathy needs to be analysed on different levels—not only cognitive empathy, which has dominated previous research, but also emotional convergence and empathic responding. That emphasis means that processing has to be concerned with relationships between signals recorded from different individuals, which in turn raises challenges for data capture and analysis.

A sharply different approach appears in the paper by D’Errico, Poggi and Vincenze. In the papers considered so far, non-verbal signals are generally thought of as conveying information which is qualitatively different from most of the information conveyed by verbal signals—broadly speaking, about global states of speakers and their relationships. D’Errico et al. reflect a tradition which considers non-verbal communication as ‘body language’ in a very literal sense, consisting of communicative acts whose meaning could be expressed in words, but happens not to be [5]. For example, slow headshakes are taken to convey “I can’t believe that he is so hopelessly stupid”. The more general categories onto which they map non-verbal behaviours are drawn from speech act theory and rhetoric rather than psychology. They use studies of political discourse to show how various moves can be used in combination with language to discredit opponents, and introduce a system for coding them.

Mehu et al. address the issue of divergence itself rather than presenting a position of their own. They emphasise that the divergence has roots in the material on which the emerging discipline has to draw, and they stress the need for a sophisticated attitude to that material. They address the issue at two main levels, vocabulary and overarching concepts. At the level of vocabulary, they set out an extensive list of key terms, and describe the different meanings that the terms carry in different disciplines. At the level of overarching concepts, they discuss different conceptions of information and meaning in general, and then of social signals in particular. They advocate a pluralistic response, and regard it as “the responsibility of each SSP scholar to get familiar with the different approaches”.

It is right and proper that the papers in the special issue should reflect different approaches. However, it is also important find ways of defining common ground. That is what SSPnet set out to do in the Declaration of Belfast, which is included here by permission of the SSPnet project members.