Towards a conceptual framework of research on social signal processing

Social Signal Processing (SSP) as an emerging research area can draw on material from many disciplines, but it needs effective ways to organise the material. We propose a framework that integrates concepts, drawn primarily from psychology, but with input from other disciplines, in a way that indicates how they relate to SSP. We identify seven core constructs: state; indicators that convey info about it; process by which indicators are generated; types of inference involved in identifying states from indicators; perceptual issues, including recovery of indicators from flux of activity and accuracy of impressions about the state; the role of macro-context; and the different levels at which analysis may be couched (the individual, the dyad, the group, the organisation). These may or may not be reflected in the structure of an SSP system, but the natural subtlety and context-sensitivity of human communication makes it important that people designing systems should consider how they relate to its task, and decide how best to take account of them. The analysis works from simple models which consider only states and indicators to models which embrace all seven constructs. At each stage it points to the literatures which discuss the relevant issues. It is fully acknowledged that attempting such a synthesis raises many problems (not least of terminology), and that alternative frameworks deal at least as well with subsets of the issues. It remains to be seen whether they could be extended to cover a comparable range.