Top

Journal on Multimodal User Interfaces

Published in:

01-11-2012 | Original Paper

Conceptual analysis of social signals: the importance of clarifying terminology

Authors: Marc Mehu, Francesca D’Errico, Dirk Heylen

Published in: Journal on Multimodal User Interfaces | Issue 3-4/2012

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

As a burgeoning field, Social Signal Processing (SSP) needs a solid grounding in the disciplines that have developed important concepts in the study of communication. However, the number and diversity of terms developed in linguistics, psychology, and the behavioural sciences may seem confusing for scholars who are not versed in the subtleties of conceptual analysis and theoretical developments. Indeed, different disciplines sometimes use the same term to mean different things or, conversely, use different terms to mean the same thing. The goals of this article are to present an overview of the different concepts developed in the various disciplines that studied animal and human communication, and to understand the differences and commonalities between concepts emerging from these disciplines. We conclude that such an understanding will greatly improve the efficiency of pluridisciplinary research projects, for the advancement of SSP requires that we look at the complexity of communication from different angles.

previous article Discrediting signals. A model of social evaluation to study discrediting moves in political debates

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Note that within the field of ethology, some authors consider signals as information carrier (e.g. [14, 103]); while other authors see signals as mostly influential [51, 65, 79].

Allwood J (1976) Linguistic communication as action and cooperation. Gothenburg monographs in linguistics, vol 2. University of Göteborg, Dept of Linguistics

Allwood J (1977) A critical look at speech act theory. In: Dahl O (ed) Logic, pragmatics, and grammar. Studentlitteatur, Lund, pp 53–99

Allwood J (2001) Cooperative multimodal communication, vol 2155. Springer, Berlin, Heidelberg

Ambady N, Rosenthal R (1992) Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychol Bull 111(2):256–274

Argyle M (1988) Bodily communication. Routledge, London

Ay N, Flack JC, Krakauer DC (2007) Robustness and complexity co-constructed in multimodal signalling networks. Philos Trans R Soc B 362:441–447

Austin JL (1962) How to do things with words. Oxford University Press, Oxford

Bachorowski J-A, Owren MJ (2001) Not all laughs are alike: voiced but not unvoiced laughter readily elicits positive affect. Psychol Sci 12(3):252–257

Bänziger T, Scherer KR (2007) Using actor portrayals to systematically study multimodal emotion expression: the GEMEP corpus. In: ACII 2007. Lecture notes in computer science, vol 4738. Springer, Berlin, Heidelberg, pp 476–487

10.

Barrett K (1996) Infants’ use of conflicting emotion signals. Cogn Emot 10(2):113–136

11.

Beer CG (1977) What is a display? Am Zool 17(1):155–165 MathSciNet

12.

Berlo DK (1960) The process of communication: an introduction to theory and practice. Holt, Rinehart and Winston, New York

13.

Bliege-Bird RL, Smith EA (2005) Signaling theory, strategic interaction, and symbolic capital. Curr Anthropol 46(2):221–248

14.

Bradbury JW, Vehrencamp SL (1998) Principles of animal communication. Sinauer Associates, Sunderland

15.

Brunswik E (1956) Perception and the representative design of psychological experiments. University of California Press, Berkeley

16.

Bühler K (1934) Sprachtheorie: Die Darstellungsfunktion der Sprache. Fischer, Jena

17.

Burgoon JK, Birk T, Pfau M (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17(1):140–169

18.

Burgoon JK, Le Poire BA (1999) Nonverbal cues and interpersonal judgments: participant and observer perceptions of intimacy, dominance, composure, and formality. Commun Monogr 66(2):105–124

19.

Caryl PG (1979) Communication by agonistic displays: what can games theory contribute to ethology? Behaviour 68(1–2):136–169

20.

Castelfranchi C, Poggi I (1987) Communication beyond the cognitive approach and speech act theory. In: Verschueren J, Bertuccelli P (eds) The pragmatic perspective. John Benjamins, Amsterdam, pp 239–254

21.

Chandler D (2007) Semiotics: the basics, 2nd edn. Routledge, Abingdon

22.

Chartrand TL, Bargh JA (1999) The chameleon effect: the perception-behavior link and social interaction. J Pers Soc Psychol 76(6):893–910

23.

Clark HH (1996) Using language. Cambridge University Press, Cambridge

24.

Cole M (1995) Socio-cultural-historical psychology: some general remarks and a proposal for a new kind of culture-genetic methodology. In: Wertsch JV, Del Rio P, Alvarez A (eds) Sociocultural studies of mind. Harvard University Press, Cambridge, pp 187–214

25.

Conte R, Castelfranchi C (1995) Cognitive and social action. University College Press, London

26.

Dijksterhuis A, Bargh JA (2001) The perception-behavior expressway: automatic effects of social perception on social behavior. Adv Exp Soc Psychol 33:1–40

27.

Dimberg U, Thunberg M, Elmehed K (2000) Unconscious facial reactions to emotional facial expressions. Psychol Sci 11(1):86–89

28.

Duncan S (1972) Some signals and rules for taking speaking turns in conversations. J Pers Soc Psychol 23(2):283–292

29.

Edinger JA, Patterson ML (1983) Nonverbal involvement and social control. Psychol Bull 93(1):30–56

30.

Efron D (1941) Gesture and environment. King’s Crown, New York

31.

Ekman P (1979) About brows: emotional and conversational signals. In: von Cranach M, Foppa K, Lepenies W, Ploog D (eds) Human ethology: claims and limits of a new discipline. Maison des Sciences de l’Homme and Cambridge University Press, Paris and Cambridge, pp 169–202

32.

Ekman P (1992) Are there basic emotions? Psychol Rev 99(3):550–553

33.

Ekman P (1997) Should we call it expression or communication? Innov Soc Sci Res 10(4):333–344 MathSciNet

34.

Ekman P, Friesen WV (1969) The repertoire of nonverbal behavior: categories, origins, usage, and coding. Semiotica 1(1):49–98

35.

Enquist M (1985) Communication during aggressive interactions with particular reference to variation in choice of behaviour. Anim Behav 33(4):1152–1161

36.

Frege FLG (1892) Über sinn und bedeutung. Z Philos Philos Kritik 100:25–50

37.

Ghiglione R, Bonnet C, Richard J-F (1990) Traité de psychologie cognitive. Dunod, Paris

38.

Grice HP (1957) Meaning. Philos Rev 66:377–388

39.

Grice HP (1969) Utterer’s meaning and intention. Philos Rev 78(2):147–177

40.

Grice HP (1989) Studies in the way of words. Harvard University Press, Cambridge

41.

Guilford T, Dawkins MS (1995) What are conventional signals? Anim Behav 49:1689–1695

42.

Hall ET (1959) The silent language. Fawcett, New York

43.

Harré R, Gillett G (1994) The discursive mind. Sage, London

44.

Hasson O (1994) Cheating signals. J Theor Biol 167:223–238

45.

Hauser MD (1996) The evolution of communication. MIT Press, Cambridge

46.

Hausser RR (2001) Foundations of computational linguistics: human-computer communication in natural language. Springer, Berlin MATH

47.

Hinde R (1975) The concept of function. In: Baerends G, Manning A (eds) Function and evolution in behaviour. Clarendon Press, Oxford, pp 3–15

48.

Hinde RA (1981) Animal signals: ethological and games-theory approaches are not incompatible. Anim Behav 29:535–542

49.

Johnstone RA (1996) Multiple displays in animal communication: ‘Backup signals’ and ‘Multiple messages’. Philos Trans R Soc Lond B, Biol Sci 351(1337):329–338

50.

Keltner D (1995) Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame. J Pers Soc Psychol 68(3):441–454

51.

Krebs JR, Dawkins R (1984) Animal signals: mind-reading and manipulation. In: Krebs JR, Davies NB (eds) Behavioural ecology: an evolutionary approach, vol 2. Blackwell Scientific Publications, Oxford, pp 380–402

52.

Kropp J, Haynes O (1987) Abusive and nonabusive mothers’ ability to identify general and specific emotion signals of infants. Child Dev 58(1):187–190

53.

Maynard Smith J (1974) The theory of games and the evolution of animal conflicts. J Theor Biol 47(1):209–221 MathSciNet

54.

Maynard Smith J, Harper DG (1988) The evolution of aggression: can selection generate variability? Philos Trans R Soc Lond B, Biol Sci 319(1196):557–570

55.

Maynard Smith J, Harper DG (1995) Animal signals: models and terminology. J Theor Biol 177:305–311

56.

Maynard Smith J, Harper DG (2003) Animal signals. Oxford University Press, Oxford

57.

McNeill D (2005) Gesture and thought. University of Chicago Press, Chicago

58.

Mehu M, Scherer KR (2012) A psycho-ethological approach to Social Signal Processing. Cogn Process. doi:10.1007/s10339-012-0435-2. Advance online publication

59.

Mehrabian A (1971) Silent messages. Wadsworth, Belmont

60.

Mininni G (2003) Il discorso come forma di vita. Guida, Napoli

61.

Morris JS, Öhman A, Dolan RJ (1998) Conscious and unconscious emotional learning in the human amygdala. Nature 393(6684):467–470

62.

Naccache L, Dehaene S (2001) Unconscious semantic priming extends to novel unseen stimuli. Cognition 80(3):215–229

63.

Niedenthal PM, Barsalou LW, Winkielman P, Krauth-Gruber S, Ric F (2005) Embodiment in attitudes, social perception, and emotion. Personal Soc Psychol Rev 9(3):184–211

64.

Ogden CK, Richards IA (1923) The meaning of meaning. A study of the influence of language upon thought and of the science of symbolism. Routledge & Kegan, London

65.

Owings DH, Morton ES (1997) The role of information in communication: an assessment/management approach. In: Owings DH, Beecher MD, Thompson NS (eds) Perspectives in ethology: communication, vol 12. Plenum Press, New York, pp 359–390

66.

Owren MJ, Bachorowski J-A (2003) Reconsidering the evolution of nonlinguistic communication: the case of laughter. J Nonverbal Behav 27(3):183–200

67.

Owren MJ, Rendall D, Ryan MJ (2010) Redefining animal signaling: influence versus information in communication. Biol Philos 25(5):755–780

68.

Patterson ML (1982) A sequential functional model of nonverbal exchange. Psychol Rev 89(3):231–249

69.

Parker G (1974) Assessment strategy and the evolution of fighting behaviour. J Theor Biol 47:223–243

70.

Partan SR, Marler P (1999) Communication goes multimodal. Science 283(5406):1272–1273

71.

Peirce CS (1931–1935) Collected chapters. Cambridge University Press, Cambridge

72.

Peirce CS (1909) A sketch of logical critics. In: Peirce Edition Project (ed) The essential Peirce: selected philosophical writings, vol 2. Indiana University Press, Bloomington & Indianapolis

73.

Peirce Edition Project (1998) The essential Peirce: selected philosophical writings (1893–1913), vol 2. Indiana University Press, Bloomington & Indianapolis

74.

Pentland A (2007) Social signal processing. IEEE Signal Process Mag 24(4):108–111

75.

Pentland A (2008) Honest signals: how they shape our world. MIT Press, Cambridge

76.

Poggi I (2007) Mind, hands, face and body. A goal and belief view of multimodal communication. Buchverlag, Weidler

77.

Poggi I, D’Errico F (2010) Cognitive modelling of human social signals. In: Proceedings of SSPW 2010—social signal processing workshop, Firenze, Italy, 29 October 2010. ACM, Sheridan Press, New Jersey

78.

Poggi I, D’Errico F (2012) Social signals. Definition and cognitive processes. Cogn Process (in press)

79.

Rendall D, Owren MJ, Ryan MJ (2009) What do animal signals mean? Anim Behav 78(2):233–240

80.

Rich BA, Fromm SJ, Berghorst LH, Dickstein DP, Brotman MA, Pine DS, Leibenluft E (2008) Neural connectivity in children with bipolar disorder: impairment in the face emotion processing circuit. J Child Psychol Psychiatry 49(1):88–96

81.

Russell JA, Fernandez-Dols J-M (1997) What does a facial expression mean? In: Russell JA, Fernandez-Dols J-M (eds) The psychology of facial expression. Cambridge University Press, Cambridge, pp 3–30

82.

Sacks H, Schegloff EA, Jefferson G (1974) A simplest systematics for the organization of turn taking for conversation. Language 50:696–735

83.

de Saussure F (1916) Cours de Linguistique générale. Payot, Paris

84.

Scharlemann JP, Eckel CC, Kacelnik A, Wilson RK (2001) The value of a smile: game theory with a human face. J Econ Psychol 22:617–640

85.

Scherer KR (1978) Personality inference from voice quality: the loud voice of extroversion. Eur J Soc Psychol 8(4):467–487

86.

Scherer KR (1992) What does facial expression express? In: Strongman KT (ed) International review of studies of emotion, vol 2. Wiley, Chichester, pp 139–165

87.

Scherer KR, Grandjean D (2008) Facial expressions allow inference of both emotions and their components. Cogn Emot 22(5):789–801

88.

Searle JR (1969) Speech acts: an essay in the philosophy of language. Cambridge University Press, Cambridge

89.

Seyfarth RM, Cheney DL (2003) Signalers and receivers in animal communication. Annu Rev Psychol 54:145–173

90.

Shannon CE, Weaver W (1949) The mathematical theory of information. University of Illinois Press, Urbana

91.

Shariff AF, Tracy JL (2011) What are emotion expressions for? Curr Dir Psychol Sci 20:395–399

92.

Silvert L, Delplanque S, Bouwalerh H, Verpoort C, Sequeira H (2004) Autonomic responding to aversive words without conscious valence discrimination. Int J Psychol 53:135–145

93.

Smith WJ (1997) The behavior of communicating, after twenty years. In: Owings DH, Beecher MD, Thompson NS (eds) Perspectives in ethology: communication, vol 12. Plenum Press, New York, pp 7–54

94.

Tomasello M (2008) The origins of human communication. MIT Press, Cambridge

95.

Tomkins SS (1962) Consciousness, imagery and affect, vol 1. Springer, New York

96.

Vehrencamp SL (2000) Handicap, index, and conventional signal elements of bird song. In: Edpmark Y, Amundsen T, Rosenqvist GG (eds) Animal signals: signalling and signal design in animal communication. Tapir Academic Press, Trondheim, pp 277–300

97.

Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comp J 27(12):1743–1759

98.

Vygotsky LS (1986) Thought and language. MIT Press, Cambridge (newly revised and edited by A Kozulin)

99.

White S (1989) Backchannels across cultures: A study of Americans and Japanese. Lang Soc 18(1):59–76

100.

Wilczynski W, Ryan MJ (2010) The behavioral neuroscience of anuran social signal processing. Curr Opin Neurobiol 20(6):754–763

101.

Woodworth RS (1961) Dynamics of behavior. Holt, New York

102.

Yik MSM, Russell JA (1999) Interpretation of faces: a cross-cultural study of a prediction from Fridlund’s theory. Cogn Emot 13(1):93–104

103.

Zahavi A (1975) Mate selection: selection for a handicap. J Theor Biol 53:205–214

Title: Conceptual analysis of social signals: the importance of clarifying terminology
Authors: Marc Mehu
Francesca D’Errico
Dirk Heylen
Publication date: 01-11-2012
Publisher: Springer-Verlag
Published in: Journal on Multimodal User Interfaces / Issue 3-4/2012
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI: https://doi.org/10.1007/s12193-012-0091-y

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3-4/2012

Conceptual frameworks for multimodal social signal processing

Discrediting signals. A model of social evaluation to study discrediting moves in political debates

Towards a conceptual framework of research on social signal processing

A three-component framework for empathic technologies to augment human interaction

A generic framework for the inference of user states in human computer interaction

Premium Partner