Skip to main content
SpringerLink
Log in

Foundations of Human Computing: Facial Expression and Emotion

  • Conference paper
Artifical Intelligence for Human Computing

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4451))

Abstract

Many people believe that emotions and subjective feelings are one and the same and that a goal of human-centered computing is emotion recognition. The first belief is outdated; the second mistaken. For human-centered computing to succeed, a different way of thinking is needed. Emotions are species-typical patterns that evolved because of their value in addressing fundamental life tasks. Emotions consist of multiple components, of which subjective feelings may be one. They are not directly observable, but inferred from expressive behavior, self-report, physiological indicators, and context. I focus on expressive facial behavior because of its coherence with other indicators and research. Among the topics included are measurement, timing, individual differences, dyadic interaction, and inference. I propose that design and implementation of perceptual user interfaces may be better informed by considering the complexity of emotion, its various indicators, measurement, individual differences, dyadic interaction, and problems of inference.

A previous version of this paper was originally published in the Proceedings of the ACM International Conference on Multimodal Interfaces, Banff, Canada, 2006 (Copyright © ACM Press).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Lazarus, R.S.: Emotion and adaptation. Oxford University Press, New York (1991)

    Google Scholar 

  2. Ekman, P.: Emotions revealed. Times, New York (2003)

    Google Scholar 

  3. Keltner, D., Haidt, J.: Social functions of emotions at multiple levels of analysis. Cognition and Emotion 13, 505–522 (1999)

    Article  Google Scholar 

  4. Darwin, C.: The expression of the emotions in man and animals, 3rd edn. Oxford University Press, New York (1872/1998)

    Google Scholar 

  5. Ekman, P., Rosenberg, E. (eds.): What the face reveals. Oxford University Press, New York (2005)

    Google Scholar 

  6. Pantic, M., Patras, I.: Dynamics of facial expressions: Recognition of facial actions and their temporal segments from profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B 36, 443–449 (2006)

    Article  Google Scholar 

  7. Tian, Y., Cohn, J.F., Kanade, T.: Facial expression analysis. In: Li, S.Z., Jain, A.K. (eds.) Handbook of face recognition, pp. 247–276. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  8. Pantic, M., Rothkrantz, M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)

    Article  Google Scholar 

  9. Davidson, R.J., et al.: Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology I. Journal of Personality and Social Psychology 58, 330–341 (1990)

    Article  Google Scholar 

  10. Levenson, R.W., Ekman, P., Friesen, W.V.: Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27, 363–384 (1990)

    Article  Google Scholar 

  11. Watson, D., Tellegen, A.: Toward a consensual structure of mood. Psychological Bulletin 98, 219–235 (1985)

    Article  Google Scholar 

  12. Cohn, J.F., Elmore, M.: Effects of contingent changes in mothers’ affective expression on the organization of behavior in 3-month old infants. Infants Behavior and Development 11, 493–505 (1988)

    Article  Google Scholar 

  13. Schmidt, K.L., Cohn, J.F.: Human facial expressions as adaptations: Evolutionary perspectives in facial expression research. Yearbook of Physical Anthropology 116, 8–24 (2001)

    Article  Google Scholar 

  14. Gottman, J., Levenson, R., Woodin, E.: Facial expressions during marital conflict. Journal of Family Communication 1, 37–57 (2001)

    Article  Google Scholar 

  15. Cohn, J.F., Campbell, S.B.: Influence of maternal depression on infant affect regulation. In: Cicchetti, D., Toth, S.L. (eds.) Developmental perspectives on depression, pp. 103–130. University of Rochester Press, Rochester (1992)

    Google Scholar 

  16. Cohn, J.F., et al.: Individual differences in facial expression: Stability over time, relation to self-reported emotion, and ability to inform person identification. In: International Conference on Multimodal User Interfaces, Pittsburgh, PA, pp. 491–496 (2002)

    Google Scholar 

  17. Bartlett, M.S., et al.: Fully automatic facial action recognition in spontaneous behavior. In: IEEE International Conference on Automatic Face and Gesture Recognition, vol. FG 2006, Southampton, England, pp. 223–228 (2006)

    Google Scholar 

  18. Valstar, M.F., et al.: Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In: ACM International Conference on Multimodal Interfaces, Banff, Canada, pp. 162–170 (2006)

    Google Scholar 

  19. Lucey, S., et al.: AAM derived face representations for robust facial action recognition. In: Seventh IEEE International Conference on Automatic Face and Gesture Recognition, Southampton, UK, pp. 155–160 (2006)

    Google Scholar 

  20. Boker, S.M., et al.: Coordinated motion and facial expression in dyadic conversation. In: NSF Human and Social Dynamics Principal Investigators Meeting, Washington, DC (2006)

    Google Scholar 

  21. Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the Facial Action Coding System. In: Coan, J.A., Allen, J.B. (eds.): The handbook of emotion elicitation and assessment. Oxford University Press Series in Affective Science. Oxford University Press, New York (In press)

    Google Scholar 

  22. Ekman, P.: An argument for basic emotions. Cognition and Emotion 6, 169–200 (1992)

    Article  Google Scholar 

  23. Ekman, P., Friesen, W.V.: Unmasking the face: A guide to emotions from facial cues. Prentice-Hall Inc., Englewood Cliffs (1975)

    Google Scholar 

  24. Ekman, P., Friesen, W.V.: Facial action coding system. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  25. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, vol. 4, Grenoble, pp. 46–53 (2000)

    Google Scholar 

  26. Pantic, M., et al.: Affective multimodal human-computer interaction. In: ACM International Conference on Multimedia, pp. 669–676 (2005)

    Google Scholar 

  27. Pantic, M., Rothkrantz, M.: Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91, 1371–1390 (2003)

    Article  Google Scholar 

  28. Keltner, D., Ekman, P.: Facial expression of emotion. In: Lewis, M., Haviland, J.M. (eds.) Handbooks of emotions, pp. 236–249. Guilford, New York (2000)

    Google Scholar 

  29. Scherer, K.R.: What does facial expression express? In: Strongman, K.T. (ed.) International Review of Studies on Emotion, vol. 2, pp. 138–165. John Wiley & Sons Ltd, Chichester (1992)

    Google Scholar 

  30. Izard, C.E., Dougherty, L.M., Hembree, E.A.: A system for identifying affect expressions by holistic judgments. Instructional Resources Center, University of Delaware, Newark, Delaware (1983)

    Google Scholar 

  31. Ekman, P., Friesen, W.V., O’Sullivan, M.: Smiles when lying. Journal of Personality and Social Psychology 54, 414–420 (1988)

    Article  Google Scholar 

  32. Cohn, J.F., Ekman, P.: Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In: Harrigan, J.A., Rosenthal, R., Scherer, K. (eds.) Handbook of nonverbal behavior research methods in the affective sciences, pp. 9–64. Oxford University Press, New York (2005)

    Google Scholar 

  33. Ekman, P., Friesen, W.V., Hager, J.C. (eds.): Facial action coding system. Research Nexus, Network Research Information, Salt Lake City, UT (2002)

    Google Scholar 

  34. Rosenthal, R.: Conducting judgment studies. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) Handbook of nonverbal behavior research methods in the affective sciences, pp. 199–236. Oxford University Press, New York (2005)

    Google Scholar 

  35. Oster, H., Hegley, D., Nagel, L.: Adult judgments and fine-grained analysis of infant facial expressions: Testing the validity of a priori coding formulas. Developmental Psychology 28, 1115–1131 (1992)

    Article  Google Scholar 

  36. Cohn, J.F., Kanade, T.: Use of automated facial image analysis for measurement of emotion expression. In: Coan, J.A., Allen, J.B. (eds.) The handbook of emotion elicitation and assessment, Oxford Unversity Press, New York (in press)

    Google Scholar 

  37. Eibl-Eibesfeldt, I.: Human ethology. Aldine de Gruvier, New York (1989)

    Google Scholar 

  38. Sayette, M.A., et al.: A psychometric evaluation of the Facial Action Coding System for assessing spontaneous expression. Journal of Nonverbal Behavior 25, 167–186 (2001)

    Article  Google Scholar 

  39. Fridlund, A.J., et al.: Audience effects on solitary faces during imagery: Displaying to the people in your head. Journal of Nonverbal Behavior 14, 113–137 (1990)

    Article  Google Scholar 

  40. Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing 2, 1–12 (2004)

    Article  MathSciNet  Google Scholar 

  41. Fleiss, J.L.: Statistical methods for rates and proportions. Wiley, New York (1981)

    MATH  Google Scholar 

  42. Messinger, D.S., et al.: Continuously measured smile dynamics in infant-mother interaction. Miami (2006)

    Google Scholar 

  43. Ibanez, L., et al.: Automated measurement of infant and mother interactive smiling. American Psychological Society (2006)

    Google Scholar 

  44. Russell, J.A., Fernandez-Dols, J.M. (eds.): The psychology of facial expression. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  45. Ekman, P.: Facial expression and emotion. American Psychologist 48, 384–392 (1993)

    Article  Google Scholar 

  46. Cohn, J.F.: Automated analysis of the configuration and timing of facial expression. In: Ekman, P., Rosenberg, E. (eds.) What the face reveals, pp. 388–392. Oxford University Press, New York (2005)

    Google Scholar 

  47. Theobald, B.J., Cohn, J.F.: Facial image synthesis. In: Sander, D., Scherer, K.R. (eds.): Oxford companion to affective sciences: An encyclopedic dictionary for the affective sciences. Oxford University Press, New York (In press)

    Google Scholar 

  48. Edwards, K.: The face of time: Temporal cues in facial expressions of emotion. Psychological Science 9, 270–276 (1998)

    Article  Google Scholar 

  49. Krumhuber, E., Kappas, A.: Moving smiles: The role of dynamic components for the perception of the genuineness of smiles. Journal of Nonverbal Behavior 29, 3–24 (2005)

    Article  Google Scholar 

  50. Cohn, J.F., et al.: Multimodal coordination of facial action, head rotation, and eye motion. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, vol. FG’04, Seoul, Korea, pp. 645–650 (2004)

    Google Scholar 

  51. Ambadar, Z., Schooler, J., Cohn, J.F.: Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions. Psychological Science 16, 403–410 (2005)

    Article  Google Scholar 

  52. Ambadar, Z., Cohn, J.F., Reed, L.I.: All smiles are not created equal: Timing characteristics and interpretation of spontaneous smiles (2006)

    Google Scholar 

  53. Dimberg, U., Thunberg, M., Grunedal, S.: Facial reactions to emotional stimuli: Automatically controlled emotional responses. Cognition and Emotion 16, 449–471 (2002)

    Article  Google Scholar 

  54. Moore, G.A., Cohn, J.F., Campbell, S.B.: Mothers’ affective behavior with infant siblings: Stability and change. Developmental Psychology 33, 856–860 (1997)

    Article  Google Scholar 

  55. Oster, H., et al.: The patterning of facial expressions in Chinese, Japanese, and American infants in fear- and anger- eliciting situations. Poster presented at the International Conference on Infant Studies, Providence, RI (1996)

    Google Scholar 

  56. Matsumoto, D., Willingham, B.: The thrill of victory and the agony of defeat: spontaneous expressions of medal winners of the 2004 Athens olympic games. Journal of Personality & Social Psychology 91, 568–581 (2006)

    Article  Google Scholar 

  57. Camras, L.A., Chen, Y.: Culture, ethnicity, and children’s facial expressions: A study of European American, mainland Chinese, Chinese American, and adopted Chinese girls. 6, 103-114 (2006)

    Google Scholar 

  58. Markus, H.R., Kitayama, S.: Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review 98, 224–253 (1991)

    Article  Google Scholar 

  59. Tsai, J.L., Levenson, R.W., McCoy, K.: Cultural and temperamental variation in emotional response. Emotion 6, 484–497 (2006)

    Article  Google Scholar 

  60. Zlochower, A.J., Cohn, J.F.: Vocal timing in face-to-face interaction of clinically depressed and nondepressed mothers and their 4-month-old infants. Infant Behavior and Development 19, 373–376 (1996)

    Article  Google Scholar 

  61. Haidt, J., Keltner, D.: Culture and facial expression: Open-ended methods find more expressions and a gradient of recognition. Emotion and Cognition 13, 225–266 (1999)

    Article  Google Scholar 

  62. Jaffe, J., et al.: Rhythms of dialogue in early infancy. Monographs of the Society for Research in Child Development 66 (2001)

    Google Scholar 

  63. Cohn, J.F., Tronick, E.Z.: Mother-Infant face-to-face interaction: Influence is bidirectional and unrelated to periodic cycles in either partner’s behavior. Developmental Psychology 34, 386–392 (1988)

    Article  Google Scholar 

  64. Cohn, J.F., Tronick, E.Z.: Mother infant interaction: The sequence of dyadic states at three, six, and nine months. Developmental Psychology 23, 68–77 (1987)

    Article  Google Scholar 

  65. Chow, S.M., et al.: Emotion as a thermostat: representing emotion regulation using a damped oscillator model. Emotion 5, 208–225 (2005)

    Article  Google Scholar 

  66. Boker, S.M., et al.: Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods 7, 338–355 (2002)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, University of Pittsburgh, 3137 SQ, 210 S. Bouquet Street, Pittsburgh, PA 15260, USA

    Jeffrey F. Cohn

Authors
  1. Jeffrey F. Cohn
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Thomas S. Huang Anton Nijholt Maja Pantic Alex Pentland

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Berlin Heidelberg

About this paper

Cite this paper

Cohn, J.F. (2007). Foundations of Human Computing: Facial Expression and Emotion. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds) Artifical Intelligence for Human Computing. Lecture Notes in Computer Science(), vol 4451. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72348-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-72348-6_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-72346-2

  • Online ISBN: 978-3-540-72348-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation