Skip to main content

Facial Recognition, Facial Expression and Intention Detection

  • Chapter
  • First Online:
Second Generation Biometrics: The Ethical, Legal and Social Context

Abstract

Visual perception is probably the most important sensing ability for humans to enable social interactions and general communication. As a consequence, face recognition is a fundamental skill that humans acquire early in life and which remains an integral part of our perceptual and social abilities throughout our life span (Allison et al. 2000; Bruce and Young 1986).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A description of the project can be found under the “Screening Technologies to detect intentions of humans” on the web page http://www.dhs.gov/xabout/structure/gc_1224537081868.shtm

References

  1. Adolphs, R., D. Tranel, H. Damsio, and A.R. Damasio. 1994. Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature 372: 669–672.

    Article  Google Scholar 

  2. Allison, T., A. Puce, and G. McCarthy. 2000. Social perception from visual cues: Role of the STS region. Trends in Cognitive Sciences 4: 267–278.

    Article  Google Scholar 

  3. Arandjelovic, O., and R. Cipolla. 2004. Face recognition from face motion manifolds using Robust Kernel resistor-average distance. In Proceedings of FaceVideo04, 88.

    Google Scholar 

  4. Bicego, M., E. Grosso, and M. Tistarelli. 2006. Person authentication from video of faces: A behavioral and physiological approach using Pseudo Hierarchical Hidden Markov Models. In Proceedings first International Conference on Biometrics 2006, Lectures Notes in Computer Science, vol. 3832, 113–120. Hong Kong, January 2006.

    Google Scholar 

  5. Bodamer, J. 1947. Die prosopagnosie. Archiv fur Psychiatrie und Nervenkrankheiten 179: 6–54.

    Article  Google Scholar 

  6. Braje, W.L., D.J. Kersten, M.J. Tarr, and N.F. Troje. 1999. Illumination effects in face recognition. Psychobiology 26: 371–380.

    Google Scholar 

  7. Bruce, V., and A.W. Young. 1986. Understanding face recognition. British Journal of Psychology 77(3): 305–327.

    Article  Google Scholar 

  8. Bryatt, G., and G. Rhodes. 1998. Recognition of own-race and other-race caricatures: Implications for models of face recognition. Vision Research 38: 2455–2468.

    Article  Google Scholar 

  9. Burton, A.M., V. Bruce, and P.J.B. Hancock. 1999. From pixels to people: A model of familiar face recognition. Cognitive Science 23: 1–31.

    Article  Google Scholar 

  10. Busso, C., Z. Deng, S. Yildirim, M. Bulut, C.M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th international conference on multimodal interfaces, (State College, PA, USA, October 13–15, 2004). ICMI ’04, 205–211, ACM, New York. doi:http://doi.acm.org/10.1145/1027933.1027968

    Google Scholar 

  11. Calder, A., and A. Young. 2005. Understanding the recognition of facial identity and facial expression. Nature Reviews Neuroscience 6: 641–651.

    Article  Google Scholar 

  12. Calder, A., A. Young, D. Rowland, D. Perrett, J. Hodges, and H. Etcoff. 1996. Facial emotion recognition after bilateral amygdala damage: Differentially severe impairment of fear. Cognitive Neuropsychology 13: 699–745.

    Article  Google Scholar 

  13. Castellano, G., L. Kessous, and G. Caridakis. 2008. Emotion recognition through multiple modalities: Face, body gesture, speech. In Affect and emotion in human-computer interaction, Lectures Notes in Computer Science, vol. 4868, ed. C. Peter and R. Beale. Heidelberg: Springer.

    Google Scholar 

  14. Chen, L.S., and T.S. Huang. 2000. Emotional expressions in audiovisual human computer interaction. In Proceedings of ICME 2000, 423–426. New York, July–August 2000.

    Google Scholar 

  15. Chen, L.S. 2000. Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction. PhD Thesis, University of Illinois at Urbana-Champaign.

    Google Scholar 

  16. Cohen, I., N. Sebe, A. Garg, L.S. Chen, and T.S. Huang. 2003. Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91(1–2): 160–187.

    Article  Google Scholar 

  17. Darwin, C. 1898. The expression of emotions in man and animals. New York: D. Appleton and Company.

    Google Scholar 

  18. Datcu, D., and L. Rothkrantz. 2008. Automatic bi-modal emotion recognition system based on fusion of facial expressions and emotion extraction from speech. In IEEE face and gesture conference FG2008. September 2008 ISBN 978-1-4244-2154-1.

    Google Scholar 

  19. D’Entremont, B., and D.W. Muir. 1997. Five-month-olds’ attention and affective responses to still-faced emotional expressions. Infant Behavior and Development 20(4): 563–568.

    Article  Google Scholar 

  20. Doddington, G., W. Liggett, A. Maartin, M. Pryzbocki, and D. Reynolds. 1998. Sheep, goats, lambs, and wolves: A statistical analysis of speaker performance in the NIST 1998 Speaker Recognition Evaluation. In International conference on spoken language processing. Sydney: Sydney Convention Centre.

    Google Scholar 

  21. Dunn, J., I. Bretherton, and P. Munn. 1987. Conversations about feeling states between mothers and their young children. Developmental Psychology 23(1): 132–139.

    Article  Google Scholar 

  22. Ekman, P. 1994. Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique. Psychological Bulletin 115(2): 268–287.

    Article  Google Scholar 

  23. Ekman, P. 2009. Telling lies: Clues to deceit in the marketplace, politics, and marriage. New York: Norton.

    Google Scholar 

  24. Ekman, P., and W.V. Friesen. 1976. Pictures of facial affect. Palo Alto: Consulting Psychologists Press.

    Google Scholar 

  25. Ekman, P., and W.V. Friesen. 1978. Facial action coding system: Investigator’s guide. Palo Alto: Consulting Psychologists Press.

    Google Scholar 

  26. Ekman, P., R.E. Sorenson, and W.V. Friesen. 1969. Pan-cultural elements in facial displays of emotion. Science 164(3875): 86–88.

    Article  Google Scholar 

  27. Ekman, P., W.V. Friesen, V. Wallace, and P. Ellsworth. 1972. Emotion in the human face: Guidelines for research and an integration of findings. Oxford: Pergamon.

    Google Scholar 

  28. Essa, I., and A. Pentland. 1995. Facial expression recognition using visually extracted facial action parameters. In Proceedings of international workshop on automatic face and gesture recognition. Zurich.

    Google Scholar 

  29. Fasel, B., and J. Luettin. 2003. Automatic facial expression analysis: A survey. Pattern Recognition 36: 259–275.

    Article  Google Scholar 

  30. Gunes, H., and M. Piccardi. 2006. A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In Proceedings of the 18th international conference on pattern recognition (ICPR’06), vol. 1, 1148–1153. Hong Kong.

    Google Scholar 

  31. Gunes, H., and M. Piccardi. 2009. Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics – Part B, Special Issue on Human Computing 39(1): 64–84.

    Google Scholar 

  32. Gunes, H., M. Piccardi, and M. Pantic. 2008. From the lab to the real world: Affect recognition using multiple cues and modalities. In Affective computing, focus on emotion expression, synthesis and recognition, ed. Jimmy Or, 185–218. Vienna: I-Tech Education and Publishing. ISBN 978-3-902613-23-3.

    Google Scholar 

  33. Hancock, P.J.B., V. Bruce, and A.M. Burton. 1991. Recognition of unfamiliar faces. Trends in Cognitive Sciences 4(9): 263–266.

    Google Scholar 

  34. Harris, P.L. 1989. Children and emotion: The development of psychological understanding. Oxford: Blackwell.

    Google Scholar 

  35. Haviland, J.M., and M. Lelwica. 1987. The induced affect response: 10-week-old infants’ responses to three emotion expressions. Developmental Psychology 23(1): 97–104.

    Article  Google Scholar 

  36. Haxby, J.V., E.A. Hoffman, and M.I. Gobbini. 2000. The distributed human neural system for face perception. Trends in Cognitive Sciences 20(6): 223–233.

    Article  Google Scholar 

  37. Hill, H., and V. Bruce. 1991. Effects of lighting on the perception of facial surface. Journal of Experimental Psychology: Human Perception and Performance 4(9): 263–266.

    Google Scholar 

  38. Horn, B.K.P., and B.G. Schunck. 1981. Determining optical flow. Artificial Intelligence 17(1–3): 185–204.

    Article  Google Scholar 

  39. Hornak, J., E. Rolls, and D. Wade. 1996. Face and voice expression identification in patients with emotional and behavioral changes following ventral frontal lobe damage. Neuropsychologia 34: 173–181.

    Article  Google Scholar 

  40. Howell, A.J., and H. Buxton. 1996. Towards unconstrained face recognition from image sequences. In Proceeding of the IEEE international conference on automatic face and gesture recognition (FGR’96), 224–229. Killington.

    Google Scholar 

  41. Humphreys, G., N. Donnelly, and M. Riddoch. 1993. Expression is computed separately from facial identity, and it is computed separately for moving and static faces: Neuropsychological evidence. Neuropsychologia 31: 173–181.

    Article  Google Scholar 

  42. Izard, C.E. 1971. The face of emotion. East Norwalk: Appleton-Century-Crofts.

    Google Scholar 

  43. Izard, C.E. 1997. Emotions and facial expressions: A perspective from Differential Emotions Theory. In The psychology of facial expression: Studies in emotion and social interaction, ed. J.A. Russell and J.M. Fernndez-Dols, 57–77. New York: Cambridge University Press.

    Chapter  Google Scholar 

  44. Jack, R.E., C. Blais, C. Scheepers, P.G. Schyns, and R. Cadara. 2009. Cultural confusions show that facial expressions are not universal. Current Biology 19: 1543–1548.

    Article  Google Scholar 

  45. Kanade, T., J. Cohn, and Y. Tian. 2000. Comprehensive database for facial expression analysis. In Proceedings of the 4th IEEE international conference on automatic face and gesture recognition (FG’00), 46–53. Grenoble, March 2000.

    Google Scholar 

  46. Kanwisher, N., J. McDermott, and M. Chun. 1997. The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience 17: 4302–4311.

    Google Scholar 

  47. Kelly, D.J., P.C. Quinn, A.M. Slater, K. Lee, L. Ge, and O. Pascalis. 2007. The other-race effect develops during infancy: Evidence of perceptual narrowing. Psychological Science 18: 1084–1089.

    Article  Google Scholar 

  48. Lewis, M., and S. Goldberg. 1969. Perceptual-cognitive development in infancy: A generalized expectancy model as a function of the mother-infant interaction. Merrill-Palmer Quarterly 15(1): 81–100.

    Google Scholar 

  49. Li, Y., S. Gong, and H. Liddell. 2000. Support vector regression and classification based multiview face detection and recognition. In Proceedings of the IEEE international conference on automatic face and gesture recognition (FGR’00), 300–305. Grenoble.

    Google Scholar 

  50. Li, Y., S. Gong, and H. Liddell. 2001. Modelling faces dynamically across views and over time. In Proceedings IEEE international conference on computer vision, 554–559. Vancouver, July 2001.

    Google Scholar 

  51. Li, S.Z., R.F. Chu, S.C. Liao, and L. Zhang. 2007. Illumination invariant face recognition using near-infrared images. IEEE Transactions on PAMI 29(4): 627–639.

    Article  Google Scholar 

  52. Light, L., F. Kayra-Stuart, and S. Hollander. 1979. Recognition memory for typical and unusual faces. Journal of Experimental Psychology: Human Learning and Memory 5: 212–228.

    Article  Google Scholar 

  53. Liu, X., and T. Chen. 2003. Video-based face recognition using adaptive hidden Markov models. In Proceedings CVPR03 (I), 340–345. Madison

    Google Scholar 

  54. Lucas, S.M. 1998. Continuous n-tuple classifier and its application to real-time face recognition. In IEE Proceedings-Vision Image and Signal Processing 145(5): 343.

    Article  Google Scholar 

  55. Lucas, S.M., and T.K. Huang. 2004. Sequence recognition with scanning N-tuple ensembles. In Proceedings ICPR04 (III), 410–413. Cambridge.

    Google Scholar 

  56. Malpass, R.S., and J. Kravitz. 1969. Recognition for faces of own and other race faces. Journal of Personality and Social Psychology 13: 330–334.

    Article  Google Scholar 

  57. Markham, R., and K. Adams. 1992. The effect of type of task on children’s identification of facial expressions. Journal of Nonverbal Behavior 16(1): 21–39.

    Article  Google Scholar 

  58. Matsumoto, D. 2001. Culture and emotion. In The handbook of culture and psychology, ed. D. Matsumoto. New York: Oxford University Press.

    Google Scholar 

  59. Meissner, C.A., and J.C. Brigham. 2001. Thirty years of investigating the own-race bias in memory for faces: A meta-analytic review. Psychology, Public Policy, and Law 7: 3–35.

    Article  Google Scholar 

  60. Mika, S., G. Ratsch, B. Scholkopf, A. Smola, J. Weston, and K.-R. Muller. 1999. Invariant feature extraction and classification in kernel spaces. In Advances in neural information processing systems, vol. 12, ed. M.I. Jordan, Y. LeCun, and Sara, A. Solla. Cambridge, MA: MIT.

    Google Scholar 

  61. Moses, Y., S. Edelman, and S. Ullman. 1996. Generalization to novel images in upright and inverted faces. Perception 25(4): 443–461.

    Article  Google Scholar 

  62. Muir, D.W., and J. Nadel. 1998. Infant social perception. In Perceptual development: Visual, auditory, and speech perception in infancy, ed. A. Slater, 247–285. Hove: Psychology Press.

    Google Scholar 

  63. O’Toole, A.J., H. Abdi, F. Jiang, and P.J. Phillips. 2007a. Fusing face recognition algorithms and humans. IEEE Transactions on Systems, Man and Cybernetics 37: 1149–1155.

    Article  Google Scholar 

  64. O’Toole, A.J., P.J. Phillips, F. Jiang, J.J. Ayadd, N. Penard, and H. Abdi. 2007b. Face recognition algorithms surpass humans matching faces across changed in illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence 29: 1642–1646.

    Article  Google Scholar 

  65. O’Toole, A.J., P.J. Phillips, A. Narvekar, F. Jiang, and J. Ayyad. 2008. Face recognition algorithms and the ‘other-race’ effect [Abstract]. Journal of Vision 8(6): 256–256.

    Article  Google Scholar 

  66. Paleari, M., R. Benmokhtar, and B. Huet. 2009. Evidence theory based multimodal emotion recognition MMM 2009, 15th international multiMedia modeling conference, 435–446. Sophia Antipolis, January 7–9, 2009.

    Google Scholar 

  67. Pantic, M., and L.J.M. Rothkrantz. 2000. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on PAMI 22(12): 1424–1445.

    Article  Google Scholar 

  68. Phillips, P.J., P.J. Flynn, W.T. Scruggs, K.W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W.J. Worek. 2005. Overview of the face recognition grand challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition, vol. 1, 947–954. San Diego: IEEE Computer Society.

    Google Scholar 

  69. Picard, R.W. 2000. Affective computing. Cambridge: MIT.

    Google Scholar 

  70. Ratliff, M.S., and E. Patterson. 2008. Emotion recognition using facial expressions with active appearance models. In Proceedings of IASTED human computer interaction 2008, 92–138. Anaheim.

    Google Scholar 

  71. Raytchev, B., and H. Murase. 2001. Unsupervised face recognition from image sequences. In Proceedings ICIP01(I), 1042–1045. Thessaloniki.

    Google Scholar 

  72. Raytchev, B., and H. Murase. 2002. VQ-Faces: Unsupervised face recognition from image sequences. In Proceedings ICIP02 (II), 809–812. Rochester, New York.

    Google Scholar 

  73. Raytchev, B., and H. Murase. 2003. Unsupervised recognition of multi-viewface sequences based on pairwise clustering with attraction and repulsion. Computer Vision and Image Understanding 91(1–2): 22–52.

    Article  Google Scholar 

  74. Roli, F., and J. Kittler (eds.). 2002. Multiple Classifier Systems, Lectures Notes in Computer Science, vol. 2364. Berlin: Springer.

    Google Scholar 

  75. Russell, J.A. 1991. Culture and the categorization of emotions. Psychological Bulletin 110(3): 426–450.

    Article  Google Scholar 

  76. Saks, M.J., and J.J. Koehler. 2005. The coming paradgm shift in forensic identification science. Science 309: 892–895.

    Article  Google Scholar 

  77. Schyns, P.G. 1998. Diagnostic recognition: Task constraints, object information, and their interactions. Cognition 67(1–2): 147–179.

    Article  Google Scholar 

  78. Sebe, N., I. Cohen, F.G. Cozman, T. Gevers, and T.S. Huang. 2005. Learning probabilistic classifiers for human-computer interaction applications. Multimedia Systems 10(6): 484–498.

    Article  Google Scholar 

  79. Sebe, N., I. Cohen, T. Gevers, and T.S. Huang. 2006. Emotion recognition based on joint visual and audio cues. In Proceedings of the 18th international conference on pattern recognition (ICPR’06), vol. 1, 1136–1139. Hong Kong.

    Google Scholar 

  80. Shan, C., and R. Braspenning. 2009. Recognizing facial expressions automatically from video. In Handbook of ambient intelligence and smart environments, ed. H. Nakashima, J. Augusto, and H. Aghajan. New York: Springer.

    Google Scholar 

  81. Soken, N.H., and A.D. Pick. 1999. Infants’ perception of dynamic affective expressions: Do infants distinguish specific expressions? Child Development 70(6): 1275–1282.

    Article  Google Scholar 

  82. Sorce, J.F., R.N. Emde, J.J. Campos, and M.D. Klinnert. 1985. Maternal emotional signaling: Its effect on the visual cliff behavior of 1-year-olds. Developmental Psychology 21(1): 195–200.

    Article  Google Scholar 

  83. Sroufe, L.A. 1995. Emotional development: The organization of emotional life in the early years. New York: Cambridge University Press.

    Google Scholar 

  84. Stern, D. 1985. The interpersonal world of the infant: A view from psychoanalysis and developmental psychology. New York: Basic Books.

    Google Scholar 

  85. Tistarelli, M. 1996. Multiple constraints to compute optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(12): 1243–1250.

    Article  Google Scholar 

  86. Tistarelli, M., M. Bicego, and E. Grosso. 2008. Dynamic face recognition: From human to machine vision. In Image and Vision Computing: Special issue on Multimodal Biometrics, M. Tistarelli and J. Bigun eds. doi:10.1016/j.imavis.2007.05.006.

    Google Scholar 

  87. Tranel, D., A. Damasio, and H. Damasio. 1988. Intact recognition of facial expression, gender, and age in patients with impaired recognition of face identity. Neurology 38: 690–696.

    Google Scholar 

  88. Tretiak, O., and L. Pastor. 1984. Velocity estimation from image sequences with second order differential operators. In Proceeding of 7th IEEE international conference on pattern recognition, 16–19. IEEE.

    Google Scholar 

  89. Troje, N.F., and H.H. Bülthoff. 1996. Face recognition under varying pose: The role of texture and shape. Vision Research 36: 1761–1771.

    Article  Google Scholar 

  90. Valentine, T. 1991. A unified account of the effects of distinctiveness, inversion, and race in face recognition. Quarterly Journal of Experimental Psychology 43A: 161–204.

    Google Scholar 

  91. Walker-Andrews, A.S. 1997. Infants’ perception of expressive behaviors: Differentiation of multimodal information. Psychological Bulletin 121(3): 437–456.

    Article  Google Scholar 

  92. Walker-Andrews, A.S., and L.E. Bahrick. 2001. Perceiving the real world: Infants’ detection of and memory for social information. Infancy 2(4): 469–481.

    Article  Google Scholar 

  93. Wellman, H.M., P.L. Harris, M. Banerjee, and A. Sinclair. 1995. Early understanding of emotion: Evidence from natural language. Cognition and Emotion 9(2–3): 117–149.

    Google Scholar 

  94. Widen, S.C., and J.A. Russell. 2003. A closer look at preschoolers’ freely produced labels for facial expressions. Developmental Psychology 39(1): 114–128.

    Article  Google Scholar 

  95. Widen, S.C., and J.A. Russell. 2008a. Children acquire emotion categories gradually. Cognitive Development 23(2): 291–312.

    Article  Google Scholar 

  96. Widen, S.C., and J.A. Russell. 2008b. Young children’s understanding of others’ emotions. In Handbook of emotions, 3rd ed, ed. M. Lewis, J.M. Haviland-Jones, and L.F. Barrett, 348–363. New York: Guilford Press.

    Google Scholar 

  97. Young, A.W., F. Newcombe, de Haan, M. Small, and D.C. Hay. 1993. Face perception after brain injury: Selective impairments affecting identity and expression. Brain 116(4): 941–959.

    Google Scholar 

  98. Zeng, Z., M. Pantic, G.I. Roisman, and T.S. Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on PAMI 31(1): 39–58.

    Article  Google Scholar 

  99. Zhou, S.K., V. Krueger, and R. Chellappa. 2003. Probabilistic recognition of human faces from video. Computer Vision and Image Understanding 91(1–2): 214–245.

    Article  Google Scholar 

  100. Zhou, S.K., R. Chellappa, and B. Moghaddam. 2004. Visual tracking and recognition using appearance-adaptive models in particle filters. Image Processing 13(11): 1491–1506.

    Article  Google Scholar 

Download references

Acknowledgements

Funding from the European Union COST action 2101 “Biometrics for Identity Documents and Smart Cards” and from the Regional Research Authority are acknowledged.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Massimo Tistarelli .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer Science+Business Media B.V.

About this chapter

Cite this chapter

Tistarelli, M., Barrett, S.E., O’Toole, A.J. (2012). Facial Recognition, Facial Expression and Intention Detection. In: Mordini, E., Tzovaras, D. (eds) Second Generation Biometrics: The Ethical, Legal and Social Context. The International Library of Ethics, Law and Technology, vol 11. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-3892-8_11

Download citation

Publish with us

Policies and ethics