Skip to main content
SpringerLink
Log in

Effective recognition of facial micro-expressions with video motion magnification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition has been intensively studied for decades, notably by the psychology community and more recently the pattern recognition community. What is more challenging, and the subject of more recent research, is the problem of recognizing subtle emotions exhibited by so-called micro-expressions. Recognizing a micro-expression is substantially more challenging than conventional expression recognition because these micro-expressions are only temporally exhibited in a fraction of a second and involve minute spatial changes. Until now, work in this field is at a nascent stage, with only a few existing micro-expression databases and methods. In this article, we propose a new micro-expression recognition approach based on the Eulerian motion magnification technique, which could reveal the hidden information and accentuate the subtle changes in micro-expression motion. Validation of our proposal was done on the recently proposed CASME II dataset in comparison with baseline and state-of-the-art methods. We achieve a good recognition accuracy of up to 75.30 % by using leave-one-out cross validation evaluation protocol. Extensive experiments on various factors at play further demonstrate the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. 1 Original paper [49] reported the best possible result of 63.41 % using different set of settings.

  2. 2 Leave-one-subject-out cross validation (LOSOCV) is a subject-independent evaluation where the videos of one object are held out as testing set while the remaining videos form the training set. This is then repeated for all subjects in dataset, and the average accuracy rate across all folds is taken.

    Table 2 Confusion tables of the proposed and baseline methods using SVM classifier with linear kernel (LOSOCV protocol)
    Full size table
    Table 3 Confusion tables of the proposed and baseline methods using SVM classifier with RBF kernel (LOSOCV protocol)
    Full size table

  3. 3 We empirically found that k=5 yielded the best possible result for kNN.

References

  1. Ahonen T, Matas J, He C, Pietikainen M (2009) Rotation invariant image description with local binary pattern histogram fourier. In: Image analysis, pp 61–70

  2. Azmi R, Yegane S (2012) Facial expression recognition in the presence of occussion using local gabor binary patterns. In: Iranian Conf. on Elect Eng. (ICEE), pp 742–747

  3. Bartlett M S, Littlewort G C, Frank M G, Lainscsek C, Fasel I R, Movellan J R (2006) Automatic recognition of facial actions in spontaneous expressions. J Multimed 1(6):22–35

    Article  Google Scholar 

  4. Bashyal S, Venayagamoorthy G K (2008) Recognition of facial expressions using gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21(7):1056–1064

    Article  Google Scholar 

  5. Belhumeur P, Hespanha J, Kriegman D (1997) Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  6. Burt P, Adelson E (1983) The laplacian pyramid as a compact image code. IEEE Trans Commun 31(4):532–540

    Article  Google Scholar 

  7. Chang C C, Lin C J (2011) A library for support vector machines. ACM Trans Intell Syst Tech:1–39

  8. Ekman P (2009) Lie catching and microexpressions. Philos Deception:118–133

  9. Fehr J, Burkhardt H (2008) 3D rotation invariant local binary patterns. ICPR 2008:1–4

    Google Scholar 

  10. Frank M, Herbasz M, Sinuk K, Keller A, Nolan C (2009) I see how you feel: training laypeople and professionals to recognize fleeting emotions, presented at the Annual Meeting of the International Communication Association, New York Unpublished manuscript

  11. Heikkila M, Pietikainen M, Schmid C (2009) Description of interest regions with center-symmetric local binary patterns. Pattern Recognit 42(3):425–436

    Article  MATH  Google Scholar 

  12. Hsu C W, Lin C J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13(2):415–425

    Article  Google Scholar 

  13. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary pattern and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):765–781

    Article  Google Scholar 

  14. Kanade T, Cohn J F, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53

  15. Le Ngo A C, Phan R C W, See J (2014) Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision, pp 33–48

  16. Li X, Pfister T, Huang X, Zhao G, Pietikainen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: IEEE Int. Conf. and workshops on automatic face and gesture recognition (FG), 10th, pp 1–6

  17. Liao S, Chung A C (2007) Face recognition by using elongated local binary patterns with average maximum distance gradient magnitude. In: Asian conf. on computer vision (ACCV), pp 672–679

  18. Liong S T, Phan R C W, See J, Oh Y H, Wong K (2014) Optical strain based recognition of subtle emotions. In: 2014 international symposium on Intelligent signal processing and communication systems (ISPACS). IEEE, pp 180–184

  19. Liong S T, See J, Phan R C W, Le Ngo A, Oh YH, Wong K (2014) Subtle expression recognition using optical strain weighted features. In: 1st workshop on computer vision for affective computing, asian conf. on computer vision (ACCV), pp 644–657

  20. Lucey P, Cohn J F, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 94–101

  21. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition, 1998. IEEE, pp 200–205

  22. McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn J F, Picard R (2013) Affectiva-MIT facial expression dataset (AM-FED): naturalistic and spontaneous facial expressions collected” in-the-wild”. In: 2013 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 881– 888

  23. Nguyen H T, Caplier A (2013) Elliptical local binary patterns for face recognition. In: Computer vision-ACCV 2012 workshops, pp 85–96

  24. Ojala T, Pieikainen M, Harwood D (1996) A comparative study of texture measures with classification on featured distribution. Pattern Recogn 29:51–59

    Article  Google Scholar 

  25. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  26. Oliveira L, Mansano M, Koerich A, De Souza Britto Jr A (2011) 2d principal component analysis for face and facial-expression recognition. Comput Sci Eng 13(3):9–13

    Article  Google Scholar 

  27. Oliveira L S, Sabourin R, Bortolozzi F, Suen C Y (2003) A methodology for feature selection using multiobjective genetic algorithms for handwritten digit string recognition. Int J Patt Recog Artif Intell 17(06):903–929

    Article  Google Scholar 

  28. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo, 2005. ICME 2005. IEEE, p 5

  29. Park S, Kim D (2009) Subtle facial expression recognition using motion magnification. Pattern Recogn Lett 30(7):708–716

    Article  Google Scholar 

  30. Park S, Lee H, Shin J, Kim D (2008) The POSTECH subtle facial expression database 2007 (SFED07). In: Proceedings of the 8th POSTECH-KYUTECH joint workshop on neuroinformatics, pp 653–654

  31. Park S Y, Lee S H, Ro Y M (2015) Subtle facial expression recognition using adaptive magnification of discriminative facial motion. In: Proceedings of the 23rd annual ACM conference on multimedia, pp 911–914

  32. Pfister T, Li X, Zhao G, Pietikainen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 IEEE international conference on computer vision (ICCV). IEEE, pp 1449–1456

  33. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In: 3rd international conference on crime detection and prevention (ICDP 2009). IET, pp 1–6

  34. Rahulamathavan Y, Phan R, Veluru S, Cumanan K, Rajarajan M (2013) Privacy-preserving multi-class support vector machine for outsourcing the data classification in cloud. IEEE Trans Dependable and Secure Computing 99:467–479

    Google Scholar 

  35. Rahulamathavan Y, Phan R W, Chambers J, Parish D (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans on Affect Comput 4 (1):83–92

    Article  Google Scholar 

  36. Russell T A, Chu E, Phillips M L (2006) A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br J Clin Psychol 45(4):579–583

    Article  Google Scholar 

  37. Sarnarawickrame K, Mindya S (2013) Facial expression recognition using active shape models and support vector machines. In: 2013 International Conference on advances in ICT for emerging regions (ICTer), pp 51–55

  38. Satiyan M, Nagarajan R (2010) Recognition of facial expression using haar-like feature extraction method. In: 2010 International Conference on intelligent and advanced systems (ICIAS). IEEE, pp 1–4

  39. Schopler E, Mesibov G B (1998) Kunce. L.J., Asperger syndrome or high-functioning autism?

    Google Scholar 

  40. Shan C, Gong S, McOwan P W (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  41. Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011) Macro- and micro-expression spotting in long videos using spatio-temporal strain. In: IEEE International conference on automatic face & gesture recognition and workshops (FG 2011), pp 51–56

  42. Shreve M, Godavarthy S, Manohar V, Goldgof D, Sarkar S (2009) Towards macro- and micro-expression spotting in video using strain patterns. In: Workshop on applications of computer vision (WACV), pp 1–6

  43. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19:1635–1650

    Article  MathSciNet  Google Scholar 

  44. Uddin M Z, Lee J, Kim T S (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans. on Consum Electron 55 (4):2216–2224

    Article  Google Scholar 

  45. Wang X, Han T X, Yan S (2009) An hog-lbp human detector with partial occlusion handling. In: 2009 IEEE 12th international conference on computer vision, pp 32–39

  46. Wang Y, See J, Phan R C W, Oh Y H (2014) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision (ACCV), pp 525–537

  47. Wang Y, See J, Phan R C W, Oh Y H (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS One 10:e0124,674

    Article  Google Scholar 

  48. Wu H Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph (TOG) 31(4):65:1–65:8

    Article  Google Scholar 

  49. Yan W J, Wang S J, Zhao G, Li X, Liu Y J, Chen Y H, Fu X (2014) Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9:e86,041

    Article  Google Scholar 

  50. Yan W J, Wu Q, Liang J, Chen Y H, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230

    Article  Google Scholar 

  51. Yan W J, Wu Q, Liu Y J, Wang S J (2013) Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7

  52. Yin L, Wei X, Sun Y, Wang J, Rosato M J (2006) A 3d facial expression database for facial behavior research. In: 7th international conference on automatic face and gesture recognition, pp 211–216

  53. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by Zhejiang Provincial Natural Science Foundation of China (Grant Nos. LQ17F020002 and LQ14F020006), TM Grant under project UbeAware and 2beAware, and MOHE Grant FRGS/1/2016/ICT02/MMU/02/2. The authors would like to thank the Chinese Academy of Sciences for the CASME II micro-expression database and Su-Jing Wang for providing more details of their CASME II work in [49].

Author information

Authors and Affiliations

  1. School of Physics and Electronic Information Engineering, Wenzhou University, Wenzhou, Zhejiang, 325035, China

    Yandan Wang & Xujie Li

  2. Faculty of Computing and Informatics, Multimedia University, Jalan Multimedia, 63100, Cyberjaya, Selangor, Malaysia

    John See

  3. Faculty of Engineering, Multimedia University, Jalan Multimedia, 63100, Cyberjaya, Selangor, Malaysia

    Yee-Hui Oh, Raphael C.-W. Phan & Su-Wei Tan

  4. Information Security Group, City University London, London, EC1V OHB, UK

    Yogachandran Rahulamathavan

  5. Department of Electrical and Computer Engineering, Curtin University, CDT 250, 98009, Miri Sarawak, Malaysia

    Huo-Chong Ling

Authors
  1. Yandan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John See
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yee-Hui Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raphael C.-W. Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yogachandran Rahulamathavan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Huo-Chong Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Su-Wei Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xujie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yandan Wang.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(PDF 453 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., See, J., Oh, YH. et al. Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76, 21665–21690 (2017). https://doi.org/10.1007/s11042-016-4079-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4079-6

Keywords

Search

Navigation