Skip to main content
SpringerLink
Log in

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

Owing to the development and advancement of artificial intelligence, numerous works have been established in the human facial expression recognition system. Meanwhile, the detection and classification of micro-expressions have been attracting attention from various research communities in the recent few years. In this paper, we first review the processes of a conventional optical-flow-based recognition system. Concisely, it comprises four basic steps: facial landmarks annotations (to detect the face and locate the landmark coordinates), optical flow guided images computation (to describe the dynamic changes on the face), feature extraction (to summarize the features encoded) and emotion class categorization (to build a classification model based on the given training data). Secondly, a few approaches have been implemented to improve the feature extraction part, such as exploiting GAN to generate more image samples. Particularly, several variations of optical flow are computed in order to generate optimal images, which lead to high recognition accuracies. Next, GAN, a combination of Generator and Discriminator, is utilized to generate new “fake” images to increase the sample size. Thirdly, a modified state-of-the-art convolutional neural networks is proposed. In brief, multiple optical flow derived components are adopted in the OFF-ApexNet structure to better represent the facial subtle motion changes. From the experiment results obtained, the additional optical flow information computed does not complement the feature extraction stage, and thus leading to poorer recognition performance. On the other hand, the implementation of GAN to the input data improves the performance in SMIC dataset, by achieving the accuracy of 61.80%, 62.20% and 60.98% for AC-GAN, SAGAN and without GAN images, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14

Similar content being viewed by others

References

  1. Face++ research toolkit. https://www.faceplusplus.com, 2013.

  2. Amos, B., Ludwiczuk, B., Satyanarayanan, M. (2016). Openface A general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118 CMU School of Computer Science.

  3. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M. (2013). Robust discriminative response map fitting with constrained local models. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3444–3451).

  4. Barron, J.L., Fleet, D.J., Beauchemin, S.S., Burkitt, T. (1992). Performance of optical flow techniques. In Proceedings IEEE computer society conference on computer vision and pattern recognition (p. 1992): IEEE.

  5. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J. (1995). Active shape models-their training and application. Computer vision and image understanding, 61(1), 38–59.

    Google Scholar 

  6. Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In International conference on computer vision & pattern recognition (CVPR’05), (Vol. 1 pp. 886–893): IEEE Computer Society.

  7. Davison, A., Merghani, W., Lansley, C., Ng, C.-C., Yap, M. H. (2018). Objective micro-facial movement detection using facs-based regions and baseline evaluation. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) (pp. 642–649): IEEE.

  8. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H. (2018). Samm a spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing, 9(1), 116–129.

    Google Scholar 

  9. Davison, A.K., Yap, M.H., Lansley, C. (2015). Micro-facial movement detection using individualised baselines and histogram-based descriptors. In IEEE international conference on systems, man, and cybernetics (pp. 1864–1869): IEEE.

  10. Dollár, P., Welinder, P., Perona, P. (2010). Cascaded pose regression. In IEEE computer society conference on computer vision and pattern recognition (p. 2010): IEEE.

  11. Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T. (2015). Flownet Learning optical flow with convolutional networks (pp. 2758–2766).

  12. Drucker, H., Burges, C.J., Kaufman, L., Smola, A.J., Vapnik, V. (1997). Support vector regression machines. In Advances in neural information processing systems (pp. 155–161).

  13. Edwards, G. J., Cootes, T. F., Taylor, C. J. (1998). Face recognition using active appearance models. In European conference on computer vision (pp. 581–595): Springer.

  14. Ekman, P., & Friesen, W.V. (1969). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106.

    Google Scholar 

  15. Ekman, P., & Friesen, W.V. (1978). Facial action coding system consulting psychologists press. Palo Alto.

  16. Endres, J., & Laidlaw, A. (2009). Micro-expression recognition training in medical students: a pilot study. BMC Medical Education, 9(1), 47.

    Google Scholar 

  17. Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis (pp. 363–370): Springer.

  18. Gan, Y., & Liong, S.-T. (2018). Bi-directional vectors from apex in cnn for micro-expression recognition. In IEEE 3rd international conference on image, vision and computing (ICIVC) (p. 2018): IEEE.

  19. Gan, Y., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K. (2019). Off-apexnet on micro-expression recognition system. Signal Processing: Image Communication, 74, 129–139.

    Google Scholar 

  20. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672–2680).

  21. Goshtasby, A. (1988). Image registration by local approximation methods. Image and Vision Computing, 6 (4), 255–261.

    Google Scholar 

  22. Gunn, S.R., & et al. (1998). Support vector machines for classification and regression. ISIS technical report, 14(1), 5–16.

    Google Scholar 

  23. Happy, S., & Routray, A. (2017). Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Transactions on Affective Computing.

  24. Holmes, M. (2011). National security behavioral detection: a typography of strategies, costs, and benefits. Journal of Transportation Security, 4(4), 361.

    Google Scholar 

  25. Horn, B.K., & Schunck, B.G. (1981). Determining optical flow. Artificial intelligence, 17(1-3), 185–203.

    Google Scholar 

  26. House, C., & Meyer, R. (2015). Preprocessing and descriptor features for facial micro-expression recognition.

  27. Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M. (2016). Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 175, 564–578.

    Google Scholar 

  28. Huang, Z., Zhou, E., Cao, Z. (2015). Coarse-to-fine face alignment with multi-scale local patch regression. arXiv:1511.04901.

  29. Hui, T.-W., Tang, X., Change Loy, C. (2018). Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8981–8989).

  30. Kim, D.H., Baddar, W., Jang, J., Ro, Y.M. (2017). Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Transactions on Affective Computing, (1),1–1.

  31. Kowalski, M., Naruniec, J., Trzcinski, T. (2017). Deep alignment network a convolutional neural network for robust face alignment. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 88–97).

  32. Le Ngo, A. C., Johnston, A., Phan, R.C.-W., See, J. (2018). Micro-expression motion magnification: Global lagrangian vs. local eulerian approaches. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) (pp. 650–656): IEEE.

  33. Le Ngo, A.C., See, J., Phan, R. C.-W. (2017). Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Transactions on Affective Computing, 8(3), 396–411.

    Google Scholar 

  34. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4681– 4690).

  35. Li, J., Soladie, C., Seguier, R. (2018). Ltp-ml: micro-expression detection by recognition of local temporal pattern of facial movements. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) (pp. 634–641): IEEE.

  36. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M. (2018). Towards reading hidden emotions a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Transactions on Affective Computing, 9(4), 563–577.

    Google Scholar 

  37. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M. (2013). A spontaneous micro-expression database: Inducement, collection and baseline. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1–6): IEEE.

  38. Li, Y., Liu, S., Yang, J., Yang, M.-H. (2017). Generative face completion. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3911–3919).

  39. Liaw, A., Wiener, M., et al. (2002). Classification and regression by randomforest. R news, 2(3), 18–22.

    Google Scholar 

  40. Liong, S.T. (2017). Micro-expression recognition analysis using facial strain/Liong Sze Teng. PhD thesis, University of Malaya.

  41. Liong, S.-T., Gan, Y., See, J., Khor, H.-Q. (2019). A shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition system. arXiv:1902.03634.

  42. Liong, S.-T., Gan, Y., Yau, W.-C., Huang, Y.-C., Ken, T.L. (2018).

  43. Liong, S.-T., See, J., Phan, R.C.-W., Le Ngo, A.C., Oh, Y.-H., Wong, K. (2014). Subtle expression recognition using optical strain weighted features. In Asian conference on computer vision (pp. 644–657): Springer.

  44. Liong, S.-T., See, J., Phan, R. C.-W., Wong, K., Tan, S.-W. (2018). Hybrid facial regions extraction for micro-expression recognition system. Journal of Signal Processing Systems, 90(4), 601–617.

    Google Scholar 

  45. Liong, S.-T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.-H., Phan, R. (2015). Automatic apex frame spotting in micro-expression database. In 2015 3rd IAPR Asian conference on pattern recognition (ACPR) (pp. 665–669): IEEE.

  46. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W. (2016). Automatic micro-expression recognition from long video using a single spotted apex. In Asian conference on computer vision (pp. 345–360): Springer.

  47. Liong, S. -T., See, J., Wong, K., Phan, R.C.-W. (2018). Less is more micro-expression recognition from video using apex frame. Signal Processing: Image Communication, 62, 82–92.

    Google Scholar 

  48. Liu, Y., Du, H., Zheng, L., Gedeon, T. (2019). A neural micro-expression recognizer. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019) (pp. 1–4): IEEE.

  49. Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Fu, X. (2016). A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 7(4), 299–310.

    Google Scholar 

  50. Lodder, S., & Goossens, L. (2016). Loneliness and the social monitoring system Emotion recognition and eye gaze in a real-life conversation. British Journal of Psychology, 107, 135–153.

    Google Scholar 

  51. Lu, H., Kpalma, K., Ronsin, J. (2018). Motion descriptors for micro-expression recognition. Signal Processing: Image Communication, 67, 108–117.

    Google Scholar 

  52. Moilanen, A., Zhao, G., Pietikäinen, M. (2014). Spotting rapid facial movements from videos using appearance-based feature difference analysis. In 2014 22nd international conference on pattern recognition (pp. 1722–1727): IEEE.

  53. Odena, A., Olah, C., Shlens, J. (2017). Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th international conference on machine learning-volume 70. JMLR.org (pp. 2642–2651).

  54. Ojala, T., Pietikäinen, M., Harwood, D. (1996). A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29(1), 51–59.

    Google Scholar 

  55. Patel, D., Hong, X., Zhao, G. (2016). Selective deep features for micro-expression recognition. In 2016 23rd international conference on pattern recognition (ICPR) (pp. 2258–2263): IEEE.

  56. Peng, M., Wang, C., Chen, T., Liu, G., Fu, X. (2017). Dual temporal scale convolutional neural network for micro-expression recognition. Frontiers in psychology, 8, 1745.

    Google Scholar 

  57. Pfister, T., Li, X., Zhao, G., Pietikäinen, M. (2011). Recognising spontaneous facial micro-expressions. In 2011 international conference on computer vision (pp. 1449–1456): IEEE.

  58. Polikovsky, S., Kameda, Y., Ohta, Y. (2009). Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor.

  59. Polikovsky, S., Kameda, Y., Ohta, Y. (2013). Facial micro-expression detection in hi-speed video based on facial action coding system (facs). IEICE Transactions on Information and Systems, 96(1), 81–92.

    Google Scholar 

  60. Qian, J.X.Z. Grand mediation in chinese rural judicial——based on semi-strangers and endogenous society in the village.Journal of the Postgraduate of Zhongnan University of Economics and Law, (3), 22, 2009.

  61. Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., Fu, X. (2018). Cas (me) 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 9(4), 424–436.

    Google Scholar 

  62. See, J., Yap, M. H., Li, J., Hong, X., Wang, S.-J. (2019). Megc 2019–the second facial micro-expressions grand challenge. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019) (pp. 1–5): IEEE.

  63. Senst, T., Eiselein, V., Sikora, T. (2012). Robust local optical flow for feature tracking. IEEE Transactions on Circuits and Systems for Video Technology, 22(9), 1377–1387.

    Google Scholar 

  64. Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., Sarkar, S. (2014). Automatic expression spotting in videos. Image and Vision Computing, 32(8), 476–486.

    Google Scholar 

  65. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S. (2011). Macro-and micro-expression spotting in long videos using spatio-temporal strain. In Face and Gesture 2011 (pp. 51–56): IEEE.

  66. Stewart, P.A., Waller, B.M., Schubert, J.N. (2009). Presidential speechmaking style emotional response to micro-expressions of facial affect. Motivation and Emotion, 33(2), 125.

    Google Scholar 

  67. Sun, D., Yang, X., Liu, M.-Y., Kautz, J. (2018). Pwc-net Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8934–8943).

  68. Sun, Y., Wang, X., Tang, X. (2013). Deep convolutional network cascade for facial point detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3476–3483).

  69. Takalkar, M.A., & Xu, M. (2017). Image based facial micro-expression recognition using deep learning on small datasets. In 2017 international conference on digital image computing: Techniques and applications (DICTA) (pp. 1–7): IEEE.

  70. Vrij, A., & Mann, S. (2005). Police use of nonverbal behavior as indicators of deception. In Riggio, R.E., & Feldman, R.S. (Eds.) Applications of nonverbal communication (pp. 63–94).

  71. Wang, S.-J., Wu, S., Fu, X. (2016). A main directional maximal difference analysis for spotting micro-expressions. In Asian conference on computer vision (pp. 449–461): Springer.

  72. Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., Fu, X. (2014). Micro-expression recognition using dynamic textures on tensor independent color space. In 2014 22nd international conference on pattern recognition (pp. 4678–4683): IEEE.

  73. Wang, S.-J., Yan, W.-J., Zhao, G., Fu, X., Zhou, C.-G. (2014). Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In Workshop at the European conference on computer vision (pp. 325–338): Springer.

  74. Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H. (2014). Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In Asian conference on computer vision (pp. 525–537): Springer.

  75. Wang, Y., See, J., Phan, R. C.-W., Oh, Y.-H. (2015). Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS one, 10(5), 1–20.

    Google Scholar 

  76. Wedel, A., Pock, T., Braun, J., Franke, U., Cremers, D. (2008). Duality tv-l1 flow with fundamental matrix prior. In 2008 23rd international conference image and vision computing New Zealand (pp. 1–6): IEEE.

  77. Wold, S., Esbensen, K., Geladi, P. (1987). Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37–52.

    Google Scholar 

  78. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W. (2012). Eulerian video magnification for revealing subtle changes in the world.

  79. Xia, Z., Feng, X., Peng, J., Peng, X., Zhao, G. (2016). Spontaneous micro-expression spotting via geometric deformation modeling. Computer Vision and Image Understanding, 147, 87–94.

    Google Scholar 

  80. Xiaohua, H., Wang, S.-J., Liu, X., Zhao, G., Feng, X., Pietikainen, M. (2017). Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing.

  81. Xu, F., Zhang, J., Wang, J.Z. (2017). Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 8(2), 254–267.

    Google Scholar 

  82. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Fu, X. (2014). Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS one, 9(1), e86041.

    Google Scholar 

  83. Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., Fu, X. (2013). How fast are the leaked facial expressions: the duration of micro-expressions. Journal of Nonverbal Behavior, 37(4), 217–230.

    Google Scholar 

  84. Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., Fu, X. (2013). Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1–7): IEEE.

  85. Yang, J., Zhang, L., Xu, Y., Yang, J.-Y. (2012). Beyond sparsity the role of l1-optimizer in pattern classification. Pattern Recognition, 45(3), 1104–1118.

    MATH  Google Scholar 

  86. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A. (2018). Self-attention generative adversarial networks. arXiv:1805.08318.

  87. Zhang, K., Zhang, Z., Li, Z., Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10), 1499–1503.

    Google Scholar 

  88. Zhang, Z., Chen, T., Meng, H., Liu, G., Fu, X. (2018). Smeconvnet a convolutional neural network for spotting spontaneous facial micro-expression from long videos. IEEE Access, 6, 71143–71151.

    Google Scholar 

  89. Zhao, G., & Pietikainen, M. (2007). Dynamic texture recognition using local binary patterns with an application to facial expressions, (Vol. 6.

  90. Zheng, H., Geng, X., Yang, Z. (2016). A relaxed k-svd algorithm for spontaneous micro-expression recognition. In Pacific rim international conference on artificial intelligence (pp. 692–699): Springer.

  91. Zhong, Z., Zheng, L., Zheng, Z., Li, S., Yang, Y. (2018). Camera style adaptation for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5157–5166).

  92. Zhou, Z., Zhao, G., Pietikäinen, M. (2011). Towards a practical lipreading system. In CVPR 2011 (pp. 137–144): IEEE.

  93. Zhu, Q., Yeh, M.-C., Cheng, K.-T., Avidan, S. (2006). Fast human detection using a cascade of histograms of oriented gradients. In IEEE computer society conference on computer vision and pattern recognition (CVPR’06), (Vol. 2 p. 2006): IEEE.

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Feng Chia University, Taichung, Taiwan

    Sze-Teng Liong

  2. Research Center For Healthcare Industry Innovation, National Taipei University of Nursing and Health Sciences, Taipei, Taiwan

    Y. S. Gan

  3. School of Electrical and Computing Engineering, Xiamen University Malaysia, Jalan Sunsuria, Sepang, Selangor, Malaysia

    Danna Zheng, Shu-Meng Li, Hao-Xuan Xu, Han-Zhe Zhang & Ran-Ke Lyu

  4. School of Informatics, Xiamen University, Xiamen, China

    Kun-Hong Liu

Authors
  1. Sze-Teng Liong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Y. S. Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Danna Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shu-Meng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao-Xuan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Han-Zhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ran-Ke Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kun-Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Y. S. Gan or Kun-Hong Liu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was funded by Ministry of Science and Technology (MOST) (Grant Number: MOST108-2221-E-035-066- and Grant Number: MOST 108-2218-E-227-002-), National Natural Science Foundation of China (No. 61772023) and National Key R&D Program of China (No. 2019QY1803)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liong, ST., Gan, Y.S., Zheng, D. et al. Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System. J Sign Process Syst 92, 705–725 (2020). https://doi.org/10.1007/s11265-020-01523-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-020-01523-4

Keywords

Search

Navigation