Top

Multimedia Systems

Published in:

06-06-2023 | Regular Paper

Micro-expression spotting network based on attention and one-dimensional convolutional sliding window

Authors: Hongbo Xing, Guanqun Zhou, Shusen Yuan, Youjun Jiang, Pinyong Geng, Yewen Cao, Yujun Li, Lei Chen

Published in: Multimedia Systems | Issue 4/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In the field of computer vision, the research on micro-expression (ME) can be divided into two main tasks: ME spotting and ME recognition. ME spotting refers to finding the occurrence and interval of ME from video stream, which is an indispensable module for automatic ME analysis. In this paper, aiming at finding of the inaccurate location of MEs in long videos, we proposed a ME spotting network based on attention mechanism and one-dimensional (1D) convolution sliding window. In our proposed scheme, convolutional neural network (CNN), Bi-directional Long Short-Term Memory (BI-LSTM), and 1D convolution are used to extract features. The attention mechanism is used to highlight the key frames. 1D convolution with sliding window is applied to detect feature intervals, which are further combined with the intervals and judged as MEs to obtain the final ME spotting result. Simulation was done on CAS(ME)2 dataset. It is shown that the proposed algorithm outperforms other superior algorithms in terms of effectiveness.

previous article HRCutBlur Augment: effectively enhancing data diversity for image super-resolution

next article Multiple forgery detection in digital video based on inconsistency in video quality assessment attributes

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Haggard, E.A., Isaacs, K.S.: Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Meth Res Psychoth. (1966). https://doi.org/10.1007/978-1-4684-6045-2_14CrossRef

Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969). https://doi.org/10.1080/00332747.1969.11023575CrossRef

Zhang, M., Zhao, K., Qu, F., Li, K., Fu, X.: brain activation in contrasts of micro-expression following emotional contexts. Front. Neurosci. 14, 329–329 (2020). https://doi.org/10.3389/fnins.2020.00329CrossRef

Curtis, D.A.: Deception detection and emotion recognition: Investigating FACE software. Psychother Res. 31(6), 802–816 (2021). https://doi.org/10.1080/10503307.2020.1836424CrossRef

Datz, F., Wong, G., Löffler-Stastka, H.: Interpretation and Working through Contemptuous Facial Micro-Expressions Benefits the Patient-Therapist Relationship. Int J Environ Res Public Health 16(24), 4901 (2019). https://doi.org/10.3390/ijerph16244901CrossRef

Jupe, L.M., Keatley, D.A.: Airport artificial intelligence can detect deception: or am i lying? Secur. J. 33(4), 622–635 (2019). https://doi.org/10.1057/s41284-019-00204-7CrossRef

Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., Fu, X.: CAS(ME)^2: A Database for Spontaneous Macro-expression and Micro-expression Spotting and Recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2018). https://doi.org/10.1109/taffc.2017.2654440CrossRef

Tran, T., Vo, Q., Hong, X., Li, X., Zhao, G.: Micro-expression spotting: A new benchmark. Neurocomputing (Amsterdam) 443, 356–368 (2021). https://doi.org/10.1016/j.neucom.2021.02.022CrossRef

Moilanen, A., Zhao, G., Pietikainen, M.: Spotting rapid facial movements from videos using appearance-based feature difference analysis. Int Conf Pattern Recog (2014). https://doi.org/10.1109/ICPR.2014.303CrossRef

10.

Davison, A.K., Yap, M.H., Lansley, C.: Micro-facial movement detection using individualised baselines and histogram-based descriptors. Int Conf Syst and Cybernet. (2015). https://doi.org/10.1109/smc.2015.326CrossRef

11.

Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikainen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. Trans Affect Comp. 9(4), 563–577 (2017). https://doi.org/10.1109/TAFFC.2017.2667642CrossRef

12.

Wang, S.-J., Wu, S., Qian, X., Li, J., Fu, X.: A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing 230, 382–389 (2017). https://doi.org/10.1016/j.neucom.2016.12.034CrossRef

13.

Beh, K.X., Goh, K.M.: Micro-Expression spotting using facial landmarks. In CSPA. (2019). https://doi.org/10.1109/CSPA.2019.8696059CrossRef

14.

He, Y., Xu, Z., Ma, L., Li, H.: Micro-expression spotting based on optical flow features. Patt recog lett 163, 57–63 (2022)CrossRef

15.

Xia, Z., Feng, X., Peng, J., Peng, X., Zhao, G.: Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Underst. 147, 87–94 (2016). https://doi.org/10.1016/j.cviu.2015.12.006CrossRef

16.

Tran, T.K., Hong, X., Zhao, G.: Sliding Window Based Micro-expression Spotting: A Benchmark. Adv Conc Intell Vision Syst. (2017). https://doi.org/10.1007/978-3-319-70353-4_46CrossRef

17.

Zhang, Z., Chen, T., Meng, H., Liu, G., Fu, X.: Smeconvnet a convolutional neural network for spotting spontaneous facial micro-expression from long videos. IEEE Access. 6, 7113 (2018)

18.

Verburg, M., Menkovski, V.: Micro-expression detection in long videos using optical flow and recurrent neural networks. In Fg (2019). https://doi.org/10.1109/FG.2019.8756588CrossRef

19.

Tran, T.-K., Vo, Q.-N., Hong, X., Zhao, G.: Dense prediction for micro-expression spotting based on deep sequence model. Electr Imag. 8, 401–411 (2019). https://doi.org/10.2352/issn.2470-1173.2019.8.imawm-401CrossRef

20.

Chen, C.R., Fan, Q., Panda, R.: CrossViT: cross-attention multi-scale vision transformer for image classification. ICCV. (2021). https://doi.org/10.1109/ICCV48922.2021.00041CrossRef

21.

Mao, X., Qi, G., Chen, Y., et al.: Towards robust vision transformer. EEE/CVF Conf Comp Vis Patt Recog. (2022). https://doi.org/10.1109/CVPR52688.2022.01173CrossRef

22.

Zhou, Y., Song, Y., Chen, L., Chen, Y., Ben, X., Cao, Y.: A novel micro-expression detection algorithm based on BERT and 3DCNN. Image Vision Comp 119, 104378 (2022). https://doi.org/10.1016/j.imavis.2022.104378CrossRef

23.

Li, Y., Yao, T., Pan, Y., Mei, T.: Contextual transformer networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 45(2), 1489–1500 (2023). https://doi.org/10.1109/TPAMI.2022.3164083CrossRef

24.

Yao, T., Li, Y., Pan, Y., Wang, Y., Zhang, X., Mei, T.: Dual vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. (2023). https://doi.org/10.1109/TPAMI.2023.3268446CrossRef

25.

Krizhevsky, A., Sutskeve, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386CrossRef

26.

Wang, C., Wang, Z.: Progressive Multi-Scale Vision Transformer for Facial Action Unit Detection. Front Neurorobot. 15, 824592 (2022). https://doi.org/10.3389/fnbot.2021.824592CrossRef

27.

King, D.E.: Dlib-ml: a machine learning toolkit. J Mach Learn Res 10, 1755–1758 (2009)

28.

Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. Lect Not Comp Sci (2003). https://doi.org/10.1007/3-540-45103-x_50CrossRefMATH

Title: Micro-expression spotting network based on attention and one-dimensional convolutional sliding window
Authors: Hongbo Xing
Guanqun Zhou
Shusen Yuan
Youjun Jiang
Pinyong Geng
Yewen Cao
Yujun Li
Lei Chen
Publication date: 06-06-2023
Publisher: Springer Berlin Heidelberg
Published in: Multimedia Systems / Issue 4/2023
Print ISSN: 0942-4962
Electronic ISSN: 1432-1882
DOI: https://doi.org/10.1007/s00530-023-01120-y

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 4/2023

Attentional weighting strategy-based dynamic GCN for skeleton-based action recognition

Face attribute recognition via end-to-end weakly supervised regional location

Prototype-based semantic consistency learning for unsupervised 2D image-based 3D shape retrieval

FDS_2D: rethinking magnitude-phase features for DeepFake detection

Residual guided coordinate attention for selection channel aware image steganalysis

Multiple forgery detection in digital video based on inconsistency in video quality assessment attributes