nach oben

Neural Computing and Applications

Erschienen in:

07.03.2022 | Review

Weakly-supervised temporal action localization: a survey

verfasst von: AbdulRahman Baraka, Mohd Halim Mohd Noor

Erschienen in: Neural Computing and Applications | Ausgabe 11/2022

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Temporal Action Localization (TAL) is an important task of various computer vision topics such as video understanding, summarization, and analysis. In the real world, the videos are long untrimmed and contain multiple actions, where the temporal boundaries annotations are required in the fully-supervised learning setting for classification and localization tasks. Since the annotation task is costly and time-consuming, the trend is moving toward the weakly-supervised setting, which depends on the video-level labels only without any additional information, and this approach is called weakly-supervised Temporal Action Localization (WTAL). In this survey, we review the concepts, strategies, and techniques related to the WTAL in order to clarify all aspects of the problem and review the state-of-the-art frameworks of WTAL according to their challenges. Furthermore, a comparison of models’ performance and results based on benchmark datasets is presented. Finally, we summarize the future works to allow the researchers to improve the model's performance.

Vorheriger Artikel Anomaly detection of industrial multi-sensor signals based on enhanced spatiotemporal features

Nächster Artikel Research on water temperature prediction based on improved support vector regression

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Lin X, Shou Z, Chang S-F (2019) Towards train-test consistency for semi-supervised temporal action localization, [Online]. Available: http://arxiv.org/abs/1910.11285

Ma F et al (2020) SF-Net: single-frame supervision for temporal action localization, [Online]. Available: http://arxiv.org/abs/2003.06845

Ding X, Wang N, Gao X, Li J, Wang X, and Liu T (2020) Weakly supervised temporal action localization with segment-level labels, 1(c), [Online]. Available: http://arxiv.org/abs/2007.01598

Sun C, Shetty S, Sukthankar R, and Nevatia R (2015) Temporal localization of fine-grained actions in videos by domain transfer from web images. In: MM 2015 - Proc. 2015 ACM Multimed. Conf., pp. 371–380. https://doi.org/10.1145/2733373.2806226

Park J, Lee J, Jeon S, Kim S, and Sohn K (2019) Graph regularization network with semantic affinity for weakly-supervised temporal action localization. In: Proceedings - international conference on image processing, ICIP, 2019:3701–3705. https://doi.org/10.1109/ICIP.2019.8803589

Nguyen P, Han B, Liu T, and Prasad G (2018) Weakly supervised action localization by sparse temporal pooling network. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, pp. 6752–6761. https://doi.org/10.1109/CVPR.2018.00706

Narayan S, Cholakkal H, Khan FS, and Shao L (2019) 3C-Net: category count and center loss for weakly-supervised action localization. Proc IEEE Int Conf Comput Vis 2019: 8678–8686. https://doi.org/10.1109/ICCV.2019.00877

Wang C, Cai H, Zou Y, and Xiong Y (2021) RGB stream is enough for temporal action detection, [Online]. Available: http://arxiv.org/abs/2107.04362

Alwassel H, Giancola S, and Ghanem B (2020) TSP: temporally-sensitive pretraining of video encoders for localization tasks, [Online]. Available: http://arxiv.org/abs/2011.11479

10.

Nawhal M and Mori G (2021) Activity graph transformer for temporal action localization, [Online]. Available: http://arxiv.org/abs/2101.08540

11.

Alwassel H, Pardo A, Heilbron FC, Thabet A, and Ghanem B (2019) RefineLoc: iterative refinement for weakly-supervised action localization, [Online]. Available: http://arxiv.org/abs/1904.00227

12.

Bojanowski P et al (2014) Weakly supervised action labeling in videos under ordering constraints. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 8693 LNCS, no. Part 5, pp. 628–643. https://doi.org/10.1007/978-3-319-10602-1_41

13.

Huang DA, Fei-Fei L, and Niebles JC (2016) Connectionist temporal modeling for weakly supervised action labelling. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9908 LNCS, pp. 137–153. https://doi.org/10.1007/978-3-319-46493-0_9

14.

Yang H, He X, Porikli F (2018) One-shot action localization by learning sequence matching network. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2018.00157CrossRef

15.

Chéron G, Alayrac JB, Laptev I, Schmid C (2018) A flexible model for training action localization with varying levels of supervision. Adv Neural Inf Process Syst 2018:942–953

16.

Xia H, Zhan Y (2020) A survey on temporal action localization. IEEE Access 8:70477–70487. https://doi.org/10.1109/ACCESS.2020.2986861CrossRef

17.

Zhou ZH (2018) A brief introduction to weakly supervised learning. Natl Sci Rev 5(1):44–53. https://doi.org/10.1093/nsr/nwx106CrossRef

18.

Kolesnikov A and Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Lecture notes in computer science (including subseries Lecture notes in artificial intelligence and lecture notes in bioinformatics), 9908 LNCS, pp. 695–711. https://doi.org/10.1007/978-3-319-46493-0_42

19.

Carbonneau MA, Cheplygina V, Granger E, Gagnon G (2018) Multiple instance learning: a survey of problem characteristics and applications. Pattern Recognit 77:329–353. https://doi.org/10.1016/j.patcog.2017.10.009CrossRef

20.

Vanwinckelen G, Tragante do VO, Fierens D, Blockeel H (2016) Instance-level accuracy versus bag-level accuracy in multi-instance learning. Data Min Knowl Discov 30(2):313–341. https://doi.org/10.1007/s10618-015-0416-zMathSciNetCrossRefMATH

21.

Wang L, Xiong Y, Lin D, and Van Gool L (2017) UntrimmedNets for weakly supervised action recognition and detection. In: Proc - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017, 2017: 6402–6411. https://doi.org/10.1109/CVPR.2017.678.

22.

Xu Y et al (2019) Segregated temporal assembly recurrent networks for weakly supervised multiple action detection. Proc AAAI Conf Artif Intell 33:9070–9078. https://doi.org/10.1609/aaai.v33i01.33019070CrossRef

23.

Lee P, Uh Y, and Byun H (2019) Background suppression network for weakly-supervised temporal action localization. https://doi.org/10.1609/aaai.v34i07.6793

24.

Paul S, Roy S, and Roy-Chowdhury AK (2018) W-TALC: weakly-supervised temporal activity localization and classification. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 11208 LNCS, pp. 588–607. https://doi.org/10.1007/978-3-030-01225-0_35

25.

Lee P, Wang J, Lu Y, and Byun H (2020) Background modeling via uncertainty estimation for weakly-supervised action localization. pp. 1–12, [Online]. Available: http://arxiv.org/abs/2006.07006

26.

Rashid M, Kjellstrom H, and Lee YJ (2020) Action graphs: weakly-supervised action localization with graph convolution networks. In: Proceedings - 2020 IEEE winter conference on applications of computer vision, WACV 2020, pp. 604–613. https://doi.org/10.1109/WACV45572.2020.9093404

27.

Shi B, Dai Q, Mu Y, and Wang J (2020) Weakly-supervised action localization by generative attention modelling. pp. 1006–1016. https://doi.org/10.1109/cvpr42600.2020.00109

28.

Schindler K and Van Gool L (2008) Action snippets: How many frames does human action recognition require?. In: 26th IEEE Conf Comput Vis Pattern Recognition, CVPR. https://doi.org/10.1109/CVPR.2008.4587730

29.

Liu D, Jiang T, and Wang Y (2019) Completeness modeling and context separation for weakly supervised temporal action localization. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 2019:1298–1307. https://doi.org/10.1109/CVPR.2019.00139

30.

Singh KK and Lee YJ (2017) Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization. In: Proc IEEE Int Conf Comput Vis, 2017: 3544–3553. https://doi.org/10.1109/ICCV.2017.381

31.

Shou Z, Gao H, Zhang L, Miyazawa K, and Chang SF (2018) AutoLoc: weakly-supervised temporal action localization in untrimmed videos. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 11220 LNCS, pp. 162–179. https://doi.org/10.1007/978-3-030-01270-0_10

32.

Liu Z et al (2019) Weakly supervised temporal action localization through contrast based evaluation networks. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00400CrossRef

33.

Zeng R, Gan C, Chen P, Huang W, Wu Q, Tan M (2019) Breaking winner-takes-all: iterative-winners-out networks for weakly supervised temporal action localization. IEEE Trans Image Process 28(12):5797–5808. https://doi.org/10.1109/TIP.2019.2922108MathSciNetCrossRefMATH

34.

Su H, Zhao X, and Lin T (2019) Cascaded pyramid mining network for weakly supervised temporal action localization. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 11362 LNCS, pp. 558–574. https://doi.org/10.1007/978-3-030-20890-5_36

35.

Su H, Zhao X, Lin T, and Fei H (2018) Weakly supervised temporal action detection with shot-based temporal pooling network. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 11304 LNCS, pp. 426–436. https://doi.org/10.1007/978-3-030-04212-7_37

36.

Russakovsky O et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-yMathSciNetCrossRef

37.

Kay W et al (2017) The kinetics human action video dataset, [Online]. Available: http://arxiv.org/abs/1705.06950

38.

Zach C, Pock T, and Bischof H (2007) A duality based approach for realtime TV-L1 optical flow. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 4713 LNCS, pp. 214–223. https://doi.org/10.1007/978-3-540-74936-3_22

39.

Soomro K, Zamir AR, and Shah M (2012) UCF101: a dataset of 101 human actions classes from videos in the wild, [Online]. Available: http://arxiv.org/abs/1212.0402

40.

Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2011.6126543CrossRef

41.

Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Adv Neural Inf Process Syst 1(January):568–576

42.

Wang L et al (2016) Temporal segment networks: Towards good practices for deep action recognition. Lect Notes Comput Sci (including Subser. Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 9912 LNCS, pp. 20–36. https://doi.org/10.1007/978-3-319-46484-8_2

43.

Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Li FF (2014) Large-scale video classification with convolutional neural networks. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2014.223CrossRef

44.

Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.213CrossRef

45.

Dai X, Singh B, Zhang G, Davis LS, Chen YQ (2017) Temporal context network for activity localization in videos. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2017.610CrossRef

46.

Zhong JX, Li N, Kong W, Zhang T, Li TH, and Li G (2018) Step-by-step erasion, one-by-one collection: a weakly supervised temporal action detector. In: MM 2018 - Proceedings of the 2018 ACM multimedia conference, no. 2014, pp. 35–44. https://doi.org/10.1145/3240508.3240511

47.

Huang L, Huang Y, Ouyang W, and Wang L (2020) Relational prototypical network for weakly supervised temporal action localization. Aaai

48.

Carreira J and Zisserman A (2017) Quo Vadis, action recognition? A new model and the kinetics dataset. In: Proc. - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017, 2017: 4724–4733. https://doi.org/10.1109/CVPR.2017.502

49.

Ioffe S and Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: 32nd Int Conf Mach Learn. ICML 2015, 1:448–456

50.

Nguyen P, Ramanan D, Fowlkes C (2019) Weakly-supervised action localization with background modeling. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00560CrossRef

51.

Kang Z, Wang L, Liu Z, Zhang Q, Zheng N (2019) Extracting action sensitive features to facilitate weakly-supervised action localization. IFIP Adv Inform Commun Technol. https://doi.org/10.1007/978-3-030-19823-7_15CrossRef

52.

Zhai Y, Wang L, Liu Z, Zhang Q, Hua G, Zheng N (2019) Action coherence network for weakly supervised temporal action localization. Proc - Int Conf Image Process. https://doi.org/10.1109/ICIP.2019.8803447CrossRef

53.

Zhang C et al (2019) Adversarial seeded sequence growing for weakly-supervised temporal action localization. In: MM 2019 - Proc 27th ACM Int Conf Multimed, pp. 738–746. https://doi.org/10.1145/3343031.3351044

54.

Yuan Y, Lyu Y, Shen X, Tsang IW, and Yeung DY (2019) Marginalized average attentional network for weakly-supervised learning. In: 7th Int Conf Learn. Represent. ICLR 2019, pp. 1–19

55.

Min K and Corso JJ (2020) Adversarial background-aware loss for weakly-supervised temporal activity localization. ECCV 2020, [Online]. Available: http://arxiv.org/abs/2007.06643

56.

Nair V and Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: ICML 2010 - Proceedings, 27th Int Conf Mach Learn, pp. 807–814

57.

Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.319CrossRef

58.

Narayan S, Cholakkal H, Hayat M, Khan FS, Yang MH, and Shao L (2020) D2-Net: weakly-supervised action localization via discriminative embeddings and denoised activations. arXiv, no. December

59.

Islam A and Radke RJ (2020) Weakly supervised temporal action localization using deep metric learning. In: Proceedings - 2020 IEEE winter conference on applications of computer vision, WACV 2020, pp. 536–545. https://doi.org/10.1109/WACV45572.2020.9093620

60.

Idrees H et al (2017) The THUMOS challenge on action recognition for videos ‘in the wild.’ Comput Vis Image Underst 155:1–23. https://doi.org/10.1016/j.cviu.2016.10.018CrossRef

61.

Heilbron FC, Escorcia V, Ghanem B, Niebles JC (2015) ActivityNet: a large-scale video benchmark for human activity understanding. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2015.7298698CrossRef

62.

Sigurdsson GA, Varol G, Wang X, Farhadi A, Laptev I, and Gupta A (2016) Hollywood in homes: crowdsourcing data collection for activity understanding. Lect Notes Comput Sci (including Subser. Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 9905 LNCS, pp. 510–526. https://doi.org/10.1007/978-3-319-46448-0_31

63.

Zhao H, Torralba A, Torresani L, Yan Z (2019) HACS: Human action clips and segments dataset for recognition and temporal localization. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00876CrossRef

64.

Huang Z, Wang X, Wang JJ, Liu W, and Wang JJ (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp. 7014–7023. https://doi.org/10.1109/CVPR.2018.00733

65.

Islam A, Long C, and Radke RJ (2021) A hybrid attention mechanism for weakly-supervised temporal action localization, no. Mil, [Online]. Available: http://arxiv.org/abs/2101.00545

66.

Ge Y, Qin X, Yang D, Jagersand M (2021) Deep snippet selective network for weakly supervised temporal action localization. Pattern Recognit 110:107686. https://doi.org/10.1016/j.patcog.2020.107686CrossRef

67.

Yu T, Ren Z, Li Y, Yan E, Xu N, Yuan J (2019) Temporal structure mining for weakly supervised action detection. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00562CrossRef

68.

Hendrycks D and Gimpel K (2016) A baseline for detecting misclassified and out-of-distribution examples in neural networks. 5th Int Conf Learn Represent ICLR 2017 – Conf Track Proc, pp. 1–12. [Online]. Available: http://arxiv.org/abs/1610.02136

69.

Hou R, Sukthankar R, and Shah M (2017) Real-time temporal action localization in untrimmed videos by sub-action discovery. Br Mach Vis Conf, BMVC . https://doi.org/10.5244/c.31.91

70.

Heidarivincheh F, Mirmehdi M, and Damen D (2019) Weakly-supervised completion moment detection using temporal attention. Proc. - 2019 Int Conf Comput Vis Work. ICCVW 2019, pp. 1188–1196. https://doi.org/10.1109/ICCVW.2019.00150

71.

Luo Z et al (2020) Weakly-supervised action localization with expectation-maximization multi-instance learning. Lect. Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 12374 LNCS, no. Mil, pp. 729–745. https://doi.org/10.1007/978-3-030-58526-6_43

72.

Zhang XY, Li C, Shi H, Zhu X, Li P, Dong J (2020) AdapNet: adaptability decomposing encoder-decoder network for weakly supervised action recognition and localization. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2019.2962815CrossRef

73.

Snell J, Swersky K, Zemel R (2017) Prototypical networks for few-shot learning. Adv Neural Inform Process Syst 2017:4078–4088

74.

Kingma DP and J. L. Ba (2015) Adam: a method for stochastic optimization, 3rd Int Conf Learn Represent. ICLR 2015 - Conf Track Proc, pp. 1–15

75.

Zhao Y, Xiong Y, Wang L, Wu Z, Tang X, Lin D (2020) Temporal action detection with structured segment networks. Int J Comput Vis 128(1):74–95. https://doi.org/10.1007/s11263-019-01211-2MathSciNetCrossRefMATH

76.

Defferrard M, Bresson X, and Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. Adv Neural Inform Process Syst, no. Nips, pp. 3844–3852

77.

Pang J, Cheung G (2017) Graph laplacian regularization for image denoising: analysis in the continuous domain. IEEE Trans Image Process 26(4):1770–1785. https://doi.org/10.1109/TIP.2017.2651400MathSciNetCrossRefMATH

78.

Zhai Y, Wang L, Tang W, Zhang Q, and Yuan J (2020) Two-stream consensus network for weakly-supervised temporal action localization. In: Proc Eur. Conf Comput Vis, no. Mil, pp. 1–17

79.

Gong G, Wang X, Mu Y, Tian Q (2020) Learning temporal co-attention models for unsupervised video action localization. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR42600.2020.00984CrossRef

Titel: Weakly-supervised temporal action localization: a survey
verfasst von: AbdulRahman Baraka
Mohd Halim Mohd Noor
Publikationsdatum: 07.03.2022
Verlag: Springer London
Erschienen in: Neural Computing and Applications / Ausgabe 11/2022
Print ISSN: 0941-0643
Elektronische ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-022-07102-x

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 11/2022

A new PDE learning model for image denoising

Joining features by global guidance with bi-relevance trihard loss for person re-identification

Research on water temperature prediction based on improved support vector regression

Parallel tensor factorization for relational learning

Graph alternate learning for robust graph neural networks in node classification

A novel emergency situation awareness machine learning approach to assess flood disaster risk based on Chinese Weibo

Premium Partner