Top

International Journal of Computer Vision

Published in:

01-03-2015

Continuous Action Recognition Based on Sequence Alignment

Authors: Kaustubh Kulkarni, Georgios Evangelidis, Jan Cech, Radu Horaud

Published in: International Journal of Computer Vision | Issue 1/2015

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Continuous action recognition is more challenging than isolated recognition because classification and segmentation must be simultaneously carried out. We build on the well known dynamic time warping framework and devise a novel visual alignment technique, namely dynamic frame warping (DFW), which performs isolated recognition based on per-frame representation of videos, and on aligning a test sequence with a model sequence. Moreover, we propose two extensions which enable to perform recognition concomitant with segmentation, namely one-pass DFW and two-pass DFW. These two methods have their roots in the domain of continuous recognition of speech and, to the best of our knowledge, their extension to continuous visual action recognition has been overlooked. We test and illustrate the proposed techniques with a recently released dataset (RAVEL) and with two public-domain datasets widely used in action recognition (Hollywood-1 and Hollywood-2). We also compare the performances of the proposed isolated and continuous recognition algorithms with several recently published methods.

previous article Utilizing Local Phase Information to Remove Rain from Video

next article Tractable Algorithms for Robust Model Estimation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

https://team.inria.fr/perception/research/car/

http://perception.inrialpes.fr/datasets/Ravel/

Alameda-Pineda, X., Sanchez-Riera, J., Wienke, J., Franc, V., Cech, J., Kulkarni, K., et al. (2013). RAVEL: An annotated corpus for training robots with audiovisual abilities. Journal on Multimodal User Interfaces, 7(1–2), 79–91.CrossRef

Alon, J., Athitsos, V., Yuan, Q., & Sclaroff, S. (2009). A unified framework for gesture recognition and spatiotemporal gesture segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(9), 1685–1699.CrossRef

Blackburn, J., & Ribeiro, E. (2007). Human motion recognition using isomap and dynamic time warping. Human motion-understanding, modeling, capture and animation (pp. 285–298). Berlin: Springer.CrossRef

Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. New York, NY: Cambridge University Press.CrossRefMATH

Brendel, W., & Todorovic, S. (2010). Activities as time series of human postures. In N. Paragios (Ed.), Computer Vision-ECCV 2010 (pp. 721–734). Berlin: Springer.CrossRef

Chen, S. S., Donoho, D. L., & Saunders, M. A. (2001). Atomic decomposition by basis pursuit. SIAM Rev, 43(1), 129–159.CrossRefMATHMathSciNet

Csurka, G., Dance, C. R., Fan, L., Willamowski, J., & Bray, C. (2004). Visual categorization with bags of keypoints. In ECCV Workshop on Statistical Learning in Computer Vision.

Escalera, S., Gonzàlez, J., Baró, X., Reyes, M., Lopes, O., Guyon, I., Athitsos, V., & Escalante, H. J. (2013). Multi-modal gesture recognition challenge 2013: Dataset and results. In ChaLearn Multi-modal Gesture Recognition Grand Challenge and Workshop, 15th ACM International Conference on Multimodal Interaction.

Evangelidis, G. D., & Psarakis, E. Z. (2008). Parametric image alignment using enhanced correlation coefficient maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(10), 1858–1865.CrossRef

Gales, M., & Young, S. (2008). The application of hidden Markov models in speech recognition. Foundations and Trends in Signal Processing, 1(3), 195–304.CrossRef

Gill, P. R., Wang, A., & Molnar, A. (2011). The in-crowd algorithm for fast basis pursuit denoising. IEEE Transactions on Signal Processing, 59(10), 4595–4605.CrossRefMathSciNet

Gong, D., & Medioni, G. (2011) Dynamic manifold warping for view invariant action recognition. In IEEE International Conference on Computer Vision, (pp. 571–578). IEEE.

Hienz, H., Bauer, B., & Kraiss, K. F. (1999). HMM-based continuous sign language recognition using stochastic grammars. In A. Braffort, R. Gherbi, S. Gibet, D. Teil, & J. Richardson (Eds.), Gesture-based communication in human-computer interaction (Vol. 1739, pp. 185–196)., Lecture Notes in Computer Science Berlin: Springer.CrossRef

Hoai, M., Lan, Z. Z., & De la Torre, F. (2011). Joint segmentation and classification of human actions in video. In 2011 IEEE Conference on Computer Vision and Pattern Recognition CVPR. (pp. 3265–3272). IEEE.

Ikizler, N., & Duygulu, P. (2009). Histogram of oriented rectangles: A new pose descriptor for human action recognition. Image and Vision Computing, 27(10), 1515–1526.CrossRef

Jain, M., Jégou, H., & Bouthémy, P. (2013). Better exploiting motion for better action recognition. In Computer Vision and Pattern Recognition, (pp. 2555–2562). IEEE.

Jiang, Y. G., Dai, Q., Xue, X., Liu, W., & Ngo, C. W. (2012). Trajectory-based modeling of human actions with motion reference points. In European Conference on Computer Vision, (pp. 425–438). Berlin :Springer.

Kulkarni, K., Cherla, S., Kale, A., & Ramasubramanian, V. (2008). A framework for indexing human actions in video. In The 1st International Workshop on Machine Learning for Vision-based Motion Analysis-MLVMA’08.

Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) Learning realistic human actions from movies. In IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, (pp. 1–8). IEEE.

Lee, C., & Rabiner, L. (1989). A frame-synchronous network search algorithm for connected word recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 37(11), 1649–1658.CrossRef

Liang, R., & Ouhyoung, M. (1998). A real-time continuous gesture recognition system for sign language. In Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, (pp. 558–567). IEEE.

Lv, F., & Nevatia, R. (2006). Recognition and segmentation of 3-d human action using HMM and multi-class AdaBoost. In European Conference on Computer Vision, (pp. 359–372). Berlin: Springer.

Lv, F., & Nevatia, R. (2007). Single view human action recognition using key pose matching and Viterbi path searching. In Computer Vision and Pattern Recognition, 2007. CVPR’07, (pp. 1–8). IEEE.

Manning, C., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge: Cambridge University Press.CrossRefMATH

Marszalek, M., Laptev, I., & Schmid, C. (2009) Actions in context. In IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2929–2936). IEEE.

Morency, L., Quattoni, A., & Darrell, T. (2007). Latent-dynamic discriminative models for continuous gesture recognition. In Computer Vision and Pattern Recognition, (pp. 1–8). IEEE.

Mueller, M. (2007). Dynamic time warping. Information retrieval for music and motion (pp. 69–84). Berlin: Springer.CrossRef

Ney, H. (1984). The use of a one-stage dynamic programming algorithm for connected word recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 32(2), 263–271.CrossRef

Ney, H., & Ortmanns, S. (1999). Dynamic programming search for continuous speech recognition. IEEE Signal Processing Magazine, 16(5), 64–83.CrossRef

Ning, H., Xu, W., Gong, Y., Huang, T. (2008). Latent pose estimator for continuous action recognition. In European Conference on Computer Vision, (pp. 419–433). Springer.

Rabiner, L., & Juang, B. (1993). Fundamentals of speech recognition. Salt Lake: Prentice hall.

Sakoe, H. (1979). Two-level DP-matching - a dynamic programming-based pattern matching algorithm for connected word recognition. IEEE Transactions on Acoustic, Speech, and Signal Processing, 27(6), 588–595.CrossRef

Sanchez-Riera, J., Cech, J., Horaud, R. P. (2012). Action recognition robust to background clutter by using stereo vision. In The Fourth International Workshop on Video Event Categorization, Tagging and Retrieval, LNCS: Springer.

Shi, Q., Wang, L., Cheng, L., & Smola, A. (2011). Discriminative human action segmentation and recognition using SMMs. IJCV, 93(1), 22–32.CrossRefMATH

Sigal, L., Balan, A., & Black, M. (2010). Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision, 87(1), 4–27.CrossRef

Sivic, J., & Zisserman, A. (2009). Efficient visual search of videos cast as text retrieval. IEEE Transactions on PAMI, 31(4), 591–606.CrossRef

Sminchisescu, C., Kanaujia, A., & Metaxas, D. N. (2006). Conditional models for contextual human motion recognition. CVIU, 104(2–3), 210–220.

Solmaz, B., Assari, S. M., & Shah, M. (2013). Classifying web videos using a global video descriptor. Machine vision and applications, 24(7), 1473–1485.CrossRef

Starner, T., Weaver, J., & Pentland, A. (1998). Real-time american sign language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1371–1375.CrossRef

Tropp, J. A., & Gilbert, A. C. (2007). Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on Information Theory, 53(12), 4655–4666.CrossRefMATHMathSciNet

Ullah, M. M., Parizi, S. N,, Laptev, I. (2010). Improving bag-of-features action recognition with non-local cues. In British Machine Vision Conference. (Vol. 10, pp. 95–101).

Vail, D., Veloso, M., & Lafferty, J. (2007). Conditional random fields for activity recognition. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, (p. 235). ACM.

Vintsyuk, T. (1971). Element-wise recognition of continuous speech composed of words from a specified dictionary. Cybernetics and Systems Analysis, 7(2), 361–372.

Vogler, C., & Metaxas, D. (1998). ASL recognition based on a coupling between HMMs and 3D motion analysis. In Sixth International Conference on Computer Vision, (pp. 363–369).

Vogler, C., & Metaxas, D. (2001). A framework for recognizing the simultaneous aspects of american sign language. Computer Vision and Image Understanding, 81(3), 358–384.CrossRefMATH

Wang, H., & Schmid, C. (2013). Action recognition with improved trajectories. In International Conference on Computer Vision, (pp. 3551–3558). IEEE.

Young, S., Russell, N. H., & Thornton, J. (1989). Token passing: a simple conceptual model for connected speech recognition systems. Technical Report 38, University of Cambridge, Department of Engineering.

Young, S., Woodland, P., & Byrne, W. (1993). HTK: Hidden Markov model toolkit v1. 5. Technical Report, University of Cambridge, Department of Engineering.

Young, S., Evermann, G., Kershaw, D., Moore, G., Odell, J., Ollason, D., et al. (2009). The HTK book. Technical Report: University of Cambridge, Department of Engineering.

Zhou, F., & la Torre, F. D. (2009). Canonical time warping for alignment of human behavior. In Advances in Neural Information Processing Systems, (pp. 2286–2294).

Title: Continuous Action Recognition Based on Sequence Alignment
Authors: Kaustubh Kulkarni
Georgios Evangelidis
Jan Cech
Radu Horaud
Publication date: 01-03-2015
Publisher: Springer US
Published in: International Journal of Computer Vision / Issue 1/2015
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-014-0758-9

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 1/2015

Utilizing Local Phase Information to Remove Rain from Video

Robust Temporally Coherent Laplacian Protrusion Segmentation of 3D Articulated Bodies

Combination of Piecewise-Geodesic Paths for Interactive Segmentation

Tractable Algorithms for Robust Model Estimation

Erratum to: Continuous Action Recognition Based on Sequence Alignment

Monocular Extraction of 2.1D Sketch Using Constrained Convex Optimization

Premium Partner