Frame-Level Covariance Descriptor for Action Recognition

Activity recognition is a fundamental task in areas such as video-surveillance, gesture recognition, robotics, multimedia applications among much others. Such task remains as an open problem because the variability of many factors such as the appearance of actors, illumination changes in real scenarios and the dynamic developed for each action. Despite favorable results in recent works for several academic datasets, the proposed methodologies require a huge number of training samples and the output descriptor result in a high dimensional array that difficult the implementation in real conditions. This work proposes a spatio-temporal descriptor that model human activities by using a fast regional covariance representation for each frame. At each frame, a set of motion and geometrical map measures are quantified into a pyramidal regional structure to describe the instantaneous action. Such low-level primitive maps are codified into a integral covariance that allows a fast and compact description of local correlation among features. The set of pyramidal-frame-covariances along the video sequence represent a manifold that coexist in a positive Riemannian space. Then, a set of means are approximated in Riemannian space for each regional covariance sequence to represent a very compact action descriptor. The proposed action descriptor is mapped to a Euclidean space to perform an automatic classification using a Support vector Machine. The proposed approach was evaluated in two different public datasets: (1) in UT-Interaction with a k-fold cross-validation scheme was achieved a 70.8% of accuracy with a descriptor size of just 10 features per video sequence and (2) in UCF Sports achieve an accuracy of 71.7% using 13 features.