Histogram of oriented rectangles: A new pose descriptor for human action recognition

Abstract

Most of the approaches to human action recognition tend to form complex models which require lots of parameter estimation and computation time. In this study, we show that, human actions can be simply represented by pose without dealing with the complex representation of dynamics. Based on this idea, we propose a novel pose descriptor which we name as Histogram-of-Oriented-Rectangles (HOR) for representing and recognizing human actions in videos. We represent each human pose in an action sequence by oriented rectangular patches extracted over the human silhouette. We then form spatial oriented histograms to represent the distribution of these rectangular patches. We make use of several matching strategies to carry the information from the spatial domain described by the HOR descriptor to temporal domain. These are (i) nearest neighbor classification, which recognizes the actions by matching the descriptors of each frame, (ii) global histogramming, which extends the idea of Motion Energy Image proposed by Bobick and Davis to rectangular patches, (iii) a classifier-based approach using Support Vector Machines, and (iv) adaptation of Dynamic Time Warping on the temporal representation of the HOR descriptor. For the cases when pose descriptor is not sufficiently strong alone, such as to differentiate actions “jogging” and “running”, we also incorporate a simple velocity descriptor as a prior to the pose based classification step. We test our system with different configurations and experiment on two commonly used action datasets: the Weizmann dataset and the KTH dataset. Results show that our method is superior to other methods on Weizmann dataset with a perfect accuracy rate of 100%, and is comparable to the other methods on KTH dataset with a very high success rate close to 90%. These results prove that with a simple and compact representation, we can achieve robust recognition of human actions, compared to complex representations.

Introduction

Human action recognition is one of the appealing, yet challenging problems of computer vision. Reliable and effective solutions to this problem can serve many areas, ranging from human–computer interaction to security surveillance. However, current solutions are very limited, and understanding what people are doing remains unresolved.

Human action recognition has been a widely studied topic (for extensive reviews see [[1], [2]]), but the solutions to the problem that have been submitted to date are very premature and still specific to the dataset at hand.

There are three key elements that define an action:

• pose of the body (and parts),

• speed of body motion (and parts),

• relative ordering of the poses.

We can formulate action recognition as a mixture of these three elements. The relative importance of these elements is based on the nature of the actions that we aim to recognize. For example, if we want to differentiate an instance of a “bend” action from a “walk” action, the pose of the human figure gives sufficient information. However, if we want to discriminate between “jog” and “run” actions, the pose alone may not be enough, due to the similarity in the nature of these actions in the pose domain, and in such cases, the speed information needs to be incorporated. Similarly, for recognizing “stand up” and “sit down” actions, the relative ordering of the poses will be important, since these two actions include same poses in reverse temporal orders.

Various attempts in action recognition literature try to model some or all of these aspects. For instance, methods based on spatio-temporal templates mostly pay attention to the pose of the human body, whereas methods based on dynamical models focus on modeling the ordering of these poses in greater detail.

We argue that the human pose encapsulates many useful clues for recognizing the ongoing activity. Actions can mostly be represented by configurations of the body parts, before building complex models for understanding the dynamics.

Using this idea, we base the foundation of our method on defining the pose of the human body to discriminate actions, and by introducing a new pose descriptor, we want to evaluate how far we can go only with a good description of the pose of the body. We also evaluate how our system benefits from adding the remaining action components whenever necessary. Unlike most of the methods that use complex modeling of body configurations, we follow the analogy of Forsyth and Fleck [3], which represents the body as a set of rectangles, and explore the layout of these rectangles.

Our pose descriptor is based on a basic intuition: the human body can be represented by a collection of oriented rectangles in the spatial domain and the orientations of these rectangles form a signature for each action. Rather than detecting and learning the exact configuration of body parts, we are only interested in the distribution of the rectangular regions which may be the candidates for the body parts.

This idea is similar to the bag-of-words approach, where the images are represented by a collection of regions, ignoring their spatial relationships. The bag-of-words approach – which is adapted from text retrieval literature – has shown to be successful for object and scene recognition [[4], [5]] and for annotation and retrieval of large image and video collections [[6], [7]]. In such approaches, the images are represented by the distribution of words from a fixed visual vocabulary (i.e. image patches) which is usually obtained by vector quantization of visual features. In our approach, we use rectangles as our visual words and achieve vector quantization by histogramming over their orientation angles. However, our approach is basically different from bag-of-words. First, we are using the distribution of rectangular regions as opposed to complex visual words. Second, we place a grid over these rectangles to capture their spatial layout.

In this study, our main contribution is to show how a good pose descriptor can boost the performance of action recognition. We introduce a novel pose descriptor which is based on candidate rectangular regions over the human body. We show that using our pose descriptor, we can recognize human actions even in complicated settings.

In the rest of the paper, we first cover the literature on human action recognition within a brief overview. Then, we present the details of our pose descriptor, which represents the human figure as a distribution of oriented rectangular patches. After that, we list the matching methods that can be applied to our pose descriptor for efficient identification of human actions. These are, namely, nearest neighbor classification, global histogramming, SVM classification and Dynamic Time Warping. We test our system with different configurations and compare the results to state-of-art action recognition methods. We also provide the run time evaluations of our system. After reporting comprehensive experiments and their results, we conclude our discussion with future research directions.