Real-time gesture recognition system and application

Abstract

In this paper, we consider a vision-based system that can interpret a user's gestures in real time to manipulate windows and objects within a graphical user interface. A hand segmentation procedure first extracts binary hand blob(s) from each frame of the acquired image sequence. Fourier descriptors are used to represent the shape of the hand blobs, and are input to radial-basis function (RBF) network(s) for pose classification. The pose likelihood vector from the RBF network output is used as input to the gesture recognizer, along with motion information. Gesture recognition performances using hidden Markov models (HMM) and recurrent neural networks (RNN) were investigated. Test results showed that the continuous HMM yielded the best performance with gesture recognition rates of 90.2%. Experiments with combining the continuous HMMs and RNNs revealed that a linear combination of the two classifiers improved the classification results to 91.9%. The gesture recognition system was deployed in a prototype user interface application, and users who tested it found the gestures intuitive and the application easy to use. Real time processing rates of up to 22 frames per second were obtained.

Introduction

As computers become more pervasive in society, facilitating natural human–computer interaction (HCI) will have a positive impact on their use. Hence, there has been growing interest in the development of new approaches and technologies for bridging the human–computer barrier. The ultimate aim is to bring HCI to a regime where interactions with computers will be as natural as interactions between humans, and to this end, incorporating gestures in HCI is an important research area [1].

We are interested in developing a vision-based system which can interpret a user's gestures in real time to manipulate windows and objects within a graphical user interface (GUI). Various works by Kadobayashi et al. [2], Pavlovic et al. [3], Freeman et al. [4] and Kjeldsen et al. [5] indicate that there is keen interest among current researchers to incorporate gestures into traditional HCI interfaces. Our work expands on their ideas and also looks into the possibility of using two-handed gestures while imposing fewer constraints on the users.

Much of the research on real-time gesture recognition has focused on the space-time trajectory of the hand without considering the shape or posture of the hand [6], [7], [8]. These works utilized only relative or oscillatory motion of the hand to recognize the gesture. However, in many situations, the meaning of gestures depends very much on the hand posture, in addition to the hand movement. Hence, our work incorporates hand posture as well as hand motion to recognize gestures. Also, unlike other works where users are required to wear artificial devices like data gloves [9] or green markers [10], it is our aim to allow the users to perform gestures in a natural and unencumbered manner.

In the work by Kjeldson and Kender, gestures were incorporated into a windowing user interface to manipulate windows [5]. The hand was segmented by using a neural network whose inputs were images coded in the hue-saturation-intensity (HSI) color model. Another neural network was trained to classify each pose. Gesture interpretation was performed by a state machine which implemented a gesture grammar. Their work demonstrated the feasibility of using gestures in a modern GUI. Our system differs from Ref. [5] in a few ways. We defined a slightly larger gesture set, and our system is specifically designed to allow the user to employ two-handed gestures, if he wishes. Moreover, our system does not need to be re-trained for every new user; it needs only to be trained once to achieve a relatively high level of user-invariance. Finally, the processing steps are quite different.

In the following, we present an overview of the system in Section 2. In Section 3, we describe the segmentation procedure to locate the hand(s) in the image. We then discuss a wrist-cropping method to isolate the segmented hand from the rest of the arm in Section 4. Next in Section 5, we describe pose classification using RBF networks, with Fourier descriptors of the segmented hand boundary as input features. We develop and compare the performance of the gesture recognizer based on hidden Markov Models (HMM) and recurrent neural networks (RNN) in Section 6. We also consider enhancing the recognition performance by combining the classifiers. In Section 7, we present and discuss the recognition results. A prototype GUI application that was developed to run in real time is described in Section 8, and Section 9 concludes the paper.