Human Motion Sensing and Recognition

Human motion analysis mainly involves body sensing, which employs sensors to extract static or dynamic information of the human body while performing behaviours including gestures, actions and activities, and behaviour recognition, which uses software tools to automatically interpret and understand these behaviours to help the computer make further actions, for example, alarming or decision making.

This Chapter presents a fuzzy qualitative representation of conventional trigonometry with the goal of bridging the gap between symbolic cognitive functions and numerical sensing and control tasks in the domain of physical systems, especially in intelligent robotics. Fuzzy qualitative coordinates are defined by replacing a unit circle with a fuzzy qualitative circle; a Cartesian translation and orientation are defined by their normalised fuzzy partitions. Conventional trigonometric functions, rules and the extensions to triangles in Euclidean space are converted into their counterparts in fuzzy qualitative coordinates using fuzzy logic and qualitative reasoning techniques. This approach provides a promising representation transformation interface to analyse general trigonometry-related physical systems from an artificial intelligence perspective. Fuzzy qualitative trigonometry has been implemented as a MATLAB toolbox named XTRIG in terms of 4-tuple fuzzy numbers. Examples are given throughout the chapter to demonstrate the characteristics of fuzzy qualitative trigonometry.

This chapter proposes a fuzzy qualitative (FQ) version of robot kinematics with the goal of bridging the gap between symbolic or qualitative functions and numerical sensing and control tasks for intelligent robotics. The trigonometry role in robot kinematics is replaced using FQ trigonometry and the proposed derivative extension, which leads to a FQ version of robot kinematics. FQ transformation, position, and velocity of a serial kinematics robot are derived and discussed. Then, an aggregation operator to extract robot behaviours is presented with the highlight of the impact of the proposed methods to intelligent robotics. The proposed methods have been integrated into XTRIG MATLAB toolbox and a case study on a PUMA robot has been implemented to demonstrate their effectiveness.

This chapter proposes a fuzzy qualitative (FQ) approach to vision-based human motion analysis with an emphasis on human motion recognition. It achieves feasible computational cost for human motion recognition by combining FQ robot kinematics with human motion tracking and recognition algorithms. First, a data quantisation process is proposed to relax the computational complexity suffered from visual tracking algorithms. Secondly, a novel human motion representation, Qualitative Normalised Template (QNT), is developed in terms of the FQ robot kinematics framework to effectively represent human motion. The human skeleton is modelled as a complex kinematic chain, its motion is represented by a series of such models in terms of time. Finally, experiment results are provided to demonstrate the effectiveness of the proposed method. An empirical comparison with conventional Hidden Markov Model (HMM) and Fuzzy Hidden Markov Model (FHMM) shows that the proposed approach consistently outperforms both hidden Markov models in human motion recognition.

In this chapter, in order to improve both the performance and the efficiency of the conventional Gaussian Mixture Models (GMMs), generalised GMMs are firstly introduced by integrating the conventional GMMs and the active curve axis GMMs for fitting non-linear datasets, and then two types of Fuzzy Gaussian Mixture Models (FGMMs) with a faster convergence process are proposed based on the generalised GMMs, inspired from the mechanism of Fuzzy C-means (FCMs) which introduces the degree of fuzziness on the dissimilarity function based on distances. One is named as probability based FGMMs defining the dissimilarity as the multiplicative inverse of probability density function, and the other is distance based FGMMs which define the dissimilarity function focusing the degree of fuzziness only on the distances between points and component centres. Different from FCMs, both of the proposed dissimilarity functions are based on the exponential function of the distance. The FGMMs are compared with the conventional GMMs and the generalised GMMs in terms of the fitting degree and convergence speed. The experimental results show that the proposed FGMMs not only possess the non-linearity to fit datasets with curve manifolds but also have a much faster convergence process saving more than half computational cost than GMMs’.

Empirical copula is a non-parametric algorithm to estimate the dependence structure of high-dimensional arbitrarily distributed data. The computation of empirical copula is, however, very costly so that it cannot be implemented into applications at a realtime context. In this chapter, fuzzy empirical copula is proposed to reduce the computation time of dependence structure estimation. First, a brief introduction of empirical copula is provided. Next, a new version of Fuzzy Clustering by Local Approximation of Memberships (FLAME) is proposed to be integrated into empirical copula. The FLAME\(^+\) algorithm is implemented to identify the highest density objects which are used to represent the original dataset and then empirical copula is used to estimate its independence structure. Finally, two case studies have been carried out to demonstrate the effectiveness and efficiency of the fuzzy empirical copula.

Unconstrained human hand motions consisting grasp motions and in-hand manipulations lead to a fundamental challenge that many algorithms have to face in both theoretical and practical development, mainly due to the complexity and dexterity of the human hand. There is no effective solution reported to recognise in-hand manipulations though recognition algorithms have been proposed to recognise grasp motions in constrained scenarios. Following Chaps. 5 and 6, this chapter proposes a novel unified fuzzy framework of a set of recognition algorithms: time clustering, FGMMs and FEC, from numerical clustering to data dependency structure in the context of optimally real-time human hand motion recognition. Time clustering is a fuzzy time-modelling approach based on fuzzy clustering and Takagi-Sugeno modelling with numerical value as output; FGMMs effectively extract abstract Gaussian pattern to represent components of hand gestures with a fast convergence; FEC utilises the dependence structure among the finger joint angles to recognise the motion type. The proposed algorithms have been evaluated on a wide range of scenarios of human hand recognition: (a) datasets including 13 grasps and 10 in-hand manipulations; (b) single subject and multiple subjects. (c) varying training samples. The experimental results have demonstrated that the proposed framework outperforms hidden Markov model and Gaussian mixture model in terms of both effectiveness and efficiency criteria.

In order to study and analyse human hand motions which contain multimodal information, a generalised framework integrating multiple sensors is proposed and consists of modules of sensor integration, signal preprocessing, correlation study of sensory information and motion identification. Three types of sensors are integrated to simultaneously capture the finger angle trajectories, the hand contact forces and the forearm electromyography (EMG) signals. To facilitate the rapid acquisition of human hand tasks, methods to automatically synchronise and segment manipulation primitives are developed in the signal preprocessing module. Correlations of the sensory information are studied by using FEC introduced in Chap. 6 and demonstrate there exist significant relationships between muscle signals and finger trajectories and between muscle signals and contact forces. In addition, recognising different hand grasps and manipulations based on the EMG signals is investigated by using FGMMs presented in Chap. 5 and results of comparative experiments show FGMMs outperform GMMs and Support Vector Machine (SVM) with a higher recognition rate. The proposed framework integrating the state-of-the-art sensor technology with the developed algorithms provides researchers a versatile and adaptable platform for human hand motion analysis and has potential applications especially in robotic hand or prosthetic hand control and Human Computer Interaction (HCI).

This chapter proposes a novel approach to extract human hand gesture features in real-time from RGB-D images based on the earth mover’s distance and Lasso algorithms. Firstly, hand gestures with hand edge contour are segmented using a contour length information based de-noise method. A modified finger earth mover’s distance algorithm is then applied to locate the palm image and extract fingertip features. Lastly and more importantly, a Lasso algorithm is proposed to effectively and efficiently extract the fingertip feature from a hand contour curve. Experimental results are discussed to demonstrate the effectiveness of the proposed approach.

Enormous uncertainties in unconstrained human motions lead to a fundamental challenge that many recognising algorithms have to face in practice: motion recognition has to be efficiently correct, but verifying whether or not the algorithm is robustly following the true target motion tends to be demanding, especially when human kinematic motions heavily overlap and occlusion occurs. Due to the lack of a good solution to this problem, many existing methods tend to be either effective but computationally intensive or efficient but vulnerable to false alarms. This chapter presents a novel inference engine for recognising occluded 3D human motion assisted by the recognition context. First, uncertainties are wrapped into a fuzzy membership function via a novel method called fuzzy quantile generation which employs metrics derived from the probabilistic quantile function. Then, time-dependent and context-aware rules are produced via a genetic programming to smooth the qualitative outputs represented by fuzzy membership functions. Finally, occlusion in motion recognition is taken care of by introducing new procedures for feature selection and feature reconstruction. Experimental results demonstrate the effectiveness of the proposed inference engine for 3D occluded human motion recognition.

Most of the existed action recognition methods mainly utilise spatio-temporal descriptors of single interest point ignoring their potential integral information, such as spatial distribution information. By combining local spatio-temporal feature and global positional distribution information (PDI) of interest points, a novel motion descriptor is proposed in this chapter. The proposed method detects interest points by using an improved interest points detection method. Then 3-dimensional scale-invariant feature transform (3D SIFT) descriptors are extracted for every interest point. In order to obtain compact description and efficient computation, Principal Component Analysis (PCA) method is utilised twice on the 3D SIFT descriptors of single-frame and multi-frame. Simultaneously, the PDI of the interest points are computed and combined with the above features. The combined features are quantified and selected and finally tested by using Support Vector Machine (SVM) and AdaBoost-SVM recognition algorithm on the public KTH dataset. The testing results showed that the recognition rate has been significantly improved. Meantime, the test results verified the proposed features can more accurately describe human motion with high adaptability to scenarios.

Visual-based action recognition has already been widely used in human-machine interfaces. However it is a challenging research to recognise the human actions from different viewpoints. In order to solve this issue, a novel multi-view space Hidden Markov Models (HMMs) algorithm for view-invariant action recognition is proposed. Firstly a view-insensitive feature representation by combining the bag-of-words of interest point with the amplitude histogram of optical flow is utilised for describing the human action sequences. The combined features could not only solve the problem that there was no possibility in establishing an association between traditional bag-of-words of interest point method and HMMs, but also greatly reduce the redundancy in the video. Secondly the view space is partitioned into multiple sub-view space according to the camera rotation viewpoint. Human action models are trained by HMMs algorithm in each sub-view space. By computing the probabilities of the test sequence (i.e. observation sequence) for the given multi-view space HMMs, the similarity between the sub-view space and the test sequence viewpoint are analysed during the recognition process. Finally the action with unknown viewpoint is recognised via the probability weighted combination. The experimental results on multi-view action dataset IXMAS demonstrate that the proposed approach is highly efficient and effective in view-invariant action recognition.