Abnormal gesture recognition based on multi-model fusion strategy

In this paper, we present a novel refined fused model combining masked Res-C3D network and skeleton LSTM for abnormal gesture recognition in RGB-D videos. The key to our design is to learn discriminative representations of gesture sequences in particular abnormal gesture samples by fusing multiple features from different models. First, deep spatiotemporal features are well extracted by 3D convolutional neural networks with residual architecture (Res-C3D). As gestures are mainly derived from the arm or hand movements, a masked Res-C3D network is built to decrease the effect of background and other variations via exploiting the skeleton of the body to reserve arm regions with discarding other regions. And then, relative positions and angles of different key points are extracted and used to build a time-series model by long short-term memory network (LSTM). Based the above representations, a fusion scheme for blending classification results and remedy model disadvantage by abnormal gesture via a weight fusion layer is developed, in which the weights of each voting sub-classifier being advantage to a certain class in our ensemble model are adaptively obtained by training in place of fixed weights. Our experimental results show that the proposed method can distinguish the abnormal gesture samples effectively and achieve the state-of-the-art performance in the IsoGD dataset.