PTL-LTM model for complex action recognition using local-weighted NMF and deep dual-manifold regularized NMF with sparsity constraint

Complex action recognition possesses significant academic research value, potential commercial value and broad market application prospect. For improving its performance, a local-weighted nonnegative matrix factorization with rank regularization constraint (LWNMF_RC) is firstly presented, which removes complex background and then obtains motion salient regions. Secondly, a dual-manifold regularized nonnegative matrix factorization with sparsity constraint (DMNMF_SC) is proposed, which not only considers the short-term and middle-term temporal dependencies implied in data manifold, but also mines the geometric structure hidden in feature manifold. In addition, the introduction of sparsity constraint makes features possess better discriminativeness. Thirdly, a deep DMNMF_SC method is constructed, which acquires more hierarchical and discriminative features. Finally, a long-term temporal memory model with probability transfer learning (PTL-LTM) is proposed, which accurately memorizes the long-term temporal dependency among multiple simple action segments and, meanwhile, makes full use of the probability features of rich labeled simple actions and then applies the knowledge learned from simple actions for complex action recognition. Consequently, the performance is effectively improved.