Cross-media web video event mining based on multiple semantic-paths embedding

Web video event mining based on cross-media fusion has become a research hotspot. However, each video is only described by a dozen noisy words, resulting in extremely unstable textual features. Moreover, different people might describe the same video with completely different words. Thus, the semantic association between textual and visual information would be much sparse, which brings great challenges to web video event mining based on cross-media associations. To address this issue, this paper proposes a novel framework to enrich the associations between near duplicate keyframes (NDK) and terms based on multiple semantic-paths embedding. After data preprocessing, we build a heterogeneous information network to establish associations among NDKs, terms and videos. Then, semantic-path walk strategy is designed to generate meaningful semantic-node sequences for embedding. Next, an embedding fusion method is proposed to predict the distribution characteristics of each term in NDKs. Finally, multiple correspondence analysis is used to mine web video events. Experiments on web videos from YouTube show that our proposed method performs better than several state-of-the-art baseline models, with an average F1 score improvement of 19–50%.