Bridging semantic gap between high-level and low-level features in content-based video retrieval using multi-stage ESN–SVM classifier

Abstract

Content-based video retrieval system aims at assisting a user to retrieve targeted video sequence in a large database. Most of the search engines use textual annotations to retrieve videos. These types of engines offer a low-level abstraction while the user seeks high-level semantics. Bridging this type of semantic gap in video retrieval remains an important challenge. In this paper, colour, texture and shapes are considered to be low-level features and motion is a high-level feature. Colour histograms convert the RGB colour space into YcbCr and extract hue and saturation values from frames. After colour extraction, filter mask is applied and gradient value is computed. Gradient and threshold values are compared to draw the edge map. Edges are smoothed for sharpening to remove the unnecessary connected components. These diverse shapes are then extracted and stored in shape feature vectors. Finally, an SVM classifier is used for classification of low-level features. For high-level features, depth images are extracted for motion feature identification and classification is done via echo state neural networks (ESN). ESN are a supervised learning technique and follow the principle of recurrent neural networks. ESN are well known for time series classification and also proved their effective performance in gesture detection. By combining the existing algorithms, a high-performance multimedia event detection system is constructed. The effectiveness and efficiency of proposed event detection mechanism is validated using MSR 3D action pair dataset. Experimental results show that the detection accuracy of proposed combination is better than those of other algorithms