Video Segmentation and Its Applications

Segmentation plays an important role in digital media processing, pattern recognition, and computer vision. The task of image/video segmentation emerges in many application areas, such as image interpretation, video analysis and understanding, video summarization and indexing, and digital entertainment. Over the last two decades, the problem of segmenting image/video data has become a fundamental one and had significant impact on both new pattern recognition algorithms and applications.This chapter has several objectives: (1) to survey the current status of research activities including graph-based, density estimator-based, and temporal-based segmentation algorithms. (2) To discuss recent developments while providing a comprehensive introduction to the fields of image/video segmentation. (3) To identify challenges ahead, and outline perspectives for the years to come.

In this chapter, object segmentation algorithms dependent on the characteristics of eigen-structure are proposed. The eigen-subspaces are obtained from eigen-decomposition of the covariance matrix, which is computed from the selected color samples. Hence, the color space can be transformed into the signal subspace and its orthogonal noise subspaces. After statistical analysis of eigen-structure of target color samples, the color eigen-structure segmentation algorithms are then designed to extract the desired objects, which are close to the color samples. The principal component transformation (PCT) techniques, which only use the signal subspace can be treated as a subset of color eigenspace algorithms. The eigenspaces discriminated as signal and noise subspaces related to original color samples should be effectively utilized. The adaptive eigen-subspace segmentation (AESS) algorithm, which can save the computation of eigen-decomposition, is applied to adaptively adjust the eigen-subspaces. Finally, the Eigen-based fuzzy C-means (FCM) clustering algorithm has been proposed to effective segment color object. By jointly consideration of signal and noise subspace projections of desired colors, the separate eigen-based FCM (SEFCM) and coupled eigen-based FCM (CEFCM) are used to achieve effective color object segmentation. With these proposed algorithms, the color objects can be successfully extracted by using eigen-subspace projections.

Semantic object segmentation is to label each pixel in an image or a video sequence to one of the object classes with semantic meanings. It has drawn a lot of research interest because of its wide applications to image and video search, editing and compression. It is a very challenging problem because a large number of object classes need to be distinguished and there is a large visual variability within each object class. In order to successfully segment objects, local appearance of objects, local consistency between labels of neighboring pixels, and long-range contextual information in an image need to be integrated under a unified framework. Such integration can be achieved using conditional random fields. Conditional random fields are discriminative models. Although they can learn the models of object classes more accurately and efficiently, they require training examples labeled at pixel-level and the labeling cost is expensive. The models of object classes can be learned with different levels of supervision. In some applications, such as web-based image and video search, a large number of object classes need to be modeled and therefore unsupervised learning or semi-supervised learning is preferred. Therefore some generative models, such as topic models, are used in object segmentation because of their capability to learn the object classes without supervision or with weak supervision of less labeling work. We will overview different technologies used in each step of the semantic object segmentation pipeline and discuss major challenges for each step. We will focus on conditional random fields and topic models, which are two types of frameworks widely used in semantic object segmentation. In video segmentation, we summarize and compare the frameworks of Markov random fields and conditional random fields, which are the representative models of the generative and discriminative approaches respectively.

With the increasing proliferation of digital video contents, learning-based video scene analysis has proven to be an effective methodology for improving the access and retrieval of large video collections. This chapter is devoted to present a survey and tutorial on the research in this topic. We identify two major categories of the state-of-the-art tasks based on their application setup and learning targets: generic methods and genre-specific analysis techniques. For generic video scene analysis problems, we discuss two kinds of learning models that aim at narrowing down the semantic gap and the intention gap, two main research challenges in video content analysis and retrieval. For genre-specific analysis problems, we take sports video analysis and surveillance event detection as illustrating examples.

Image segmentation and video tracking (ISVT) is a necessary and important preliminary step in many high-level vision tasks such as activity recognition, rendering and modeling, and scene analysis. Comparing with the mono-view ISVT, multi-view ISVT is capable of characterizing the visual object and dynamic scene with three-dimensional (3D) interpretation, which prevails over the traditional two-dimensional (2D) representation. In this chapter, we categorize and review the representative and state-of-the-art approaches in multiview image segmentation and video tracking. Additionally, our proposed depth-based segmentation in the initial frame and feature-based tracking algorithms from multi-view video for both separated and overlapping human objects are discussed respectively, following the ensuring experimental results to demonstrate the algorithms’ superior performance.