Elsevier

Computer Science Review

Volume 28, May 2018, Pages 157-177
Computer Science Review

Review article
New trends on moving object detection in video images captured by a moving camera: A survey

https://doi.org/10.1016/j.cosrev.2018.03.001Get rights and content

Abstract

This paper presents a survey on the latest methods of moving object detection in video sequences captured by a moving camera. Although many researches and excellent works have reviewed the methods of object detection and background subtraction for a fixed camera, there is no survey which presents a complete review of the existing different methods in the case of moving camera. Most methods in this field can be classified into four categories; modeling based background subtraction, trajectory classification, low rank and sparse matrix decomposition, and object tracking. We discuss in details each category and present the main methods which proposed improvements in the general concept of the techniques. We also present challenges and main concerns in this field as well as performance metrics and some benchmark databases available to evaluate the performance of different moving object detection algorithms.

Introduction

In the field of computer vision, detection of moving objects from a video sequence, which is based on representing moving objects by a binary mask in each frame, is an important issue and interested in many vision based applications such as action recognition [1], traffic controlling [2], industrial inspection [3], human behavior identification [4], and intelligent video surveillance [5]. In many of these applications, a moving camera is inherently utilized. For example, in most intelligent video surveillance systems, we use camera movement techniques such as pan–tilt–zoom (PTZ) to better focus and track the targets [6]. Recently, progress in drone technology with using relatively cheap drones with advanced imaging capabilities promises vast future commercial applications [7]. Here, the camera may operate with various degrees of movement and autonomy. Besides, with advances in camera phone technology for mobile phones, more and more people are interested to capture video sequences with their mobile phone capable to detect and track the moving objects [8]. Here, the camera may have free movements. Even in capturing outdoor scenes by a fixed camera, the camera cannot completely be considered as stationary due to the non-controlled environment [9]. Here, we are facing jitter problems in camera or camera shake problems. Thus, the increasing use of moving cameras along with growing interests in detecting moving objects make it essential to develop robust methods of moving object detection for moving cameras.

In the simple case of a fixed camera, the only changes between consecutive frames are caused by moving objects. However, all these changes are not due to the objects of interest (targets) for a user or a desired application. Concerning an indoor scene, even under a controlled environment, shadow regions and illumination source changes may occur and are undesired for moving object detection [10]. For an outdoor scene, because typically the environment is not controllable, many undesired changes such as branch movement, cloud movement and illumination variations can cause serious problems for moving object detection [9]. Many previous works [[11], [12]] have addressed the moving object detection in video sequences captured by a fixed camera in the presence or absence of undesired changes in the scene. To do that, the principal idea is to create a stable background modeling and then to apply a background subtraction technique, namely to subtract current frame from background to detect moving objects.

For the case of moving camera, it is important that the method of moving object detection considers not only all problems arise in a fixed camera but also certain difficulties due to compensation of camera motion. This is why a simple background subtraction with a naïve motion compensation model cannot efficiently be applied for a moving camera. Indeed, inaccuracy in motion compensation, which is highly possible for a free movement of the camera, causes the background modeling to fail creating a good model for background and foreground pixels [10].

For detecting moving objects in the case of a moving camera, one strategy is to differentiate the movements caused by moving objects from those caused by the camera. There are two main categories of solution. One is based on background modeling [[13], [14]] which tries to create an appropriate background for each frame of the sequence by using a motion compensation method. Another one is trajectory classification [[15], [16]] in which long term trajectories are computed for feature points using an appropriate tracker and next a clustering approach is used to differentiate the trajectories belonging to the same objects from those of background.

Another strategy is to extend background subtraction methods based on low rank and sparse matrix decomposition developed for the case of static cameras [[17], [18], [19], [20], [21], [22]] for the case of a moving camera [[23], [24]]. The principal idea is that if certain coherency exists between a set of image frames, low rank representation of the matrix formed by these frames contains this coherency and sparse representation of this matrix contains outliers. Since the moving objects give intensity changes, which are different from the background and cannot be fitted into the low-rank model of the background, they can be considered as outliers for the low rank representation. Thus, sparse representation of the frames contains the moving objects in these frames. However, it is true based on the assumption that the background is the same for all frames, i.e. the camera is static. Although, this technique cannot directly be applied for the case of a moving camera, where the background changes between frames, a transformation can be integrated into the model in order to compensate for the background motion caused by the moving camera [[23], [24]]. This transformation can be an 2D parameter transform in which the parameters can be adjusted (e.g. using the affine transform for PTZ motions [25] or the perspective transform for free motions [26]).

Object tracking strategy can also be considered as moving object detection although its objective is different. Indeed, in object tracking, typically we mark an object as our desired object (target) and then try to localize it in the next frames of the video sequence. To do that, target information such as histogram, color, texture, statistics etc., is extracted from current frame and then the best candidate in the next frame is obtained using a model of similarity or an appropriate classifier. Finally, the characteristics of the target will be updated to be used for next frames [12].

We will present the methods proposed in each strategy in details and compare their advantages and disadvantages. We will describe different aspects of moving object detection with focus on the case of moving camera. We will also introduce some benchmark video datasets used in the related works and the metrics used to evaluate the performance of the implemented algorithms.

As we have exhaustively searched the publications that presented surveys of different moving object detection methods, nearly all of them have focused on the case of a fixed camera [[27], [28], [29], [30], [31]], where background image pixels maintain their position in the corresponding frames throughout a video sequence. Although the methods introduced in this domain can be applied successfully to the special case of automated surveillance, where the cameras mounted on a fixed platform, they cannot directly be extended for the cases of moving camera such as video taken by mobile phones, hand held cameras or cameras are mounted on a moving platform where the background image pixels do not maintain their position throughout the video sequence. Most of the review publications in this regard have focused on presenting primitives for detecting moving objects in video and methodologies specifically for tracking objects [[32], [12]]. For instance, Shantaiya et al. in [32] reviewed the works done under the general term of object detection in video and categorized them as featured based, template based, classifier based and motion based with no constraints on camera motion. In the literature survey [12], it has been introduced various segmentation methods relevant to tracking objects in video and categorized object tracking into point tracking, kernel tracking and silhouette tracking and compared the methods in each category. In another work, Deori and Thounaojam [33] divided object tracking methods into contour based, feature based and region based. In [34], Parekh et al. also focused on tracking objects by dividing it into three steps of object detection, object classification and object tracking and compared the methods proposed in each steps. In all these surveys, no constraints on camera motion were imposed. A survey on moving object detection for mobile camera has been introduced in [35], however, Sanap et al. briefly discussed some categories and did not clearly compare the methods. In [36], although Shahre and Shende elaborated the moving object detection in presence of static and dynamic background, their main concern has been the case of fixed camera and the case of moving camera was discussed shortly by considering the complexities arise in this case without categorizing the methods.

The goal of this survey is to group moving object detection methods for moving camera into broad categories and give general concepts for representative methods in each category. The main contributions of this survey are as follows:

  • We categorize the methods in different ways based on the general concept of methods belonging to each category and describe proposed improvements with respect to this general concept.

  • We extensively study the whole challenges facing accurate detection of moving objects in the video sequences captured by a moving camera.

  • We introduce performance metrics and a list of existing benchmark datasets for evaluating the performance of different proposed methods in the case of moving cameras.

The paper is organized as follows. In the next section, we discuss different aspects regarding the detection of moving objects in a video sequence. In Section 3, we group the methods of moving object detection for moving camera into different categories, describe their general concept and introduce main methods which brought the improvements in each category. We then introduce in Section 4 some benchmark datasets used to compare the different methods of moving object detection. Section 5 presents some performance metrics used to evaluate the algorithms for moving detection for moving camera. Finally, we terminate the paper with a discussion and conclusion.

Section snippets

Challenges to moving object detection in video

The objective of moving object detection is to take a video sequence from a fixed/moving camera and outputs a binary mask representing moving objects for each frame of the sequence. However, this is not an easy task to do due to many challenges and difficulties involved when a camera is used to capture a video sequence of moving objects. Here, we present these challenging issues in details.

Moving object detection methods

Contrary to vast algorithms proposed for the moving object detection in the case of fixed camera over last decade, few algorithms directly focused their work on the case of moving camera and most of them have been recently published since this subject became lately very attractive and showed many future promising applications. We can broadly classify all efforts in this field in four categories which will be detailed as follows.

Video databases for moving cameras

Usually, the performance of a new moving object detection algorithm should be evaluated and compared with that of the state of the art methods using same databases. Although many benchmark datasets for evaluation of moving object detection algorithms for fixed cameras are available and can be easily found in the Internet [[136], [188], [189], [190]], a few of those for moving camera are available and hereby some authors have preferred to report their comparative results based on using their own

Performance metrics

In order to evaluate and compare various moving object detection algorithms for moving camera, quantitative metrics which are faire and consistent should be used. Meanwhile, to apply these metrics, the established ground truth of the sequences on which the experiments have been done should be available. Contrary to fixed camera case, a few ground truths are available for databases for moving camera and due to this unavailability, some authors provided the qualitative comparisons based on visual

Concluding remarks

In this paper, various aspects of moving object detection with focusing on moving camera have been studied. We have divided the moving object detection methods into four broad categories based on the technique they have used, namely, methods modeling the background, methods classifying trajectories, methods based on matrix decomposition and methods tracking moving objects. We have provided extensive reviews on difficulties and challenging tasks, real-time concern of moving object detection

Acknowledgments

The authors would like to acknowledge the Region Poitou-Charente of France and Shiraz University (grant number: 94/4/10/943/10/1531) which provided funds for this work and the stay of Prof. Mehran Yazdi at the Lab. MIA of the Univ. La Rochelle.

References (217)

  • SaninA. et al.

    Shadow detection: A survey and comparative evaluation of recent methods

    Pattern Recognit.

    (2012)
  • Al-NajdawiN. et al.

    A survey of cast shadow detection algorithms

    Pattern Recognit. Lett.

    (2012)
  • TreptowA. et al.

    Real-time object tracking for soccer-robots without color information

    Robot. Auton. Syst.

    (2004)
  • RoweS. et al.

    Statistical mosaics for tracking

    Image Vis. Comput.

    (1996)
  • RenY. et al.

    Statistical background modelling for non-stationary camera

    Pattern Recognit. Lett.

    (2003)
  • AvolaD. et al.

    A keypoint-based method for background modeling and foreground detection using a PTZ camera

    Pattern Recognit. Lett.

    (2017)
  • ZhouD. et al.

    Moving object detection and segmentation in urban environments from a moving platform

    Image Vis. Comput.

    (2017)
  • DaubneyB. et al.

    Estimating pose of articulated objects using low-level motion

    Comput. Vis. Image Underst.

    (2012)
  • S. Wu, O. Oreifej, M. Shah, Action recognition in videos acquired by a moving camera using motion decomposition of...
  • CucchiaraR. et al.

    Statistic and knowledge-based moving object detection in traffic scenes

    IEEE Intell. Transp. Syst.

    (2000)
  • HuW. et al.

    A survey on visual surveillance of object motion and behaviors

    IEEE Trans. Syst Man Cybern. C

    (2004)
  • KimJ.S. et al.

    Fast and robust algorithm of tracking multiple moving objects for intelligent video surveillance systems

    IEEE Trans. Consum. Electron.

    (2011)
  • YangJ. et al.

    Design of video surveillance and tracking system based on attitude and heading reference system and PTZ camera

    AIP Conf. Proc.

    (2017)
  • ChenP. et al.

    Real-time object tracking on a drone with multi-inertial sensing data

    IEEE Trans. Intell. Transp. Syst.

    (2018)
  • DamesP. et al.

    Detecting, localizing, and tracking an unknown number of moving targets using a team of mobile robots

    Int. J. Robot. Res.

    (2017)
  • B. Risse, M. Mangan, L. Del Pero, B. Webb, Visual tracking of small animals in cluttered natural environments using a...
  • L. Leal-Taixé, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Roth, Tracking the trackers: an analysis of the state of...
  • YilmazA. et al.

    Object tracking: A survey

    ACM Comput. Surv.

    (2006)
  • WrenC.R. et al.

    Pfinder:Real-time tracking of the human body

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • T. Brox, J. Malik, Object segmentation by long term analysis of point trajectories, in: European Conference on Computer...
  • YinX. et al.

    Background subtraction for moving camera based on trajectory-controlled segmentation and label inference

    KSII Trans. Internet Inf. Syst.

    (2015)
  • ZhouX. et al.

    Moving object detection by detecting contiguous outliers in the low-rank representation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2013)
  • BouwmansT. et al.

    Handbook on Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing

    (May 2016)
  • S. Javed, A. Sobral, T. Bouwmans, S. Jung, OR-PCA with dynamic feature selection for robust background subtraction, in:...
  • A. Sobral, T. Bouwmans, E. Zahzah, Double-constrained RPCA Based on Saliency Maps for Foreground Detection in Automated...
  • S.E. Ebadi, V.G. Ones, E. Izquierdo, Efficient background subtraction with low-rank and sparse matrix decomposition,...
  • P. Rodriguez, B. Wohlberg, Translational and rotational jitter invariant incremental principal component pursuit for...
  • S. Wu, T. Zhao, C. Broaddus, C. Yang, M. Aggarwal, Robust pan, tilt and zoom estimation for PTZ camera by using meta...
  • WuY. et al.

    Moving object detection with a freely moving camera via background motion subtraction

    IEEE Trans. Circuits Syst. Video Technol.

    (2017)
  • KinjalK.A. et al.

    A survey on moving object detection and tracking in video surveillance system

    Int. J. Soft Comput. Eng.

    (2012)
  • RamyaA. et al.

    A survey and comparative analysis of moving object detection and tracking

    Int. J. Eng. Res. Technol.

    (2013)
  • BouwmansT.

    Recent advanced statistical background modelling for foreground detection: A systematic survey

  • SharmaV. et al.

    A survey on moving object detection methods in video surveillance

    Int. Bull. Math. Res.

    (2015)
  • ShantaiyaS. et al.

    A survey on approaches of object detection

    Int. J. Comput. Appl.

    (2013)
  • DeoriB. et al.

    A survey on moving object tracking in videos

    Int. J. Inf. Theory

    (2014)
  • ParekhH.S. et al.

    A survey on object detection and tracking methods

    Int. J. Innov. Res. Comput. Commun. Eng.

    (2014)
  • SanapV.A. et al.

    Survey on moving object detection

    Int. J. Modern Trends Eng. Res.

    (2015)
  • ShahreD. et al.

    A survey on moving object detection in static and dynamic background for automated video analysis

    Int. J. Sci. Res. Dev.

    (2013)
  • AdelsonE.H.

    On seeing stuff: the perception of materials by humans and machines

    Human Vis. Electron. Imaging

    (2001)
  • BlackM.J. et al.

    Eigentracking: Robust matching and tracking of articulated objects using a view-based representation

    Int. J. Comput. Vis.

    (1998)
  • Cited by (0)

    View full text