Intelligent multi-camera video surveillance: A review

Abstract

Intelligent multi-camera video surveillance is a multidisciplinary field related to computer vision, pattern recognition, signal processing, communication, embedded computing and image sensors. This paper reviews the recent development of relevant technologies from the perspectives of computer vision and pattern recognition. The covered topics include multi-camera calibration, computing the topology of camera networks, multi-camera tracking, object re-identification, multi-camera activity analysis and cooperative video surveillance both with active and static cameras. Detailed descriptions of their technical challenges and comparison of different solutions are provided. It emphasizes the connection and integration of different modules in various environments and application scenarios. According to the most recent works, some problems can be jointly solved in order to improve the efficiency and accuracy. With the fast development of surveillance systems, the scales and complexities of camera networks are increasing and the monitored environments are becoming more and more complicated and crowded. This paper discusses how to face these emerging challenges.

Introduction

Intelligent video surveillance has been one of the most active research areas in computer vision. The goal is to efficiently extract useful information from a huge amount of videos collected by surveillance cameras by automatically detecting, tracking and recognizing objects of interest, and understanding and analyzing their activities. Video surveillance has a wide variety of applications both in public and private environments, such as homeland security, crime prevention, traffic control, accident prediction and detection, and monitoring patients, elderly and children at home. These applications require monitoring indoor and outdoor scenes of airports, train stations, highways, parking lots, stores, shopping malls and offices. There is an increasing interest in video surveillance due to the growing availability of cheap sensors and processors, and also a growing need for safety and security from the public. Nowadays there are tens of thousands of cameras in a city collecting a huge amount of data on a daily basis. Researchers are urged to develop intelligent systems to efficiently extract information from large scale data.

The view of a single camera is finite and limited by scene structures. In order to monitor a wide area, such as tracking a vehicle traveling through the road network of a city or analyzing the global activities happening in a large train station, video streams from multiple cameras have to be used. Many intelligent multi-camera video surveillance systems have been developed (Collins et al., 2001, Aghajan and Cavallaro, 2009, Valera and Velastin, 2004). It is a multidisciplinary field related to computer vision, pattern recognition, signal processing, communication, embedded computing and image sensors. This paper reviews the recent development of relevant technologies from the perspective of computer vision. Some key computer vision technologies used in multi-camera surveillance systems are shown in Fig. 1.

• Multi-camera calibration maps different camera views to a single coordinate system. In many surveillance systems, it is a key pre-step for other multi-camera based analysis.

• The topology of a camera network identifies whether camera views are overlapped or spatially adjacent and describes the transition time of objects between camera views.

• Object re-identification is to match two image regions observed in different camera views and recognize whether they belong to the same object or not, purely based the appearance information without spatio-temporal reasoning.

• Multi-camera tracking is to track objects across camera views.

• Multi-camera activity analysis is to automatically recognize activities of different categories and detect abnormal activities in a large area by fusing information from multiple camera views.

Different modules support one another and the arrows in Fig. 1 show the information flow between them.

While some existing reviews Valera and Velastin, 2004, Aghajan and Cavallaro, 2009 tried to cover all the aspects of architectures, technologies and applications, this paper emphasizes the connection and integration of these key computer vision and pattern recognition technologies in various environments and application scenarios and reviews their most recent development. Many existing surveillance systems solve these problems sequentially according to a pipeline. However, recent research works show that some of these problems can be jointly solved or even be skipped in order to overcome the challenges posed by certain application scenarios. For example, while it is easy to compute the topology of a camera network after cameras are well calibrated, some approaches are proposed to compute the topology without camera calibration, because existing calibration methods have various limitations and may not be efficient or accurate enough in certain scenarios. On the other hand, the topology information can help with calibration. If it is known that two camera views have overlap, the homography between them can be computed in an automatic manner. Therefore, these two problems are jointly solved in some approaches. Multi-camera tracking requires matching tracks obtained from different camera views according to their visual and spatio-temporal similarities. Matching the appearance of image regions is studied in object re-identification. The spatio-temporal reasoning requires camera calibration and the knowledge of topology. Some studies show that the complete trajectories across camera views can be used to calibrate cameras and to compute the topology. Therefore, multi-camera tracking can be jointly solved with camera calibration and inference of the topology. Multi-camera tracking is often a pre-step for multi-camera activity analysis, which uses the complete tracks of objects over the camera network as features. It is also possible to directly model activities in multiple camera views without tracking object across camera views. Once the models of activities are learned, they can provide useful information for multi-camera tracking, since if two tracks are classified as the same activity category, it is more likely for them to be the same object. A good understanding of the relationship of these modules helps to design optimal multi-camera video surveillance meeting the requirements of different applications.

Intelligent multi-camera video surveillance faces many challenges with the fast growth of camera networks. A few of them are briefly mentioned below. More detailed discussions are found in later sessions.

• A multi-camera video surveillance system may be applied to many different scenes and have various configurations. As the scales of camera networks increase, it is expected that the multi-camera surveillance systems can self-adapt to a variety of scenes with less human intervention. For example, it is very time consuming to manually calibrate all the cameras on a large network and the human effort has to be repeated when the configuration of the camera network changes. Therefore, automatic calibration is preferred. Object re-identification and multi-camera activity analysis prefer unsupervised approaches in order to avoid manually labeling new training samples scenes and camera views change.

• The topology of a large camera network could be complex and the fields of views of cameras are limited by scene structures. Some camera views are disjointed and may cover multiple ground planes. These bring great challenges for camera calibration, inference of topology and multi-camera tracking.

• There are often large changes of viewpoints, illumination conditions and camera settings between different camera views. It is difficult to match the appearance of objects across camera views.

• Many scenes of high security interest, such as airports, train stations, shopping malls and street intersections are very crowded. It is difficult to track objects over long distances without failures because of frequent occlusions among objects in such scenes. Although some existing surveillance systems work well in sparse scenes, there are many challenges unsolved in their applications to crowded environments.

• In order to monitor a wide area with a small number of cameras and to acquire high resolution images from optimal viewpoints, some surveillance systems employ both static cameras and active cameras, whose panning, tilting and zooming (PTZ) parameters are automatically and dynamically controlled by the system. Calibration, motion detection, object tracking and activity analysis with hybrid cameras face many new challenges compared with only using static cameras.

This paper reviews the five key computer vision and pattern recognition technologies (i.e., multi-camera calibration, computing the topology of camera views, multi-camera tracking, object re-identification and multi-camera activity analysis) from Sections 2–6. Cooperative video surveillance both with static and active cameras is discussed in Section 7. Detailed descriptions of their technical challenges and comparison of different solutions are provided under each topic. Finally some unsolved challenges and future research directions are discussed in Section 8.