Surveillance in Action

In spite of the increasing concerns raised by privacy advocates against the intrusive deployment of large scale surveillance cameras, the research community has progressed with a remarkable pace into the area of smart visual surveillance. The automation for surveillance systems becomes an obligation to avoid human errors and ensure an efficient strategy for tackling crimes and preventing further terrorist attacks. In this research article, we survey the recent studies in computer vision on gait recognition for covert identification and its application for surveillance scenarios and forensic investigation. The integration of biometric technologies into surveillance systems is a major step milestone to improve the automation process in order to recognize criminal offenders and track them across different places. The suitability of gait biometrics for surveillance applications emerges from the fact that the walking pattern can be captured and perceived from a distance even with poor resolution video as opposed to other biometric modalities which their performance deteriorates in surveillance scenarios.

The recent growth in CCTV systems and the challenges of automatically identifying humans under the adverse visual conditions of surveillance have increased the interest in soft biometrics, which are physical attributes that can be used to describe people semantically. Soft biometrics enable human identification based on verbal descriptions, and they can be captured in conditions where it is impossible to acquire traditional biometrics such as iris and fingerprint. The research on facial soft biometrics has tended to focus on identification using categorical attributes, whereas comparative attributes have shown a better accuracy. Nevertheless, the research in comparative facial soft biometrics has been limited to small constrained databases, while identification in surveillance systems involves unconstrained large databases. In this chapter, we explore human identification through comparative facial soft biometrics in large unconstrained databases using the Labelled Faces in the Wild (LFW) database. We propose a novel set of attributes and investigate their significance. Also, we analyse the reliability of comparative facial soft biometrics for realistic databases and explore identification and verification using comparative facial soft biometrics. The results of the performance analysis show that by comparing an unknown subject to a line up of ten subjects only; a correct match will be found in the top 2.08% retrieved subjects from a database of 4038 subjects.

In this paper, we study the problem of human identity recognition using off-angle human faces. Our proposed system is composed of (i) a physiology-based human clustering module and (ii) an identification module based on facial features (nose, mouth, etc.) fetched from face videos. In our proposed methodology we, first, passively extract an important vital sign (breath). Next we cluster human subjects into nostril motion versus nostril non-motion groups, and, then, localize a set of facial features, before we apply feature extraction and matching. Our proposed human identity recognition system is very efficient. It is working well when dealing with breath signals and a combination of different facial components acquired under challenging luminous conditions. This is achieved by using our proposed Motion Classification approach and Feature Clustering technique based on the breathing waveforms we produce. The contributions of this work are three-fold. First, we generated a set of different datasets where we tested our proposed approach. Specifically, we considered six different types of facial components and their combination, to generate face-based video datasets, which present two diverse data collection conditions, i.e., videos acquired in head full frontal pose (baseline) and head looking up pose. Second, we propose an alternative way of passively measuring human breath from face videos. We demonstrate a comparatively identical breath waveform estimation when compared against the breath waveforms produced by an ADInstruments device (baseline) (Adinstruments, http://​www.​adinstruments.​com/​ [1]). Third, we demonstrate good human recognition performance based on partial facial features when using the proposed pre-processing Motion Classification and Feature Clustering techniques. Our approach achieves increased identification rates across all datasets used, and it yields a significantly high identification rate, ranging from 96 to 100% when using a single or a combination of facial features. The approach yields an average of 7% rank-1 rate increase, when compared to the baseline scenario. To the best of our knowledge, this is the first time that a biometric recognition system positively exploits human breath waveforms, which when fused with partial facial features, it increases a benchmark face-based recognition performance established using academic face matching algorithms.

Face profile, the side view of the face, provides biometric discriminative information complimentary to the information provided by frontal view face images. Biometric systems that deal with non-cooperative individuals in unconstrained environments, such as those encountered in surveillance applications, can benefit from profile face images. Part of a profile face image is the human ear, which is referred to as the auricle. Human ears have discriminative information across individuals and thus, are useful for human recognition. In the current literature, there is no clear definition for what a face profile is. In this study, we discuss challenges related to this problem from recognition performance aspect to identify which parts of the head side view provide distinctive identity cues. We perform an evaluation study assessing the recognition performance of the distinct parts of the head side view using four databases (FERET, WVU, UND, and USTB). The contributions of this paper are three-fold: (i) by investigating which parts of the head side view increase the probability of successful human authentication, we determined that ears provide main features in the head side view. The rank-1 identification performance using the ear alone is about 90%. (ii) we examined various feature extraction methods to learn the best features for head side view and auricle recognition including shape-based, namely Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF); and texture-based, namely Multi scale Local Binary Patterns (MLBP), Local Ternary Patterns (LTP). We determined that texture-based techniques perform better considering that the MLBP yielded the best performance with 90.20% rank-1 identification; and (iii) we evaluated the effect of different fusion schemes, at the image, feature, and score levels, on the recognition performance. Weighted Score fusion of face profile and ear has the best score with 91.14% rank-1 identification.

With the advancement in iris recognition at a distance, cross-spectral iris matching is emerging as a hot topic. The importance of cross-spectral matching stems from the feasibility of performing matching in several security applications such as watch-list identification, security surveillance and hazard assessment. Typically, a person’s iris images are captured under Near-Infrared light (NIR) but most of the security cameras operate in the Visible Light (VL) spectrum. In this work, we therefore propose two methods for cross-spectral iris recognition capable of matching iris images in different lighting conditions. The first method is designed to work with registered iris images. The key idea is to synthesize the corresponding NIR images from the VL images using Artificial Neural Networks (ANN). The second one is capable of working with unregistered iris images based on integrating the Gabor filter with different photometric normalization models and descriptors along with decision level fusion to achieve the cross-spectral matching. Experimental and comparative results on the UTIRIS and the PolyU databases demonstrate that the proposed methods achieve promising results. In addition, the results indicate that the VL and NIR images provide complementary features for the iris pattern and their fusion improves the recognition performance.

This chapter discusses the use of infrared imaging to perform surveillance and recognition where the face is used for recognizing individuals. In particular, it explores properties of the infrared (IR) band, effects of indoor and outdoor illumination on face recognition (FR) and a framework for both homogeneous and heterogeneous FR systems using multi-spectral sensors. The main benefit of mid-wave infrared and long-wave infrared (MWIR, LWIR) camera sensors is the capability to use FR systems when operating in difficult environmental conditions, such as in low light or complete darkness. This allows for the potential to detect and acquire face images of different subjects without actively illuminating the subject, based on their passively emitted thermal signatures. In this chapter, we demonstrate that by utilizing the “passive” infrared band, facial features can be captured irrespective of illumination (e.g. indoor vs. outdoor). For homogeneous FR systems, we formulate and develop an efficient, semi-automated, direct matching-based FR framework, that is designed to operate efficiently when face data is captured using either visible or IR sensors. Thus, it can be applied in both daytime and nighttime environments. The second framework aims to solve the heterogeneous, cross-spectral FR problem, enabling recognition in the MWIR and LWIR bands based on images of subjects in the visible spectrum.

In this chapter, we propose a convolutional neural network (CNN) based classification framework. Our proposed CNN framework is designed to automatically categorizes face data into individual wavelengths before the face recognition algorithms (pre-processing, feature extraction and matching) are used. Our main objective is to study the impact of classification of multi-wavelength images into individual wavelengths, when using a challenging single sensor multi-wavelength face database in short wavelength infrared (SWIR) band, for the purpose of improving heterogeneous face recognition in law enforcement and surveillance applications. Multi-wavelength database is composed of the face images captured at five different SWIR wavelengths ranging from 1150  nm to 1550 nm in increments of 100 nm. For classification based on CNN networks, there are no pre-trained multi-wavelength models available for our challenging SWIR datasets. To deal with this issue, we trained the models on our database and empirically optimized the model parameters (e.g. epoch and momentum) such that classification is performed more accurately. After classification, a set of face matching experiments is performed where a proposed face matching fusion approach is used indicating that, when fusion is supported by our classification framework, the rank-1 identification rate is significantly improved, namely when no classification is used. For example, face matching rank-1 identification accuracy, when using all data is 63% versus 80% when data is automatically classified into a face dataset where face images were captured at 1550 nm wavelength.

This chapter deals with the issue of surveillance and reconnaissance through space based earth observation commercial satellites, since they first appeared as a technology available to anyone forty five (45) years ago. Analysis divides this long period into “three generations of Earth Observation satellites”. The first two generations, are distinguished from the third, using the criterion of derived products. The first two generations products were still imagery, while the third has added video too. The criterion of distinguishing the first and second generation, is the spatial resolution of their still imagery. As a result, the first provided general area surveillance, while the second provided reconnaissance. Implementing this criterion, these two first generations have lasted twenty six (26) years the first and fourteen (14) years the second accordingly. This chapter argues that the biggest breakthrough in decades actually occurred in 2014. A third generation has already breakout that year: it is the generation that has very high spatial resolutions in still imagery and videos too. This is also supplemented with high (to very high) temporal resolution (the time between image acquisitions). Minimizing the time between image acquisitions, is a key requirement for governmental users. It is this last technical characteristic which aims to cover the existing gap on the global quest (mainly from the users in the security and defence domain) for a permanent observation capability.

The increasingly complex asymmetric threats against people or infrastructure demand continuously improving surveillance strategies. Such strategies should be based on past experience, available technological capacities and foreseeable technological improvements and breakthroughs. Since rarely can either the location or the time of an attack be anticipated, efficient real-time surveillance of a potential target is of paramount importance. Such surveillance is usually performed by dedicated devices carried on manned or unmanned land, sea or air platforms. The use of manned platforms for such tasks could entail high risks for the crew, while unmanned ones can operate with considerable less constrains and in harsher environments. This later type of platforms are in fact “robots” able to be either remotely controlled or operating autonomously. The ensuing chapters present the key technological and operational elements that make remotely controlled and/or autonomous vehicles ideal for the execution of surveillance tasks for both civil and military applications and the challenges, be it technical or ethical, that such operational applications may face.

Since the 2nd World War and during the Cold War, the air defense radar has proven to be the main surveillance sensor, where each radar would cover a radius of more than 200 nautical miles. Apart from the electronic warfare, more recently the emergence of stealth or low observable technology, the evolution of ballistic and cruise missiles, as well as the democratization of UAVs (Unmanned Air Vehicles) or drones, have contested the capabilities of the typical surveillance radar. All these targets are difficult to detect, because they exhibit low RCS (Radar Cross Section), potentially flying at the upper or lower limits of the radar coverage or outside the expected velocity range (being either too slow, e.g. some UAVs, or too fast, like ballistic missiles). This chapter begins with the estimation of the RCS of various potential targets, as a function of the radar frequency band. In this way, the expected detection range against a set of targets can be calculated, for any given radar. Secondly, different radar types are taken into consideration, such as low frequency band radars or passive/multistatic radars, examining the respective advantages and disadvantages. Finally, some issues are discussed concerning the “kill chain” against difficult-to-detect targets, in an effort to defend efficiently the air space.

Maritime surveillance operations are needed worldwide to monitor and reassure safety and security across the seas. Numerous devices are employed in order to provide situational awareness of the vast sea. Lots of different technologies are involved to provide multiple views and clarify maritime conditions at a given time and place, however making interoperability a real challenge. The task is even more tedious as there is a key request to provide a single window view for multiple even all possible inputs. In this work we present an integrated mobile fusion solution for multiple tracking and monitoring sensors (e.g. low weight/high performance radar, position transmission mechanisms and electro-optic/systems and hyper-spectral sensors) to assist the detection and early identification and tracking of moving targets (e.g. with moving target indication and data fusion/correlation capabilities), as well as methods for obstacle detection and maritime surveillance. This innovative single window mobile platform presents high efficiency, low operational costs profiles and contributes to standardization in construction as it utilizes typical tracking infrastructure and standard smartphones and tablets.

Engineered microorganisms, microorganisms traveling through massive human transportation systems to intercontinental distances and the natural processes for fast-track microorganism evolution, especially to counter antibiotics, secure the continuous presence of infectious diseases within the foreseeable future. In consequence, bioweaponeers, especially bioterrorists, will find excellent grounds for nefarious improvisations by exploiting novel agents with enhanced virulence characteristics, deliverable by various means but especially by spraying aerosolized forms. To counter the spread of such agents, along with the just as hazardous prospect of unintentional harmful agent release due to biotechnological accidents, more stringent biosurveillance schemes must be enacted, possibly 24/7, preferably integrating networking principles, state-of-the-art assets for both sensing and sampling applications and the use of inexpensive unmanned platforms, preferably mobile and even better moving in three dimensions (UAVs) so as to increase the reach, depth and persistence within surveyed space. The new tendency in post-sampling sensors will be prepackaged point-of-care assays with limited or no need of energy source and consumables, detecting 3-D structures or the nucleic acid signal for identifying agents so as to implement reactive, targeted countermeasures (decontamination, treatment). Proactive, general countermeasures, such as alert, sampling procedures and protective measures will depend on pre-sampling sensors, operating on spectroscopic principles and usually using UV-Laser Induced Fluorescence principle and carried onto manned and unmanned aerial and ground platforms designed for reconnaissance and surveillance with exchangeable payloads.

This chapter examines the use of Social Networks for Surveillance and Security in relation to the deployment of intelligence resources in the UK. The chapter documents the rise of Military Intelligence agencies during both World Wars (such as GCHQ and MI5), and the subsequent use of these institutions to maintain order during peacetime. In addition to the way in which military organisations have used clandestine techniques such as double agents, spies, and various programmes designed for conducting Signals Intelligence, the Chapter offers an insight into how contemporary modes of communication (via mobile devices and the internet), shape the way in which intelligence agencies now gather information. The chapter also considers how the UK’s intelligence community responds to National Security issues such as international terror attacks, and how additional threats such as political subversion are framed in National Security discourse as being the legitimising factors behind mass surveillance. Thereafter, the chapter examines how online techniques are used by Britain’s intelligence agencies to maintain National Security, and how counter-intelligence strategies are being turned against the population to encourage political compliance. The chapter examines how online espionage techniques for entrapment, coercion, and misdirection, are being used to make something happen in the real or digital world.

After the attacks of 9/11 and the subsequent war against terrorism, many questions arose concerning the difficulties of observing international humanitarian law during asymmetrical warfare. On one hand, terrorists do not pay attention to the Geneva Conventions or any other treaties concerning the respect of human rights, the protection of non-combatants or the permissible means of fighting; instead, they attack innocent people to accomplish their goals and put pressure on their opponents.

In the context of an increasing dependence on multimedia contents, data hiding techniques, such as watermarking and steganography, are becoming more and more important. Due to the complementary nature of their general requirements, i.e., imperceptibility, robustness, security and capacity, many data hiding schemes endeavour to find the optimal performances by applying various approaches inspired from nature. In this paper, we provide a review and analysis of the main computational intelligence approaches, including Artificial Neural Networks (ANNs) and Fuzzy Sets (FSs), which are employed in information hiding. Furthermore, with the aid of the recent state of the art, we discuss the main challenges to be addressed and future directions of research.

Twitter is a microblogging service where users can post short messages and communicate with millions of users instantaneously. Twitter has been used for marketing, political campaigns, and during catastrophic events. Unfortunately, Twitter has been exploited by spammers and cybercriminals to post spam, spread malware, and launch different kinds of cyberattacks. The ease of following another user on Twitter, the posting of shortened URLs in tweets, the use of trending hashtags in tweets, and so on, have made innocent users the victims of various cyberattacks. This chapter reviews recent methods to detect spam, spammers, cybercus content, and suspicious users on Twitter. It also presents a unified framework for modeling hreats on Twitter are discussed, specifically in the context of big data and adversarial machine learning.

Complex organizational environments require highly-skilled employees who are both good at their everyday work and at the same time digitally literate, capable of using communication platforms and social media. Moreover, the familiarization of employees with technology and their tendency to bring their own devices at work, has created an additional headache for information security officers who fear that several backdoors can be opened to the organization security infrastructure not only by the misuse of the devices but also by a potentially highly-skilled employee. The proposed, in this chapter, social profiler tool aims at identifying potential inside threats using organizational information i.e., communication messages either from emails or social media. The information collected is then analyzed using a custom vocabulary which contains keywords related to the sensitive information of the organization in order to produce a list of employees who can potentially become insider threats. Finally, the social profiler tool incorporates six different visualizations of the employees under investigation with attributes such as their behavioral profile, ego network, word cloud, and a comparative profile of each employee in contrast to other profiles in their network. The tool’s effectiveness has been tested with an actual business communication dataset using a well-established generic vocabulary demonstrating promising results.

High performance computers (HPCs) have contributed to rapid scientific discovery and global economic prosperity as well as defense-related applications. However, their complex nature makes them difficult to troubleshoot thus questioning their reliability. As a result, these supercomputing systems are susceptible to malicious behavior or cyber attacks. Similar investigations have been made in the context of malicious objects in computer networks; however, limited attention has been given in the context of large-scale parallel systems. In this chapter, we present a sophisticated process that characterizes observed failures in supercomputing infrastructures due to variations of consistent intentional attacks. First, we present a data network extrapolation (DNE) process that automatically does failure accounting and error checking while considering a HPC tree-like reliability infrastructure. Next, dynamic and static characterization of failures are performed. By introducing a normalization metric, we observe that the complete spectrum of failure observations is deterministic in nature that depends on the total number of failed jobs, the time between processed jobs, and the total number of processed jobs per node. Our simulations using the Structural Simulation Toolkit (SST) show that our approach is highly effective for dynamically and statically representing observed failures. Furthermore, our results can be applied for improving job-based scheduling in supercomputing environments.