Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles (UAVs): a review of related works

The goal of a UAV-driven visual performance monitoring procedure is to (1) collect images or videos from the most informative views on a project site, (2) analyze them with or without a priori BIM to reason about performance deviations during construction (e.g. progress and quality), (3) monitor ongoing operations for productivity and safety, (4) characterize the as-is conditions of existing civil infrastructure systems, and (5) quickly and frequently visualize and communicate the most updated state of work-in-progress with onsite and offsite project participants. Figure 1 illustrates an example of next generation construction site where camera-equipped UAVs autonomously monitor the construction performance. The following sections describe the most recent research in each procedure, their challenges, and future direction for research.

Providing accurate performance information about the state of construction or existing conditions of civil infrastructures requires UAVs to collect visual data in form of images and videos (e.g., digital: RGB; thermal: T; depth: D; digital + depth: RGB + D) from the most relevant locations and views on a project site. To streamline this process, research in UAV-driven visual data collection need to address the following challenges: (1) autonomous or semi-autonomous path planning, navigation, and take-off and landing procedures; (2) characterization of the criteria necessary for data collection, including the configurations among the images to guarantee complete as-built information; and (3) identification of the most informative views for data collection (e.g., canonical view, top down view, etc. for appearance-based recognition of work-in-progress as in (Han et al. 2015)). Table 1 categorizes prior works based on their level of autonomy and with respect to their specific applications.

UAV-driven data collection in practice still relies on experienced pilots navigating the UAVs on and around project sites although there have been recent efforts on Simultaneous Localization and Mapping (SLAM) techniques by the research community. Recent studies (Fernandez Galarreta et al. 2015; Kerle 2999; Zollmann et al. 2014) and commercial systems (DJI 2015) provide autonomous navigation and data collection capabilities using GPS waypoints and pre-determined flight path. These methods are beneficial in land surveying or monitoring low-structures with large footprint. Nevertheless, a GPS-driven flight planning method that builds on existing maps has the following limitations (Lin, Han, Fukuchi et al. 2015a): (1) does not account for dynamics on a construction site and their impact on safety (e.g. the location and orientation of temporary resources such as cranes); (2) can be negatively affected by loss/interference in GPS signal at interiors or due to shadowing effects caused by nearby buildings or other structures in densely populated metropolitan areas and high-rise buildings; and (3) can be subject to navigational hazards due to the loss of calibration on magnetometer sensors in proximity to structural and non-structural steel components. In the meantime, research on SLAM techniques for UAVs and ground robots has been mainstream in the Robotics community. Using high resolution laser scanners (Zhang & Singh, S. “LOAM: Lidar Odometry and Mapping in Real-time.” Proc., Robotics: Science and Systems Conference (RSS 2014), monocular cameras (Blo, x, sch, M et al. 2010; Lui et al. 2015), and RGB-D cameras (Loianno et al. 2015), these techniques generate 3D maps of unknown scenes and localize the robot in that environment. The latest efforts such as (Michael et al. 2014) have focused on experimenting a variant of SLAM using RGD-B for scanning post-disaster buildings, yet there is almost little to no work reported on actively testing these algorithms for producing 3D maps on civil infrastructure systems for construction monitoring or condition assessment purposes. Also, methods that account for evolving structures (i.e. large numbers of shoring and frames in room partitions) and dynamic objects (i.e. equipment and human) are not reported in the literature.

As can be seen from the literature, research is mainly focused on challenge 1, and challenge 2 and 3 still lacks investigation in terms of the number of studies conducted. For example, identifying the most informative views for observing different tasks with or without BIM, and collecting visual data to monitor locations, activities of equipment, and craft workers on a jobsite are not well studied. There is an opportunity for leveraging a priori information about geometry and appearance of a site via BIM. When integrated with schedules, BIM can also report on the most likely locations for expected changes on the site to steer data collection. A BIM-driven data collection method has potential to overcome some of these challenges mentioned above such as collision avoidance with structures. Together with 4D (3D + time) point clouds, BIM definitely has potential to minimize many technical challenges facing fully autonomous navigation and data collection for the built environment.

The analytics of visual data has been a research subject for more than a decade. Many image processing, computer vision, and geometrical processing techniques are developed that can (1) generate semantically-rich 3D models from collections of overlapping images; (2) manually, semi-automatically, or automatically conduct progress monitoring, surveying, safety inspection, quality monitoring and activity analysis during construction, and (3) streamline condition assessment in existing buildings and infrastructure systems. (Cho et al. 2015; Pătrăucean et al. 2015; Son et al. 2015; Teizer 2015; Yang et al. 2015) provide thorough reviews of these techniques. These all methods can be applied to visual data collected from UAVs, yet only a few studies have validated them in such contexts. Tables 2 and 3 summarize the most recent literature from the last few years that focus on methods that are exclusively developed or applied to images and videos from UAVs.

Figure 2 categorizes these methods based on their level of automation and with respect to specific applications. As seen, prior works are mainly based on image-based 3D reconstruction procedures to analyze unordered and uncalibrated image collections. While these methods generally work better on ordered and sequential images, yet without a rigid control on the location and viewpoint of the captured images with respect to onsite construction elements, the visual content of the images may not be most suitable for certain applications. For example, for appearance-based construction progress monitoring, leveraging material recognition techniques such as Han et al. (Han et al. 2015) benefits from images that are taken in a canonical view to identify the most updated status of construction for onsite elements. This requires research that identifies the most informative views, and then techniques that can leverage such information as a priori information for data collection. In addition, a detailed assessment of work-in-progress, quality, and existing conditions requires the analysis of geometry, appearance, and interconnectivity among the construction elements. It also requires techniques for as-built and as-damaged information modeling; nevertheless, techniques that address these issues are not well studied. Ongoing efforts such as the Flying Superintendents project at the University of Illinois (Lin, Han & Golparvar-Fard 2015), and the ARIA project at Carnegie Melon University (ARIA (Team 2015)) are geared towards providing specific frameworks together with methods for construction monitoring and civil infrastructure condition assessment to address some of these gaps-in-knowledge.

Achieving effective flow of information both to and from project sites and conducting actionable analytics for construction monitoring and condition assessment require intuitive visualization of the information produced throughout the process on top of the UAV visual data - images and point clouds. While attention to visual sensing and analytics has been the mainstream of the literature, less work is conducted on interactive visualization. A recent work by (Zollmann et al. 2014) introduces an interactive multi-layer 4D visualization of information captured and analyzed through a UAV-driven procedure in form of mobile augmented reality. Their system, similar to (Karsch 2999) overlays color-coded 3D construction progress information on the physical world and adopts filtering methods to avoid information clutter and issues associated with displaying detailed augmented information. Other recent examples by (Han et al. 2015; Lin, Han, Fukuchi et al. 2015a) introduce web-based tools with scalable system architectures for visualizing and manipulating large scale 4D point cloud data, large collections of images, 4D BIM, and other project information. These methods account for level of details in data representation and dynamically consider the limited computational power and connection bandwidth for visualizing data on commodity smartphones. Since BIM is hosted on the server side of their system architecture, these tools can support pull and push of the geometry and other semantic information from BIM. This provides access to the most updated information and does not require storing BIM locally on the client device (Fig. 3). More research is still needed to map, visualize, and explore modalities of user interaction with operation-level construction performance data (e.g. locations and activities of workers and equipment) and condition assessment information (e.g. the location and characteristics of defects) through these models, both on and off site.