Intelligent Mobile System for Improving Spatial Design Support and Security Inside Buildings

Nowadays, mobile robots are able to efficiently map indoor and outdoor environments. They experience no fatigue and many state of the art systems are robust enough to perform this task autonomously. We focus on an intelligent system capable of 3D mapping and searching objects. Thus, the robot is able to compare current 3D information with the reference model to locate new objects. These objects are considered as objects of potential interest OPI that are confirmed by a classification procedure. Accurate 3D models are also used in spatial design support. The spatial design process, in the sense of activity, is described as “Conception → Modeling → Evaluation → Remodeling” cycle in [19]. The work in this paper aims at improving the last three steps. Our goal is to propose a solution to end-users that allows easier improvement and redesigning of already existing areas, especially those with incomplete or obsolete floor plans. We have built an intelligent mobile system, capable of performing qualitative reasoning.

We propose a modeling → conceptualization → evaluation process based on a 3D scanning and a Qualitative Spatio-Temporal Representation and Reasoning (QSTRR) framework capable of modeling the environment in the sense of generating a qualitative representation. The Qualitative representation used within the context of spatial assistance concept is connected with a framework of multi-modal data access for spatial assistance systems [27]. The main goal of this work is to develop an assistance system for supporting designers in decision making based on the Qualitative Spatial Representation and Reasoning (QSRR) [12]. The study of spatial-structural design processes and computational tools such as described in [14] are an example of design support tools in early stages of the design process. As pointed in [13], the main challenge and limitation for QSRR systems is providing an appropriate mathematical tool set allowing for qualitative reasoning without traditional quantitative techniques. Bhatt [4] shows the importance of integrating QSRR techniques with artificial intelligence from the application point of view, such as cognitive robotics or spatial design support. The problem of reasoning about space is discussed in [25]. It simplifies an earlier theory presented in [23] and [24] considered the original approach based on a calculus proposed in [11]. The ontological primitives of the theory include physical objects and regions, an additional set of entities is also defined. The Spatial Assistance System (SAS) described in this paper is defined as a computer implemented decision making system [27] that possesses other analytic abilities that require training, knowledge and expertise. There exist many solutions for spatial perception, but the integration of various spatial knowledge sources for solving complicated problems is an important and complex research problem [2]. The integration problem arises from the fact that conducting the spatial task in a real environment is not bound with one activity but with a set of the mutually dependent tasks, which forces a minimal level of the coordination. Thus, it determines a rich set of the integration problems.

The first step of our spatial design support system consists of acquiring 3D point clouds of the area and merging them into one consistent 3D representation using an improved 6D simultaneous localization and mapping algorithm [20]. During last decade 3D metric information is widely used in inventarization problems [16, 18]. The improvement is related with computational aspects of parallel programming discussed in [7]. In this paper we show current research results of 6D SLAM from the application point of view, thus input data for the qualitative reasoning are based on accurate 3D models of the environment. We achieve better accuracy of final 3D metric map compared to our previous work [3], where we designed ground truth 3D data setup measured with geodetic precision. For spatial design purpose we create a QSTRR model based on this accurate 3D metric map. Finally the evaluation of the area, in connection to the selected spatial design problem, can be done using reasoning capabilities of QSTRR. The preliminary results on QSTRR framework for Spatial Design was shown in [7]. This paper is focused on the integrated intelligent mobile applications into a single mobile system capable of performing qualitative reasoning in the security domain and for spatial design support.

We designed an ontology for the purpose of encoding spatial design domain knowledge, thus allowing an intelligent mobile robot to perform qualitative reasoning. It allows building a semantic model of an environment using a qualitative spatio-temporal representation. An ontology O is composed of entities: a set of concepts C, a set of relations R, a set of axioms A (e.g. transitivity, reflexivity, symmetry of relations), a concepts’ hierarchy CH, a relations’ hierarchy RH, a set of spatio-temporal events E _{s t} . It is formulated as the following definition: An ontology is a core component of a qualitative reasoning mechanism. A concept is defined as a primitive spatial entity described by a shape S composed of points or polygons in 3D space. We associate a semantic label SL to each shape. Ontology distinguishes two different types of attributes that can be assigned to a concept, quantitative A _{q n} and qualitative A _{q l} . We extends the set of four values of qualitative attributes: real physical object, functional space, operational space and range space by the new attribute: empty space. This new approach helps in reasoning about trajectories in 3D space using state of the art path planning methods. Functional, operational and range spaces are related with spatial artifacts; detailed information are given in [5, 6, 27]. Quantitative attributes are related with physical properties of spatial entities and are as follows: location, mass, center of mass, moment of inertia (how much resistance there is to change the orientation about an axis), material (friction, restitution). Therefore, the definition of the concept C is formulated as:

The set of relations R is composed of quantitative and qualitative spatial relations. For topological spatial qualitative relations the Region Connected Calculus (RCC-8) [25] is used. The ontology includes eight different topological relations between two shapes: disconnected DC, externally connected EC, partial overlap PO, equal EQ, tangential proper part TPP and its inverse T P P i, non-tangential proper part NTPP and its inverse N T P P i. Quantitative spatial relations are used for constraining the way entities move relative to another. Our ontology defines following constraints: origins locked, orientations locked, origins free, orientations free, free rotation around one axis sliding. The ontology provides a mechanism for building world models that assume spatio-temporal relations in different time intervals. It captures changes in 3D space. Our temporal representation takes temporal intervals as a primitive [1]. We distinguish the following qualitative spatio-temporal events E _{s t} related with topological spatial relations RCC-8: These four qualitative spatio-temporal events are used to express the most important spatio-temporal relationships that holds between two concepts in different intervals of time within the context of Spatial Design. In this paper we focus on the two events: onEnter and onLeave. To store the instances of ontology-based elements an instance base I B ^O is defined: where \({I_{C}^{O}}\) contains instances of concepts C, \({I_{R}^{O}}\) contains instances of relations R and \(I_{E_{st}}^{O}\) contains instances of spatio-temporal events. Finally the semantic model is defined as a pair: where O is an ontology and I B ^O is an instance base related to ontology O. The ontology is known a-priori but the instance base is being updated during spatial design process. An important tool is the semantic map being defined as a projection of semantic model onto 3D space. All concepts, relations and events have their own user-friendly 3D representation; this is the core of the proposed intelligent mobile user interface.

Our mobile mapping systems are shown in Fig. 1. The mobile robots acquire 3D data and register them using improved 6D SLAM algorithm [10]. The lightweight mobile robot PIONEER 3DX is used for 3D mapping of indoor environments and search missions. The Husky robot can provide accurate 3D data with the range of 170 meters in a single 3D scan. We are using this robot to acquire data in the indoor and outdoor environments. We improved the 6D SLAM algorithm by use of semantic classification, loop closing with Complex Shape Histogram (CSH) and parallel implementation. It is shown in Fig. 2, i.e. we have extended the work described in [10]. Important aspects concerning our implementation is given in [8]. Green rectangles correspond to novel algorithmic components using high performance computing in Compute Unified Device Architecture (CUDA). Red rectangles correspond to qualitative reasoning components: semantic classification and semantic 3D scan matching. Semantic classification is used for discrimination of points, thus nearest neighborhood search procedure in semantic 3D scan matching uses this information for finding pairs of closest points having the same semantic label. Semantic labels denote shapes around query points and it is described in Section 4 within the context of computing CSHs. In CSH we use classification of shape round each query point into two classes: flat, not flat. In semantic 3D scan matching we the extend number of classes by ceiling and floor. Ceiling corresponds to points with label: flat above robot with normal vector pointing down, floor corresponds to points with label: flat below robot with normal vector pointing up. The main part of semantic 3D scan matching is a modified Iterative Closest Points (ICP) algorithm. Our improvements concentrate on a parallel computing implementation and discriminating points into four classes during the nearest neighborhood search procedure. The key concept of the ICP algorithm can be summarized in two steps:

Iteratively repeating these two steps results in convergence to the desired transformation. Semantic discrimination of these correspondences improves the convergence. Range images are defined as model set M where and data set D where The goal of semantic 3D scan matching is to minimize following cost function: where w _{i j} is assigned 1 if the i ^{t h} point of M correspond to the j ^{t h} point in D in the sense of minimum distance and the same class. Otherwise w _{i j} = 0. Class c discriminates points into flat, n o n f l a t, ceiling or floor. R is the rotation matrix, t is the translation matrix, m \(_{i}^{c}\) corresponds to points of class c from the model set M, d \(_{j}^{c}\) corresponds to points of class c from the data set D. The algorithmic representation of the process is shown in Listing 1. The semantic approach efficiently decreases the time of the 6D SLAM convergence, by requiring a lower number of iterations, especially in the situation where odometry readings are not sufficiently accurate. It is also more reliable than other state of the art approaches for demanding scenarios, such as moving down stairs or mapping harsh environments. We show an example in Section 6 in Fig. 8.

Recognition of 3D objects is composed of OPI detection and identification. The pipeline of the autonomous search is shown in Fig. 6. To detect 3D points of OPI we use a nearest neighborhood search procedure for each query point in the current 3D measurement. We perform ICP alignment of the current 3D scan to the global reference model. This step is necessary to minimize the number of false detections. Prepared point clouds are processed using Algorithm 2. All of the query points which do not have neighbors within the certain radius are considered as a parts of OPI. Points with existing nearest neighbor get the label non OPI. In the graphical interface OPIs are marked by the red color instead of black. An important issue is the proper initialization of the system by providing a reference 3D model with minimized number of dynamic obstacles.

To identify 3D objects we build a knowledge base, thus we prepare a training data set composed of objects with assigned semantic labels SL. Recognition of 3D objects is performed by classification of the observed object into semantic label based on knowledge base. In this application we have two semantic labels: OPI and not OPI. We assign these semantic labels manually, thus the advantage of the system is the capability to train for recognition of different objects. To characterize the 3D objects we use the CSH. The CSH (Algorithm 3) is an extension of Point Pair Feature (PPF) [15], which is similar to the surflet-pair feature from [26, 28]. The PPF method uses a point pair feature that describes the relative position and orientation of two oriented points. The feature F _PPF for two points m ₁ and m ₂ with normals n ₁ and n ₂ and distance d = m ₂−m ₁ is defined as: where ∠(a,b) ∈ [0; π] denotes the angle between two vectors. CSH reduces the dimension of the feature space and uses characterisation of shape assigned by semantic label around query point, thus where ∠(a , b)∈[0;π] denotes the angle between two vectors, c(a,b) ∈ (0,1,2,...n) denotes combination of semantic labels SL denoting shapes around query points a and b. In the current implementation of CSH: 0 corresponds to (flat,flat), 1 corresponds to flat, non flat) or (non flat, non flat), 2 corresponds to (non flat, non flat). The classification into flat and not flat labels is performed with Principal Component Analysis (PCA) of the neighborhood for a given query point. Fig. 3 depicts the feature in PPF and CSH. In our experiments the Euclidean distance between points is quantized to 20 buckets. The combination of shapes is quantized to 3 buckets and the angle between normal vectors computed for a pair of points is quantized to 5 buckets. The advantage of CSH over PPF is a reduction of feature space, thus assuming the same quantization the histogram of CSH is composed of 300 buckets and histogram of PPF is composed of 2500 buckets. A neglectable disadvantage of CSH is the higher computation complexity over PPF determined by classification of neighborhood for each query point into certain semantic label SL, thus a parallel algorithm for fast CSH computation is implemented. In average on a cloud of 15000 3D data points the CSH is computed within 4 seconds on GeForce GTX680 and within 1.7 second using an approximated method.

The intelligent mobile user interfaces are implemented using a Software as a Service SaaS model with the capability to perform High Performance Computing (HPC) in the Cloud. The HPC approach is a relatively new topic, especially in mobile robotics. We are using it for the task of 3D mapping and for detection of OPI. HPC with Graphic Processing Unit GPU virtualization technology allows many users to have access to computationally demanding applications in the Cloud. We have presented this technology applied for the Search and Rescue ICARUS system in the field [22]. We decided to provide needed computation power through a mobile data center based on an NVIDIA GRID server (Supermicro RZ-1240i-NVK2 with two VGX K2 cards - in total 4 GPUs with 4GB RAM each) capable of GPU virtualization. In the ICARUS project we have used the Citrix XenApp infrastructure for building the SaaS model. In XenApp, many users can share a single GPU by the use of Microsoft Windows Server 2012 system or a group of published applications. XenApp model allows sharing a single GPU for numerous application using CUDA. Published applications can be accessed as SaaS In the ICARUS project after installing the Citrix Receiver - thin client compatible with all popular operating systems for mobile devices (Mac OS X, Linux, Windows, iOS, Android etc.). The limitation is that the newest GRID compatible GPU drivers support only the CUDA 5.5 programming framework and the GRID GPU processors offer CUDA 3.0 compute capability. The provided functionalities are sufficient for our application. Besides CUDA, GRID technology supports building applications with 3D rendering (OpenGL 4.4). Thus, in our system it is used for remote graphical user interface for authorized personnel. Our system is able to improve global awareness of the situation by providing 3D renderings of the mapped environment with marked OPIs over Ethernet in real time. Our SaaS based security system enables integration with existing closed-circuit television systems, thus it reduces blind spots in existing safety infrastructure. The advantage of the approach is that all information from the robot is transmitted over Ethernet as renders, which is significantly faster than sending raw data. Thus, for example Crisis Management Centers will be able to use robotic information to coordinate crisis action. We tested the experimental setup in Institute of Mathematical Machines shown in Fig. 7. Our system is able provide intelligent mobile user interfaces for 20 users simultaneously. This allows to quick set up command and control centers in search and rescue scenarios

We performed an end user case study. This shows the need of providing following functionalities: Therefore, we performed the following experiments as a contribution of our research. All figures shows intelligent mobile interfaces available over Ethernet via a novel SaaS system based on the NVIDIA GRID technology.

In this paper we have shown the advantages of using the QSTRR framework in security and spatial design applications. The intelligent mobile system is designed for protection of urban environments including critical infrastructures. The method for searching of objects of potential interest was discussed and the results in real task scenarios were presented. The prototype of the system was tested in the Museum of the History of Polish Jews in Warsaw and Airport in Łódź, Poland. The system was able to detect changes in 3D environment and is able to improve the existing security system. Our spatial design support system, based on modeling-validation-remodeling philosophy, allows for creation of accurate model of given environment, modify it using specialized tools and validating the results with simulation. We generated accurate 3D metric models and by adding QSTRR layer, we modeled spatio-temporal events and relations in the environment, which allows for simulating not only the view of the area but also explore the qualitative functionalities of different configurations of spatial entities. The presented experiments show the proof of concept of the system functionality looking from end-user cases point of view. Use of parallel computation allows for increase in quality of the results and lower the computation time to reasonable levels. Our intelligent mobile interfaces implemented using the SaaS model allow accessing data over Ethernet, thus end-users can work simultaneously from different locations.

The proposed classification method uses 3D information of objects, thus we will investigate how 2D information from sensor such as camera could improve the performance of the system. There is no limitation for using 2D and 3D information since we can represent the objects using the certain feature space, thus in future work we will investigate the methods for the characterization of 2D objects. The research goal will be the classification system capable working with 2D and 3D information. The new system will be dedicated for cases and applications where user can only provide a 2D images as input.