COMBI: Artificial Intelligence for Computer-Based Forensic Analysis of Persons

The learning process of OpenPose [1, 8] is based on extensive data sets [2, 13] containing labelled images capturing people in different poses from movies, sports, or everyday scenes. The labelled information describes 2D coordinates in the image that refer to a corresponding joint. Prior to the actual process, image sections, so-called windows, that surround coordinates of a joint are defined in different sizes. Ideally the coordinates of a joint are in the centre of each window, however that depends on their position in the image. The window sizes are determined by the following pixel dimensions: 9 \(\times\) 9, 26 \(\times\) 26, 60 \(\times\) 60, 96 \(\times\) 96, 160 \(\times\) 160, 240 \(\times\) 240, 320 \(\times\) 320 and 400 \(\times\) 400. This results in a denied number of windows of different sizes for a dened number of images of a training data set. Using histogram of oriented gradients (HOG), each window is then transformed into an abstract information representation of associated pixels that ultimately contributes to the formation of an informational foundation for the learning process. According to [11], the training was based on the MPII-Human-Pose (MP refers to multi-person) [2], LSP [6] and FLIC [9] data sets that have a combined capacity of 4297 images (28,000 from MPII + 11,000 from LSP + 3987 from FLIC). Consequently, for the training of joint recognition of both ankles, knees, shoulders, elbows, and wrists there are 3.438.960 image sections (8 windows * 10 joints * 42,987) serving as a training basis. During the learning process, patterns are derived from the previously abstracted image information, that are described mathematically using descriptors such as Euclidean vectors and learnt by the neural network. Finally, this learning process results in a prediction model that is specifically trained for the prediction of human joints. For the quantitative assessment of the trained prediction model, an evaluation was carried out in the same fashion as the training process using more than 2000 images of the LSP (1000 images) and Flic (1016 images). These were separated from the training data set prior to the training process. Thus, it forms an independent set of images for the evaluation process. This methodological foundation ultimately allows different prediction to be trained for a variety of object classes such as body regions or joints in the context of person recognition within images. At this stage, a cascaded application of these prediction models occurs: First, for the prediction of the whole person, followed by the prediction of individual body regions such as the upper and lower body, the upper and lower arm, etc. until finally the prediction and assessment of individual joints in from the image information of already predicted body regions. The predicted joints then serve as a basis that allows the human musculoskeletal system to be abstracted and converted into a 3D digital skeletal model, a so-called rig. In the COMBI research project, OpenPose was applied on the image material gained during the photogrammetric identification procedure as well as on the video frames from the simulated crime videos. This makes for a workable analysis that applied the same method on all data sets. An exemplary representation of predicted OpenPose key points, combined into an OpenPose skeleton (OpenPoseRig), is shown in Fig. 1.

Using terrestrial laser scanners allows to digitally capture the crime scene [3, 14, 16], convert the resulting 3D point cloud into a 3D model that can be displayed on the computer. Within the 3D model it is possible to conduct measurements and geometrical shape matching for instance. This only requires reference dimensions that are obtained with the terrestrial laser scans as well. With images or video frames, the challenge is to compare persons within the 2D information in a way that enables matches on a quantitative basis. Similarly to the 3D model, this can be achieved by using reference objects with known dimensions to infer measurements of an object in a video frame. For this purpose, the reference dimensions must either be in immediate proximity of the object that is to be measured or exhibit a well-defined geometry and size that allows inference of the size of the object in the image using classical photogrammetry. To close this gap, modern software application like Blender¹ [4, 7, 15] facilitates the merging of 2D (image) and 3D (scan) information. This combination results in a so-called 3D reference model, in which each video frame of a video sequence is linked to the 3D model of a crime scene. It permits measurements to be taken on objects in a video at any time that are always in reference to the depth information of the 3D model. Another benefit of this integrated information is the possibility of transferring objects between different 3D reference models to compare them with each other. Whilst doing so, the measurements of the transferred objects remain and are only scaled within the respective 3D reference model thus enabling a cross-camera comparison of objects. When applying this kind of object comparison to persons, the challenge arises to make the biomechanical motion sequence of a person comparable. The motion sequence is represented by a series of poses. A human pose is the posture of a person that results from the biomechanical movement of the human musculoskeletal system and that is assumed within an image or video frame. This means that the measurement of a person’s height in a frame with the measurement defined by measuring from the floor to the top of the head is dependent on the posture adopted by the person. Hence, it does not necessarily correspond to the person’s true body height. A pose-dependent height is the measurement from the floor to the top of head while the person is assuming a pose.

A person’s posture that is depicted in a video frame can be reconstructed by means of a 3D model of that person. This is done by superposing the frame onto the 3D model while at the same time ensuring that the reconstructed posture of the model corresponds to the real one from all perspectives. As the 3D model represents the dimensions of the real person, height can be directly estimated from the model. Body height or stature is the maximum pose -dependent height of a person when standing fully erect with the feet within hip width and the head oriented in the Frankfurt plane² similarly to the posture taken during an identification procedure. The body height of a person depicted in the frame is calculated in the metric 3D reference model using a mesh in form of a plane—which starting from the feet on the ground to the highest point of the head (corresponding to a vertex)—is placed sagitally in the centre of the rig and in this way acts as a measuring device. The vertical dimension of the plane then corresponds to the person’s body height in the frame. However, these measurements alone are not sufficient to assign a suspect to one of the perpetrators in a video since they are rather unspecific features to a distinct person. Nevertheless, it is associated with the musculoskeletal system which from the perspective of anthropometry is specific of a person. The method by the authors makes use of this specificity by matching the musculoskeletal system and its resulting body geometries of suspects, in particular their proportions of body parts, with perpetrators in the video. The matching of person-specific information from 2D material with the person-specific information of the real person needs to be possible if biometric identification from surveillance footage is to work. Hence, the problem is to translate suitable information from 2D to 3D and vice versa. In this light, a framework that has been developed to detect multiple persons in 2D material [12] is to be tried for its suitability for person-specific assignment. Consequently, a quantitative evaluation of similar poses provides the basis for a person classification. The number of poses that can be analysed depends primarily on the underlying image material. Furthermore, the poses have to fulfil certain requirements. One of these is that one leg must touch the ground in order to have a starting point for the analysis of the pose by a rig assignment using the person-specific rigs. Furthermore, no extremities should be covered by other parts of the body. The similarity of the poses is limited to the requirements mentioned. Depending on the underlying material and its quality, at least 15 poses for each person are taken as a basis for a rig assignment which is explained in detail in the following sections.

For the COMBI research project, study participants with different ages but approximately equal in numbers of females and males were acquired from the Lower Saxony police force. During several runs, images and videos of the study participants were recorded from a total of six camera perspectives at two different stations: one equipped with a turntable by stageonair³ as well as a camera for photogrammetric data collection and the other as a walking track recorded by five recording systems that operate in sync to simulate a crime scene (Figs. 3, 4). Both stations were situated in a large room provided by the Lower Saxony police force. While data will be collected at several stations, at this stage, however, the focus is only on two stations.

At these stations, study participants were photogrammetrically and videographically recorded in different states of clothing and while assuming different poses on the walking track (the simulated crime scene)—from standing upright to walking, bending, and turning etc.. Furthermore, for one run physical markers indicating specific anatomical landmarks on the joints were additionally attached to the participants. These markers formed the basis for subsequent rig derivations used evaluate OpenPose predictions.

In addition to the photogrammetric and video material collected this way, a second data basis consisting of 3D body scans provided by the company Avalution and stemming from the SizeGERMANY anthropometric survey in which more than 13,000 people were measured between 2007 and 2009 by means of 3D body laser scanning, was also taken into account to further evaluate the individuality of derived rigs. The data included here represents an extract of the survey sample and consists of 170 men and 170 women in two body height classes each (1.75 m and 1.85 m in the case of the men and 1.63 m and 1.73 m in the case of the women). To prove the individuality of the rigs, OpenPose performed on the image and video material as well as on the 3D body scans and both stations were scanned with a terrestrial laser scanner (type FARO FOCUS 3D X130) to create a metric 3D reference model of each station which in turn can be used to derive, fit, and measure the rigs.

As outlined in 2.1, a 3D model of the area which was recorded by surveillance cameras can be created by recording lengths and widths from and the distances between objects of that area. The model is part of a metric 3D reference model which provides the basis for matching of derived rigs of suspects with perpetrators in the video since it is important that such comparisons take place in 3D space. The next step in creating the metric 3D reference model is the alignment of virtual cameras in Blender: Virtual cameras are reconstructed at the exact positions of the actual surveillance cameras in the 3D model using fixed auxiliary objects such as door frames for the right orientation and rotation and parameterised according to the technical configuration if the physical camera such as focal length and sensor size. Once the alignment is complete, the footage from the actual surveillance cameras is imported and used as a background image for the virtual camera. Blender offers the possibility to look directly through the virtual camera, which in turn facilitates the superimposition of the imported footage on the underlying 3D reference model. This superimposition now allows true-to-scale measuring at any position in the video material. As the virtual camera acts in the same way as the actual surveillance camera (even considering distortions such as those caused by a fisheye lens), objects at a greater distance to the camera appear smaller than objects that are close to it and thus accurate measuring is possible at any position within the metric 3D reference model. The final element of the metric 3D reference model is then a selection of suitable video frames from the footage, i.e. where the perpetrator is close to the camera and with their extremities clearly visible, form the basis for comparing the derived rigs of suspects with perpetrators in the video (Fig. 5).

Images and video frames from the collected data as well as images taken of the 3D body scans were transferred into OpenPose. The basis for the prediction of the joints was the Body_25 model which represents the standard model including key points of the feet. The predictions created this way (Fig. 6) on images from the photogrammetry station were then used to generate person-specific rigs that were then compared to the predictions on frames from the footage of the simulated crime scene in the metric 3D reference system.

3D digital skeletons, so-called rigs, were created from images taken during photogrammetry to compare the anthropometric pattern between the study participants. Two kinds of rigs were generated for each participant that were either created based on physical joint markers (manually created rigs, Fig. 7) or on OpenPose predictions (automatically created rigs, Fig. 8). The physical joint markers were manually attached to the subjects prior to the photogrammetric procedure. They were placed onto the skin above palpable bony landmark within the articular area of the bone. At the same time, they approximate to the position of the keypoints of the OpenPose predictions which in turn serve as the basis for the second kind of rigs. The photogrammetric procedure provides images of the subject from all angles and allows for the creation of the person’s 3D model that can be both imported into Blender. Within a metric 3D reference model of the photogrammetric workstation, this data can be used to set up a minimum of three virtual cameras showing the subject from the frontal and both lateral views. Through the views the rig is then generated by linking the anatomical (with the physical markers for orientation) or OpenPose predicted joints. The derivation of a rig based on OpenPose includes the following predicted joints: ankles, knees, hip, shoulders, elbows, wrists and the key point for the nose. The single physical joint markers and predicted key points respectively can be merged into a rig via superimposition of the images on the 3D model within the metric 3D reference model using Blender.

The rig is then imported in the metric 3D reference model of the walking track (simulated crime scene) and aligned via its joints along the axes according to the position of the perpetrator in the video frame (Fig. 9), starting with the feet on the ground and finishing at the head. A possible match is assumed when the extremities and the joints can be perfectly aligned with those of the perpetrator. Not only the individual height of a person in a frame and rig respectively, but also the proportions of the extremities can be quantitatively measured in Blender by means of a virtual ruler and thus be included into the comparison.