Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation

Figure 3 shows a diagram of our architecture; it is based on a deep, multilayer neural network that consists of successive 3D convolutional and pooling layers. The goal of CNN pose regression is to obtain 3D Cartesian coordinates of J joints given the multi-channel 3D probabilistic visual hull volume. The target of the network is \(3*J\)-dimensional vector comprised of the concatenation of the x, y, z coordinates of the J joints of the human body, for our work \(J=17\), resulting in 51 final layer embedding (\(3*17\)).

The detailed filter parameters are listed in Table 1 for each layer in Fig. 3. By using 3D convolution filters, we are able to encode information from all cameras as a volume simultaneously. In training, the network is supervised with an L2 regression loss:where \(p_{gt}^j\) is the groundtruth location for joint j and \(p_{pr}^j\) is the predicted location for joint j. The location of each joint is expressed globally, normalised to a root joint or node at the pelvis. To further encourage pose invariance with respect to the facing direction of the performer, the training data is augmented by applying a random rotation about the central vertical axis, \(\theta =[0,2\pi ]\).

Two visual channels are employed, a 2D occupancy matte, and semantic 2D joints. The occupancy is a soft probability of foreground occupancy formed from the comparison of the current frame I and a clean-plate P taken before the recorded sequence. The thresholded L2 distance between the two images in the HSV colour domain provides the soft occupancy probability for the 1st channel. The second semantic channel consists of a human joint belief labels estimated by OpenPose (Wei et al. 2016; Cao et al. 2017), a multi-stage process that iteratively refines 2D pose estimations of joint positions using a mixture of knowledge of the image and the estimates of joint locations of the previous stage. At each stage s and for each joint label j the algorithm returns dense per pixel belief maps \(m^{j}_{s}\), which provides the confidence of a joint centre for any given pixel (x, y), and given stage s. Much of the algorithm’s power is that in stages \(s \in {2, . . . , S}\) the belief maps are a function of not just the information contained in the image but also the information computed by the previous stage. For this work we transform these per joint belief maps into a single label image M, by maximising over the confidence of all possible joint labels on a per pixel basis.Figure 4 shows an example of the soft occupancy and joint labels for an example image.

Many recent approaches use multiple 2D views (Pavlakos et al. 2017b) or infer 3D from a learnt 2D lookup (Tome et al. 2017; Chen and Ramanan 2017). However, we propose to simultaneously use multiple 2D views to produce a crude but accurate 3D representation of the human body. Integrating the multiple views into a 3D shape overcomes the unavoidable ambiguities and occlusions present in individual 2D images. However, the cost is the exponential increase in dimensionality over 2D, and also the lack of a pre-trained imagenet based model (Krizhevsky et al. 2012). Therefore to allow the training to be tractable and still provide the increase in detail over 2D, we propose to use a multi-channel based probabilistic visual hull (PVH) (Grauman et al. 2003) to infer the 3D occupancy shape from multiple camera views. A PVH quantises the volume occupancy in a soft probabilistic computation that greatly reduces the dimensionality while maintaining the detail. The volumetric representation is agnostic to the source of the data, and for this work, we propose to use both 2D foreground occupancy mattes and semantic 2D joint labels. Both are noisy and contain failure cases as a single view. However, the probabilistic nature of the PVH ensures that noise is ignored and only a consistent signal is propagated to the 3D volume.

Given a set of C wide baseline cameras, \(c=\left[ 1, \dots , C\right] \), where \(C>3\) surrounding a performance volume, and calibrated with a known orientation, \(\mathbf {R}_c\), focal point \({COP}_c\), focal length \(f_c\) and optical centre \(o^x_c, o^y_c\), the camera parameters for a given camera c areThe 3D Capture Volume is finely decimated into voxels \(v=\left[ 1, \dots , V\right] \) approximately \(10\,\mathrm {mm}^3\) in size. Then given an 2D image denoted as \(I_c\), with \(\Phi =\left[ 1, \dots , \phi \right] \) channels the voxel occupancy from a given camera view c is defined as the probability:where given a 2D image coordinate position (x, y) the voxel \(v_i\) projects to a real world 3D position of:where \(\left[ \begin{array}{ccc}x&y&z \end{array}\right] \) is the 3D real world global coordinate location. Therefore the overall probability of occupancy for a given voxel \(p(v,\phi )\) is the product over all views:this is then computed for all voxels in the volumeThe fine grained voxel occupancy approximation is then down sampled via a weighted Gaussian filter to the coarse input shape and size of the first layer in the convnet, 30 \(\times \) 30 \(\times \) 30, this roughly approximates with the same number of pixels as a \(150 \times 150\) 2D image, where each voxel approximates a 67 \(\times \) 67 \(\times \) 67 mm volume in the real world.

To estimate the pose from joint orientations, Xsens IMUs (Roetenberg et al. 2009) are placed on key body parts to estimate the pose. The end rigid joints provide the most discriminative data and will constrain the pose parameters effectively when fused later with the vision. The pose optimisation of Malleson et al. (2017) is used, this aims to minimize the energy of the following Equation:where \(E_R(\theta )\), and \(E_A(\theta )\) are orientation and acceleration constraints, respectively and \(E_{PP} (\theta )\) and \(E_{PD}(\theta )\) are the pose projection and pose deviation priors, respectively.

For each IMU, \(k \in [1, 13]\), we assume rigid attachment to a bone and calibrate the relative orientation, \(\mathbf {R}^k_{kb}\), between the IMU k and the bone b. The reference frame of the IMUs, \(\mathbf {R}_{kw}\), is also calibrated approximately against the global world w coordinates. Each local IMU orientation measurement, \(\mathbf {R}^k_{m}\), is transformed to a global bone orientation, \(\mathbf {R}^k_{b}\) as follows:Then the local (hierarchical) joint rotation, \(\mathbf {R}^k_h\), for a given bone b in the skeleton is inferred by the kinematic chain:where par(b) is the parent of bone b. The forward kinematics begins at the root and proceeds down the joint tree (with unmeasured bones kept fixed).

In addition to orientation, the IMUs provide local acceleration measurements and a window of three frames, t (current frame), and previous two frames t1 and t2 is used. For each IMU, a constraint is added which seeks to minimize the difference between the measured and solved acceleration of the track target site. The solved acceleration is computed using central finite differences using the solved pose from previous two frames along with the current frame being solved. The local accelerations from the previous frames of IMU data are converted to global coordinates in a similar method to Eq. 10 but gravity is also removed.

We use two priors based on the PCA of the pose: PCA projection (\(E_{PP}\)) and PCA deviation (\(E_{ED}\)). The projection prior encourages the solved body pose to lie close to the reduced dimensionality subspace of prior poses (a soft reduction in the degrees of freedom of the joints), while the deviation prior discourages deviation from the prior observed pose variation (soft joint rotation limits). Together these terms produce soft constraints that yield plausible motion while not strictly enforcing a reduced dimensionality on the solved pose, thus allowing novel motion to be more faithfully reproduced at run time. For full details of the cost functions used please see Malleson et al. (2017).

These joint orientations in conjunction with the calibrated performer’s skeleton allow for joints locations to be inferred to a concatenated joint vector \(\mathbf {J}_i\). For a more detailed description of relating inertial data to other sensor model coordinate systems the work by Baak et al. (2010) can provide further details. To temporally align the IMU and video data an initial foot stamp was performed by the subject, which was visible in the video and produces a strong peak in acceleration in the IMU data. The inertial reference frame of each IMU, \(\mathbf {R}^k_{kw}\) is assumed to be consistent between IMUs and in alignment with the world coordinates through the global up direction and magnetic north. The IMU-bone positions \(t_{kb}\) are specified by manual visual alignment and the IMU-bone orientations Rib are calibrated using the measured orientations with the subject in a known pose (the T-pose, facing the direction of a given axis).

Given the temporal nature of human pose sequences, it is desirable to learn and enforce temporal consistency on the two streams of per frame pose estimation. Thus allowing the rich temporal motion patterns between frames and joints to be effectively incorporated into the 3D pose prediction. Long Short Term Memory (LSTM) layers (Hochreiter and Schmidhuber 1997) have provided excellent performance in exploiting longer term temporal correlations compared to standard recurrent neural networks on many tasks, e.g. speech recognition (Sak et al. 2014) and video description (Donahue et al. 2015). LSTM layers can store and access information over long periods of time but mitigate the vanishing gradient problem common in RNNs through a specialised gating mechanism.

Given an input vector \(\mathbf {J}_i(t)\) at time t consisting of concatenated joint spatial coordinates and resulting output joint vector \(\mathbf {J}_o(t)\). The aim is to learn the function that minimises the loss between the input vector and the output vector \(\mathbf {J}_o = o_t \circ tanh(c_t)\) (\(\circ \) denotes the Hadamard product), \(o_t\) is the output gate, and \(c_t\) is the memory cell, a combination of the previous memory \(c_{t-1}\) multiplied by a forget gate, and the input gate as shown in Fig. 5. Thus, intuitively it is a combination of the previous memory and the new input. For example, the old memory could be completely ignored (forget gate all 0s) or ignore the newly computed state completely (input gate all 0s), but in practice it is of course between those two extremes. The memory cell \(c_t\) is shown in Eq. 12.

Within each gates in there are two weights that are learnt, W and U. The input gate \(i_t\) defines the extent to which the newly computed state for the current input \(\mathbf {J}_i(t)\) is kept in the memory,A forget gate \(f_t\) defines how much of the previous state remains in memory,and an output gate \(o_t \) defines how much of the internal state is exposed to the external network (higher layers and the next time step).To learn the weights, they are trained using back propagation employing the loss function from Eq. 1 Each data modality has a distinct layer, with the temporal consistency using the previous f frames to predict the current frame joint vector for both the visual and IMU pose based estimation. With two layers both with 1024 memory cells, a look back of \(f=5\) and a learning rate of \(10^{-3}\) trained with RMS-prop (Dauphin et al. 2015).

The vision and IMU sensors both independently provide a 3D coordinate per joint estimate to reconstruct the performer’s pose. Therefore, it would make sense to incorporate both modes into the final estimate, given their complementary nature. Naively, an average pool of the two joint estimates could be used; this would be fast and efficient assuming both modalities have small errors. However, it is likely that often significant errors will be present on one of the modes due to their different measurement approaches. We, therefore, propose to fuse the two modes with a further fully connected layer. We are able to utilise the idea of using a dense layer to fuse our visual and IMU joint skeleton predictions, that can combine both measurements in a more meaningful way than simply taking the average. This allows errors in the pose from the vision and IMU to be identified and corrected by the combined fused model. This fully connected dense layer consists of 64 units and was trained with an RMS-prop optimiser (Dauphin et al. 2015) with a learning rate of \(10^{-4}\) to provide the feedback to reinforce the prediction. All stages of the model are implemented using Tensorflow.

We evaluate 3D pose estimation on the Human 3.6M dataset (Ionescu et al. 2014) where 3D ground truth key points are available from a marker-based motion capture system. It consists of 3.6 million video frames captured on four camera viewpoints in a 360-degree arrangement. There are five female and six male subjects, performing typical activities such as posing, sitting and giving directions. There is no IMU data within the dataset, and so we only evaluate the visual component, the PVH + LSTM. This is the upper red and green layers from Fig. 3 without the fusion of the IMU kinematic solve. To allow comparison to other approaches we follow the same data partition protocol as in previous works  (Ionescu et al. 2014; Li et al. 2015; Tekin et al. 2016, 2016a; Tome et al. 2017; Gilbert et al. 2017). The training data consists of subjects S1, S5, S6, S7, S8 and it is tested on unseen subjects S9, S11. The standard 3D Euclidean error metric is used to evaluated accuracy, it calculates the Euclidean error averaged over all frames and 17 joints (in human 3.6M) in millimetres (mm). The Results of our multi-channel 3D volumetric approach with the temporal consistency are evaluated qualitatively in Fig. 6 and quantitatively in Table 2, in particular we compare to the approach of  Mude Lin Liang Lin and Cheng (2017) who use 2D joint estimates with a 3D recurrent network,  Tome et al. (2017), which infers 3D probabilistic estimates from monocular 2D joint predictions. Also we compare to a baseline approach Tri CPM LSTM, a 3D triangulated version of the 2D pose estimation (Cao et al. 2016) with error rejection. In this approach per camera 2D joint estimatesare triangulated into a 3D point, using an error rejection method that maximises the number of 2D estimates with the lowest 3D re-projection error. This is a frame wise detection based approach, and therefore temporal consistency is introduced with two learnt LSTM layers as described in Sect. 3.5, Tri CPM LSTM.

As one can see from Table 2, our proposed approach outperforms all compared methods at time of publication [the newer works of Martinez et al. (2017) and Trumble et al. (2018)] indicate the speed of improvement in field of 3D pose estimation) despite excluding the fusion with the kinematic based IMU, with the mean error reduced by 15% compared with (Tome et al. 2017), the Tri CPM LSTM approach and our previous method (Gilbert et al. 2017). While compared to the state of the art results by Mude Lin Liang Lin and Cheng (2017), many activities have a similar error around 5 or 6cm. However, there is marked performance improvement in our approach for the activities; dog walking and sitting down, while Lin achieves better performance for greeting and waiting. Qualitative comparison to the ground truth is shown in Fig. 6, it shows the high degree of accuracy achievable, representing complex human poses. Although as shown in the bottom right pose, some unusual poses, probably not sufficiently represented in the training data, are still poorly estimated. To validate the superiority of the proposed multi-channel and temporally consistent approach, we evaluate the Human3.6M dataset with separate parts of the approach in Table 3.

It can be seen that the single channels of Matte or CPM based PVH perform worse than the multi-channel PVH, with both channels combined. This is likely to be due to the semantic information of the CPM labels complementing the occupancy based soft mattes. Also the improvement for enforcing the temporal consistency through the LSTM can be seen to be around 25 mm on average.

In recent years, high quality labelled datasets have been a catalyst for rapid development in a number of areas including object recognition (Deng et al. 2009) and 2D human pose datasets (Andriluka et al. 2014; Lin et al. 2014). These have been hand labelled, providing excellent accuracy and detail, however, this is far harder in 3D, where the labelling still in general relies on expensive and less common optical motion capture systems such as (http://​www.​vicon.​com). This constraint greatly reduces the quantity and variability of existing datasets; Table 4 shows the features of current 3D human pose datasets. As can be seen Human3.6M has a large amount of synchronised multi-view video and is popular, however no IMU sensor data. HumanEva, is a smaller dataset also with no IMU information. While TNT15, contains IMU data and MVV it is small in size. Given these restrictions, we propose a new dataset to address these short comings, TotalCapture.¹ It contains a large amount of MVV, and synchronised IMU and Vicon labelling for ground truth. It was captured indoors in a volume measuring roughly 8 \(\times \) 4 m with 8 calibrated HD video cameras at 60 Hz. The variation in the dataset is shown in Fig. 7. To provide accurate labelled ground truth, the optical marker based (http://​www.​vicon.​com) system was utilised, calculating 17 3D joint positions and angles, by triangulating small (\(0.5\,\mathrm {cm}^3\)) dots visible to infrared cameras, note these dots are not used explicitly by our algorithm, and their size is negligible compared to the performance volume. The IMU data is provided by 13 sensors on key body parts, head, upper/lower back, upper/lower arms and legs and feet, providing per unit orientation and acceleration. The location of the IMU sensors is shown in Fig. 8. The dataset consists of four male and one female subjects each performing four diverse performances, repeated three times: ROM, Walking, Acting and Freestyle, with each sequence lasting around 3000–5000 frames. An example of each performance and subject variation is shown in Fig. 7. There is a total of 1,892,176 frames of synchronised video, IMU and Vicon data (although some are withheld as test footage for unseen subjects). The variation and body motions contained in particular within the acting and freestyle sequences are very challenging with actions such as yoga, giving directions, bending over and crawling performed in both the train and test data. The train and test partitions are performed wrt to the subjects and sequences, the training consists of ROM1, 2, 3; Walking1, 3; Freestyle1, 2 and Acting1, 2 on subjects 1, 2 and 3. The test set is the performances Freestyle3 (FS3), Acting (A3) and Walking 2 (W2) on subjects 1, 2, 3, 4 and 5. This split allows for a comparison of unseen and seen subjects but always unseen sequences.

To provide a reference of our approach to other methods we compare to three state of the art approaches, the 3D triangulated CPM, Tri-CPM, described in Sect. 4.1 a flattened multi-view matte based 2D convolutional neural network approach (Trumble et al. 2016), 2D Matte, and our previously published results without the semantic 2D pose labels in the probabilistic visual hull (Gilbert et al. 2017). The results are shown with and without temporal consistency provided by the learnt LSTM model. As with Human3.6M, we show performance using the 3D Euclidean error metric over the 17 joints quantitatively in Table 5, and then qualitatively in Fig. 9 and in the accompanying video (The video is available at http://​youtu.​be/​CLDqpze53lU). The table shows that our combined semantic and occupancy based fusion with IMU approach outperforms all other methods, including our previous work (Gilbert et al. 2017) by 6 mm, and the triangulated CPM by 13 mm, which also performed well on the Human3.6M. The ability of the LSTM layers to introduce the temporal consistency and remove failure cases, improves all approaches by around 20 mm.

Our ablation study cumulatively enables each of our individual contributions on top of a classic baseline of a 3D Matte PVH. 3D pose estimation performance error is presented in Table 6 for separate parts of the approach.

The table shows how that the two channels of the PVH, 3D Matte PVH and 3D CPM PVH separately have a similar performance error, however by employing a two channel PVH it is possible to reduce the error by 20 mm. We also show the accuracy of using a 3 channel PVH (3D RGB Matte PVH ) with the foreground RGB pixel values instead, this performs worse, due to the increased dimensionally of the 3 channels but without the increased complementary knowledge that combining the occupancy and semantic label channels provides. With regards the IMU, the Raw IMU LSTM, uses the raw global orientation of the IMU units without an kinematic solve, an LSTM model is trained on the raw IMU input and this performs badly with nearly double the error of the Solved-IMU. Part of the reason for this higher error is likely to be due to sensor drift within the IMU being unable to be modelled correctly by the LSTM. However, through constraining the noisy IMU unit responses with inverse kinematics, we are able to negate the IMU sensor drift to some degree. By then fusing the SolvedIMU and two channel PVH, the error is further reduced. This is likely to be due to the complementary nature of the two data sources. Also, we show the result of just averaging the two data streams as the fusion method, this produces a high error, as expected as it is unable to learn anything about how the two data stream interact. It is possible to examine the per frame error for a sequence for subject 2 and sequence Acting3, in Fig. 10. Looking at the framewise errors, it shows that the two modes of data, the 3D PVH and SolvedIMU have lower errors at times, however through the use of the fusion layer, the overall error is lower than both. At around frame 1250, the Solved IMU increases in error due to a failure, however, the overall error rate of our proposed approach is relativity unchanged. While at frame 2500, the IMU is out performing the 3D PVH allowing the fused result to maintain a low error. However, at frame 4000 both modes fail, to cause higher errors in both data modes and the fused results, qualitative results of these three frames are shown in Fig. 11. For frame 4000 the higher errors can be seen to be caused by the arms not being extended correctly. The differences between the inferred poses can be quite small, indicating the contribution of all components of the approach. Although it’s important to notice that the errors in the SolvedIMU pose for frames 2800 and 4000 aren’t introduced to the final fused results. Run-time performance is 25 fps, including PVH generation. The ability of the approach to generalise between datasets is an interesting topic, therefore we compared applying a model trained on the TotalCapture dataset to the Human 3.6M dataset. We used the trained TotalCapture model from Table 6, 3D Matte CPM PVH-LSTM i.e. the input to the fusion layer (as we cant use a model that takes in IMU data on the Human3.6M dataset). Given the different number of cameras and far poorer resultant PVHs formed by human3.6M, we fine-tune the TotalCapture trained model on the human3.6M using unfixed weights with a single epoch of the Human3.6M training data (normally the model is trained with 100 epochs where an epoch is a complete pass of the training data). The new fine-tuned model was then shown all the test sequences from Human3.6M and achieved an average joint error of 75.3 mm. This is similar to the performance of our approach with exclusive training on Human3.6M of 71.9 mm as shown in Table 2. This indicates that the learnt model is similar, although a small amount of adaptation is required between the datasets due in this case to the poor PVH generalisation for the Human3.6M dataset, later in Sect. 5 we will shown results on the TotalCaptureOutdoor without any generalisation.

In this section, we explore and analyse some of the parameters in the approach. We investigate the effect of the number of cameras used, the amount of training data, the number of previous frames used for the temporal consistency and the effect the size of the voxels in the PVH volume has on the overall performance.