Spatio-temporal 3D Pose Estimation of Objects in Stereo Images

In this contribution we describe a vision system for model-based 3D detection and spatio-temporal pose estimation of objects in cluttered scenes. As low-level features, our approach requires 3D depth points along with information about their motion and the direction of the local intensity gradient. We extract these features by spacetime stereo based on local image intensity modelling. After applying a graph-based clustering approach to obtain an initial separation between the background and the object, a 3D model is adapted to the 3D point cloud based on an ICP-like optimisation technique, yielding the translational, rotational, and internal degrees of freedom of the object. We introduce an extended constraint line approach which allows to estimate the temporal derivatives of the translational and rotational pose parameters directly from the spacetime stereo data. Our system is evaluated in the scenario of person-independent “tracking by detection” of the hand-forearm limb moving in a non-uniform manner through a cluttered scene. The temporal derivatives of the current pose parameters are used for initialisation in the subsequent image. Typical accuracies of the estimation of pose differences between subsequent images are 1–3 mm for the translational motion, which is comparable to the pixel resolution, and 1–3 degrees for the rotational motion.