Adaptive Non-rigid Registration and Structure from Motion from Image Trajectories

This paper addresses the problem of registering a known 3D model to a set of 2D deforming image trajectories. The proposed approach can adapt to a scenario where the 3D model to register is not an exact description of the measured image data. This results in finding a 2D–3D registration, given the complexity of having both 2D deforming data and a coarse description of the image observations. The method acts in two distinct phases. First, an affine step computes a factorization for both the 2D image data and the 3D model using a joint subspace decomposition. This initial solution is then upgraded by finding the best projection to the image plane complying with the metric constraints given by a scaled orthographic camera. Both steps are computed efficiently in closed-form with the additional feature of being robust to degenerate motions which may possibly affect the 2D image data (i.e. lack of relevant rigid motion). A further extension of the approach allows to compute the full 3D deformations of the shape given the first initial (rigid) registration. This step results in solving a Non-rigid Structure from Motion (NRSfM) problem using the 3D known shape as a prior. Experimental results show the robustness of the method in registration tasks such as pose estimation and 3D reconstruction when degenerate image motion is present.