Abstract: Enabling Geometry Aware Learning Through Differentiable Epipolar View Translation

Epipolar geometry is exploited in several applications in the field of Cone-Beam Computed Tomography (CBCT) imaging. By leveraging consistency conditions between multiple views of the same scene, motion artifacts can be minimized, the effects of beam hardening can be reduced, and segmentation masks can be refined. So far, these conditions have been formulated as optimization criteria to be minimized in post-processing. In this work, we explore the idea of enabling deep learning models to access the known geometrical relations between views in order to improve the prediction. The implicit 3D information contained in the relative pose between views can potentially enhance various projection domain algorithms such as segmentation, detection, or inpainting. Based on this hypothesis, we introduce a differentiable feature translation operator, which uses available projection matrices to calculate and integrate over the epipolar line in a second view. After geometrically translating all channels of a layer’s output feature map, we concatenate these activations to a second instance of the same architecture processing a second view of the same 3D scene. As an example application, we evaluate the effects of this operation on the task of projection domain metal segmentation. By re-sampling a stack of projections into orthogonal viewpairs,we segment each projection image jointly with a second view acquired 90° apart. The comparison with an equivalent single-view segmentation model reveals an improved segmentation performance of 0.95 over 0.91 measured by the dice coefficient. By providing an implementation of this operator as an open-access differentiable layer, we seek to enable future research [1].