Uncertainty Aware Deep Reinforcement Learning for Anatomical Landmark Detection in Medical Images

Deep reinforcement learning (DRL) is a promising technique for anatomical landmark detection in 3D medical images and a useful first step in automated medical imaging pathology detection. However, deployment of landmark detection in a pathology detection pipeline requires a self-assessment process to identify out-of-distribution images for manual review. We therefore propose a novel method derived from the full-width-half-maxima of q-value probability distributions for estimating the uncertainty of a distributional deep q-learning (dist-DQN) landmark detection agent. We trained two dist-DQN models targeting the locations of knee fibular styloid and intercondylar eminence of the tibia, using 1552 MR sequences (Sagittal PD, PDFS and T2FS) with an approximate 75%, 5%, 20% training, validation, and test split. Error for the two landmarks was 3.25 ± 0.12 mm and 3.06 ± 0.10 mm respectively (mean ± standard error). Mean error for the two landmarks was 28% lower than a non-distributional DQN baseline (3.16 ± 0.11 mm vs 4.36 ± 0.27 mm). Additionally, we demonstrate that the dist-DQN derived uncertainty metric has an AUC of 0.91 for predicting out-of-distribution images with a specificity of 0.77 at sensitivity 0.90, illustrating the double benefit of improved error rate and the ability to defer reviews to experts.