Prototypical Interaction Graph for Unsupervised Domain Adaptation in Surgical Instrument Segmentation

Surgical instrument segmentation is fundamental for the advanced computer-assisted system. The variability of the surgical scene, a major obstacle in this task, leads to the domain shift problem. Unsupervised domain adaptation (UDA) technique can be employed to solve this problem and adapt the model to various surgical scenarios. However, existing UDA methods ignore the relationship among different categories, hindering the model learning discriminative features from a global view. Additionally, the adversarial strategy utilized in these methods only narrows down the domain gap at the end of the network, leading to the poor feature alignment. To tackle above mentioned problems, we advance a semantic-prototype interaction graph (SePIG) framework for surgical instrument type segmentation to grasp the category-level relationship and further align the feature distribution. The proposed framework consists of prototypical inner-interaction graph (PI-Graph) and prototypical cross-interaction graph (PC-Graph). In PI-Graph, EM-Grouping module is designed to generate multi-prototypes representing the semantic information adequately. Then, propagation is performed upon these multi-prototypes to communicate semantic information inner each domain. Aiming at narrowing down the domain gaps, the PC-Graph constructs hierarchical graphs upon multi-prototypes and category centers, and conducts dynamic reasoning to exchange the correlated information among two domains. Extensive experiments on the EndoVis Instrument Segmentation 2017 \(\rightarrow \) 2018 scenarios demonstrate the superiority of our SePIG framework compared with state-of-the-art methods. Code is available at https://​github.​com/​CityU-AIM-Group/​SePIG.