Depth Image Matching Algorithm for Deforming and Cutting a Virtual Liver via Its Real Liver Image Captured Using Kinect v2

In this paper, we propose a smart deforming and/or cutting transcription algorithm for rheology objects such as human livers. Moreover, evaluation of performance and shape precision under the proposed algorithm are experimentally verified by deforming a real clay liver and/or cutting a gel block prepared at human body temperature. First, we capture the image of the liver of a patient by digital imaging and communication in medicine (DICOM) generated by magnetic resonance imaging (MRI) and/or computed tomography (CT) scanner. Then, the DICOM data is segmented and converted into four types of stereo-lithography (STL) polyhedra, which correspond to the whole liver and three blood vessels. Second, we easily overlap the virtual and real liver images in our mixed reality (MR) surgical navigation system using our initial position/orientation/shape adjustment system that uses color images to differentiate between real and virtual depth images. After overlapping, as long as the real liver is deformed and/or cut by a human (doctor), the liver is constantly captured by Kinect v2. Subsequently, by using the real depth image captured in real time, many vertices around the virtual polyhedral liver in STL format are pushed/pulled by viscoelastic elements called the Kelvin–Voigt materials located on the vertices. Finally, after determining the displacements of the vertices, we obtain an adequately shaped STL. The vertex position required for fixing the shape is calculated using the Runge–Kutta method.