Autonomous guidewire navigation in a two dimensional vascular phantom

Introduction

Catheter-based interventions, e.g., for the treatment of cerebro- and cardiovascular diseases, often require complex navigation of a catheter from the groin to the lesion through the vascular tree. Even highly trained specialists regularly struggle with the fact that catheter movements at the proximal end of the catheter translate into unexpected movements at its distal end. Automation of this task will unburden the mental workload for physicians and may improve the average treatment result.

Attempts to automate endovascular catheter navigation perform autonomous movements for a highly specialized task and specific anatomy [1], [2], [3], [4]. These use a H∞ loop-shaping controller for the bendable tip made of shape memory alloy and a robot manipulator for the translation to follow a given catheter tip trajectory, automatically orienting the bendable catheter tip towards the target vessel while the translation movement is performed manually, extracting the vessel centerlines and controlling a robotic bendable catheter to follow the desired centerlines or using magnetic motion capture sensors to control the speed of a robotically actuated catheter following a given trajectory. So far, no approach promises quick adaptability to individual patients’ vessel geometries, vessel characteristics and the various tasks to be performed.

Utilizing Neural Networks trained by Deep Reinforcement Learning as control algorithm for the catheter manipulation robot has the potential to generalise the methodology to perform the navigation in catheter-based interventions. In Reinforcement Learning an agent interacts with a environment by performing actions and receiving state observations and rewards for every such action. In Deep Reinforcement Learning a Neural Network is choosing the next action and the agent trains the Neural Network, such that the reward is maximised. The reward is given by the developer and should include all relevant optimisation factors, e.g., reaching a target, reducing catheter forces and reducing vessel wall contacts while navigating a catheter precisely. Solving numerous Atari games with a single Neural Network configuration shows the potential of this approach [5].

This paper extends own previous work, where initial navigation trials of an autonomous catheter guidance through a simple transparent acrylic glass vascular phantom is shown [6]. The simulation framework, testbench manipulator and camera-based guidewire tracking are reused. The vessel geometry of the phantom is modified to resemble natural vessel shapes and to allow investigation of learned navigation behaviors. The simulation scene is adapted to the new vessel geometry and the control algorithm is adjusted to solve the navigation task in the modified vascular phantom.

Materials and methods

The testbench setup is presented in Figure 1. The guidewire is actuated by a manipulator with two degrees of freedom (translation and rotation) through a rigid 2D phantom filled with a mixture of glycerin and water to model blood inside the vessels. The phantom is 3D printed using Stereolithographie. A camera mounted above and a light source mounted below the phantom emulate the fluoroscopy image usually obtained from X-ray. The catheter position is extracted from the camera image with 5 Hz in accordance with the control frequency of 5 Hz. The neural network calculates the commands for the guidewire manipulator. To enable the principle of human-oversight by the human-in-command approach [7] for future research, a gamepad provides the option to the user to manually override the manipulator commands.

Figure 1:

The test bench consists of a guidewire manipulator with two degrees of freedom, the transparent vascular phantom, the camera with light source, a gamepad for manual control and a computer running the camera tracking software and the neural network controller.

The geometry of the vascular phantom is illustrated in Figure 2a. Most branches can be reached by the shortest path, which would be the result of a breadth-first-search. However, the phantom is designed, such that a conventional guide wire cannot directly reach branch 3 by going directly through branch 1 due to its structural rigidity (Figure 2c). Instead, it is necessary to navigate through branch 8 and the loop between branch 1 and 8 (Figure 2d). Furthermore, a bifurcation is inserted which is mechanically difficult to navigate. When inserting the guidewire from branch 1 into branch 8 it needs to be translated and rotated at the same time to avoid kinking. A failed attempt with a kinked guidewire can be seen in Figure 2b.

Figure 2:

Vascular phantom and different guidewire positions.

The neural network is trained using Deep Deterministic Policy Gradients [8] with Hindsight Experience Replay [9]. The state of the guidewire is defined by five points in XZ-Coordinates evenly spaced with a distance of 5 mm along the guidewire starting from the tip. The input to the neural network is the catheter state of the current and the last three timesteps and the last three actions taken by the neural network. The reward system gives a −1 reward for every timestep where the preselected target is not reached and 0 for every timestep where the target is reached within a threshold of 5 mm. Output from the neural network are the continuous translation and rotation commands, which are sent to the guidewire manipulator.

Training of the neural network is performed in a simulation environment using the SOFA-Framework [10] with the BeamAdapter Plugin [11]. Phantom walls are assumed rigid and the lumen empty. Friction between wall and guidewire and guidewire stiffness have been iteratively tuned to mimic guidewire failure in the testbench. Main failure of the guidewire includes entanglement of the guidewires at bifurcations where increased bending occurs.

During the training in the simulation environment the control algorithm is evaluated every 500 training episodes by performing 100 test episodes, in which the guidewire has to be navigated from a random start point to a random target within the vessel geometry of the phantom. An episode is regarded successful, if the target is reached within 25 seconds. Subsequently, the trained neural network is transferred from the simulation environment to the testbench and tested within the real phantom. The non-successful episodes are analyzed by observing the navigation process on the testbench. To improve the success rate on the testbench, it is also evaluated providing the neural network with the coordinates of the next bifurcation along the path towards the target instead of simply setting the target.

Results

Figure 3 shows the success rate of the catheter navigation during the training process in the simulation environment. The success rate reached a maximum of 96% after 52,500 training episodes. Transferring the trained neural network to the testbench resulted in the same results as navigating in the simulation environment.

Figure 3:

The success rate of the catheter navigation over the amount of trained episodes during the training process in the simulation model.

The non-successful epsiodes can be split up in two groups of failures. The first group is the high-level control failure, where the navigation fails, because the controller tries to navigate a path to the target which is impossible to reach. The second group is the low-level control failure, where the navigation at a single branching point fails, because the controller is not able to maneuver the guidewire into the desired branch. The fully trained controller shows only high-level control failures. These occur either when the target cannot be reached by trying to navigate the shortest connection, e.g., navigation from branch 4 to branch 3, or when the target is in close proximity to the current branch, e.g., navigating from branch 3 to branch 2. Low-level control failures only happen with partially trained controllers and cease to occur when the controller is fully trained.

Providing the coordinates of the next bifurcation along the path towards the target as interim target to the controller improved the success rate to 100% for this phantom.

Discussion and conclusion

A neural network trained by Deep Deterministic Policy Gradients has been shown to learn navigating a guidewire within a complex and rigid vascular phantom in 2D. The low-level control of the guidewire states no problem for the neural network controller. The learned high-level control, especially during path planning where the direct path is not ideal, poses a challenge for the neural network. A possible reason for this is the fact, that most training episodes do not require advanced path planning. In most start/target constellations it is sufficient for the controller to navigate the shortest path. Learning more complex motion behavior is presumed to be slow due to seldom appearance during training. A method to improve training may be adjusting the start/target constellations to paths that are hard to reach for the control algorithm or providing Human Demonstration data for these constellations.

Alternatively the high level path planning can be performed by a separate algorithm, such that the target for the controller is the next bifurcation instead of the final target. This way, the neural network of the controller is not required to learn the high-level control.

Concluding it is shown that a controller based on a neural network trained by Deep Reinforcement Learning is able to navigate a guidewire through a complex two dimensional vascular phantom. The low-level control task of adapting to the mechanics of the guidewire are achieved effortlessly. The high-level control task of finding the correct path is difficult to learn if the target cannot be allocated to a branch clearly or the path is not straightforward.

Future work includes improving the neural network for high-level control and navigation through more complex vessel structures, e.g., 3D navigation, changing geometries, and combination of a guidewire with a catheter.