Catheter manipulation analysis for objective performance and technical skills assessment in transcatheter aortic valve implantation

Minimally invasive (MI) endovascular interventions are nowadays the favoured option for treating a number of vascular diseases, particularly in high-risk patients with comorbidities that are deemed unsuitable for open heart surgery [7]. Transcatheter aortic valve implantation (TAVI) is a MI procedure for replacing a malfunctioning stenosed aortic valve, where a catheter is percutaneously inserted in the vasculature and navigated through the aorta in order to deploy the artificial valve. TAVI operations offer the advantage of shorter recovery times for patients, since the operational trauma is minimal. However, TAVI is a complex procedure that requires surgeons to navigate steerable instruments in the fragile, often highly sclerotic vascular system, under friction and the presence of calcium deposits. This, combined with the restricted visualisation of the operating environment, raises the risk of embolisation and tissue damage (e.g. vessel rupture) due to intra-operative errors. Modern endovascular surgeons must possess a high level of manual dexterous ability as well as be able to operate under significant physical and mental load.

Performance evaluation is a fundamental element of outcome-driven medicine and healthcare delivery, yet objective methods and metrics for developing training programmes and evaluating surgical performance and workflow are not fully developed or in widespread use. Traditionally, surgical expertise is evaluated with written and oral examinations, procedural and case logs, as well as expert monitoring during exercises and studies on cadavers, animals, phantom models or virtual simulators [16]. A more objective approach is the use of standardised, structured grading scales (GRS) or checklists which again require expert observation and manual scoring of the trainee either in real time or through reviewing video recordings. Although subjective to some extent, these methods are the current gold standard for performance evaluation. The problem in their application arises due to inefficiency issues since they are laborious and time-consuming. In addition, supervised assessment is inherently subjective to a degree and introduces bias which makes difficult the global standardisation of an acquired level of technical expertise [9].

In recent years, research efforts have focused on developing methods for evaluating surgical skills that are more objective and can complement traditional expert assessment [10, 20, 23]. Empirical evidence suggests that the level of technical expertise is represented by the proper handling of surgical equipment in such a way that is efficient and minimises the risk of errors occurring [3, 15]. In the endovascular domain, the previous work attempted to objectively characterise the skill level of surgeons through analysing the kinematics of the catheter and other surgical tools and extracting features and metrics. Rolls et. al. utilised software to track the position of the catheter’s tip in experiments with the Procedicus VIST simulator. In results from 21 participants of varying experience, the total path length correlated well with manually scored GRS [21]. Additional metrics from catheter tip tracking data that have been identified as potential indicators of endovascular skills are non-dimensional jerk, spectral arc length, number of sub-movements and average sub-movement duration [4, 5]. Other type of information that has been investigated is the force and torque loads exerted on the catheter by endovascular operators. From experiments with a robotic system, it was deduced that the force/torque loads that were exerted on the catheter by experienced and inexperienced subjects were different. Five features median speed, the mean value of displacement, push force, torque and the number of twists demonstrated correlation with surgical skills [12]. Motion and contact force information were then used for training hidden Markov models (HMM) for classifying cannulation tasks executed from surgeons of varying experience [13]. Recent work on modelling and tracking the entire shape of catheters in fluoroscopic video sequences will lead to a plethora of new kinematic information that may prove extremely useful for discriminating surgical performance [2].

A potential solution for the intricate challenges in TAVI could be the use of robotically controlled surgical systems. Interest in the design and application of robotics for endovascular operations is gradually increasing, with the previous work evaluating potential performance benefits from the use of robotic systems [11]. Advantages include shorter training periods, decreased cognitive load, enhanced precision and catheter manoeuvrability which translates to increased safety and lower risk of vascular damage, compared to conventional catheters [17, 18]. Subsequently, there has been growing interest in the clinical evaluation of robotic systems for TAVI operations [14, 19].

Virtually all established endovascular GRS and checklists contain criteria, albeit their description might differ slightly that evaluate three key technical attributes. These can be summarised as: (1) avoiding vessel wall damage with precise catheter manipulation, especially near delicate or sclerotic areas; (2) smooth catheter navigation in a timely fashion without unnecessary movements; (3) efficient use of the imaging modalities and contrast agents. Focusing on (1) and (2), we hypothesise that potential quantitative measures extracted from analysing medical images and tracking the instruments trajectory and combined appropriately will contain the necessary information to discriminate levels of surgical experience.

Figure 1 illustrates sequential tracks of the tool tip during the cannulation of the aortic arch, from a novice (Fig. 1a) and an expert surgeon (Fig. 1b). We observe that the expert surgeon manipulates the tool in a more efficient, economical way than the novice, resulting in smoother trajectory. We use this information to calculate a number of kinematic features (e.g. speed, acceleration, jerk) for our investigation. Furthermore, we introduce the distance of the tip to the vessel wall as a metric to evaluate the safe navigation of the catheter with respect to adjacent vascular tissue. Statistical analysis results are indicative of the experts developed dexterity and technical competence as represented by lower median values of procedure time and dimensionless jerk, and higher average speed and acceleration than novices. Ultimately, we employed the features that demonstrated statistical significance as inputs in two clustering algorithms (k-means, expectation-maximisation). Our objective was to categorise the participants according to their experience level and the different types of surgical equipment in executions from the same group. The obtained level of accuracy was 83–91 %, for experience group classification and 75–91 % for equipment classification.

The 2D location, in pixel coordinates, of the catheter/guidewire tip in the fluoroscopic image was extracted, on a frame-by-frame basis, using a semi-automated tip tracking algorithm. The semi-automatic software allows frame-by-frame video seeking with user interaction at each frame allowing the definition of points and contours over the video. The software embeds tracking of points using the Lucas–Kanade algorithm or by brute force, normalised cross-correlation matching [21, 22]. User interaction is required to bootstrap the tracking at points of failure or drift because of fast catheter motions, blur or poor contrast. The output of the tracking algorithm is the catheter tip motion. In the video recordings, the temporal limits of the two TAVI stages were manually annotated and the total procedure time for each stage was extracted. The fluoroscopy screen was recorded during the experiments, and the video sequences were analysed. A pre-processing step was required because the fluoroscopy monitor was captured with an external camera from an unfixed position thus contained some degree of screen view bias. In all recordings, the entire fluoroscopy screen was visible which allowed us to shear the screens from different videos and using affine transformations, warp them to the same size (\(720\times 576\)). This step rectified the perspective distortion and normalised the obtained trajectory coordinates from each recording.

In order to evaluate the safe navigation of the catheter in a quantitative way, we elected to investigate the tip’s trajectory with respect to its distance from the vessel wall. The first step was to segment the shape and geometry of the vasculature in the fluoroscopic image. This was done manually to avoid segmentation inaccuracies, by isolating the portion of the image that contains the aorta model and transforming it into a greyscale image. We then extracted the shape of the phantom by converting this image into a binary one using a cut-off threshold value. Pixels with greyscale value above the threshold were assigned value “1” while the ones below, the value “0”. In the obtained binary image, the aorta model is delimitated by the white pixels. Finally, we calculate the distance of each white (“1”) pixel that corresponds to a track of the catheter’s trajectory to the nearest black (“0”) pixel, which in essence defines the edge of the vessel wall. This allows us to calculate features like the average distance of the tip’s locations to the nearest point of the vessel wall. Our main hypothesis is that experienced interventionists should be able to maintain a relatively large distance from the vessel wall, thus minimising the danger for potential tissue damage. To augment our dataset for this analysis, the shape of the catheter was manually annotated, through crowdsourcing in a subset of frames from each recorded sequences. Annotation was carried out in Amazon Mechanical Turk with an HTML-/JavaScript-based contour annotation tool tuned for catheter-like shapes [1]. The catheter-shape annotations were normalised in a similar way as the tracked coordinates and then interpolated using spline interpolation so that the catheter was annotated with the same number of points in every frame. Figure 3 shows two examples of catheter-shape annotation.

Figure 4 illustrates the three processing steps we followed (catheter/guidewire tip tracking, model segmentation, distance calculation) in representative experimental executions from an expert (rows 1 and 3) and a novice participant (rows 2 and 4) with both the conventional catheter (rows 1 and 2) and the robotic system (rows 3 and 4). The higher efficiency of the expert surgeon, in terms of time and movement, is again evident by the trajectory of the tip particularly during the first stage. It is also clear that the two operators required more steps to complete the procedure when using the robotic catheter.

From the manual annotation of the two stages of the experiment, the total procedure time was extracted for each one. Motion-based metrics were calculated from the tracks of the catheter’s path. During the experiments, we noticed that occasionally the instrument would remain stationary for a period of time. We therefore elected to calculate the average speed (calculated as \(v_\mathrm{d} = \hbox {PL}/T_\mathrm{p}\), \(T_\mathrm{p}\): Procedure time, PL: total path length) as an indication of movement efficiency as the path length alone would not be indicative of these pauses. Additional features that were calculated include the average magnitude of the acceleration vector (calculated as the 2-norm of the acceleration components in the two axes: \(a_\mathrm{d}=\sqrt{a_x^2+a_y^2}\)) and the dimensionless jerk. Dimensionless jerk is a measure used to describe smoothness in the shape of a movement and is defined in such a way to be independent of duration and amplitude [6]. Considering 2D motion it is given as:The catheter’s trajectory coordinates were also used to calculate the average distance (measured in pixels) to the nearest vessel wall point from the derived distance images.

This paper presented an investigation on deriving and validating objective measures for endovascular surgical skills assessment, based on analysing the motion and manipulation patterns of surgical tools. A cohort of 12 participants separated into two groups (novices, experts) performed two stages of a TAVI operation on a phantom silicon model, with conventional surgical tools and the Magellan endovascular robotic platform. Video sequences of the fluoroscopy screen were recorded and used to track the 2D position of the catheter/guidewire tip as well as to segment the aorta model. The motion-based metrics that were investigated are: total procedure time, average speed, average acceleration magnitude, dimensionless jerk and average distance (of the catheter’s tip and shape) to the vessel wall, a metric we believe is indicative of safer catheter navigation. As expected, the two groups demonstrated different procedure times especially in the arch navigation stage (\(p=0.008\)), with experts completing the tasks faster than novices. Moreover, expert surgeons exhibited higher median values in average speed and acceleration in both stages. Movement smoothness was evaluated with the dimensionless jerk, a metric designed to be independent of duration and amplitude, in which experts, as expected, demonstrated smaller median values with the difference being statistically important (\(p=0.008\)) in stage 1. Experiments with the Magellan platform resulted in higher value for procedure times and dimensionless jerk and lower speed and acceleration values. This can be attributed to the robot motion scaling for fine manipulation and potentially to the participants’ lack of experience in operating the robotic system. The discrimination potential of the derived motion-based metrics was evaluated with unsupervised clustering experiments. Two established algorithms were employed (k-means, EM) and achieved 83 % (k-mean)–91 % (EM) accuracy in classifying the participants to their respective experience group and 75 % (k-means), 91 % (EM) in discriminating among the two types of equipment used. The obtained high level of accuracy validates our basic hypothesis that motion-based analysis can indeed provide skill-representative metrics. Inspired by the criteria evaluated in traditional surgical assessment methods (GRS, checklists), this work introduced the distance to the vessel wall as a metric to evaluate safe catheter navigation. Although the average distance to the vessel wall did not demonstrate statistical significance among the two groups, it was found to be higher during stage 2 which involves the navigation of the catheter through the aortic valve. Based on this, the robotic system appears to facilitate safer catheter manipulation.

Future work will focus on the kinematic analysis of the entire catheter, instead of just the tip, as well as on modelling surgical execution using the kinematic features in order to assess performance more thoroughly besides producing a classification result. In addition, we target at fully automating the catheter tracking algorithm and phantom segmentation, currently semi-automated, so that minimum user interaction is required.