Laparoscopic augmented reality registration for oncological resection site repair

The overall two-step registration procedure is summarised in Fig. 1: a primary registration process is completed before the resection is started, but after the intraoperative resection planning is complete. We propose that the primary registration be conducted before vessel clamping to reduce time pressure on this registration procedure. The focus for this registration lies on accuracy rather than speed. This can be conducted by any established means, as described in the literature.

In our case, the primary registration consisted of two steps: an initial alignment and a surface-based refinement step. For initial alignment, we used four anatomical landmarks. For surface-based refinement, we used the Iterative Closest Point (ICP) algorithm [5]. In the clinical application, this may be further refined by non-rigid deformation adaptation. However, this was outside of this work’s scope for reasons discussed further below. Other registration methods may also be used for this primary registration.

After the primary registration is complete, we assume that the virtual and physical kidneys are registered as accurately as possible. The surgeon then places four artificial markers around the planned incision path and, thereby, around the intended resection area. We propose using adhesive markers [29]. These markers’ positions are recorded with an optically tracked pointing tool. The recorded positions are stored in the virtual model for later re-registration. It should be noted that the recorded positions of the tracked pointing tool are situated slightly above the organ surface. This is because the adhesive markers are not thin slices but rather of unknown thickness. We store the position as it is recorded, i.e. slightly above the virtual organs’s surface. Due to any remaining registration errors from the primary registration, the recorded point may be located below the virtual model’s surface. In that case, we also store the point as it is recorded. After this step, the surgeons can proceed with the tumour resection, while the artificial markers remain in place.

When the resection is completed, the secondary registration is conducted by the surgeon. At this point, the system’s graphical user interface (GUI) displays the virtual model with the previously recorded points. Following these, the surgeon acquires the artificial markers with the tracked pointing tool. In the concept’s current implementation, the re-acquired points are used for a rigid re-registration. The aim of providing the artificial landmarks during the secondary registration is to increase the speed and accuracy of the landmark identification and, thereby, the point acquisition, compared to “naive” acquisition of anatomical landmarks.

We set up a simulated AR environment to test a prototypical implementation for the proposed registration procedure.

Regarding the first study objective, we compared registration performance between the initial alignment that was based on anatomical landmarks and the secondary registration that was based on the artificial adhesive fiducials, i.e. we compared the performance at two different stages of the same registration procedure. The independent variable in this aspect was the method applied at the respective stage of the two-step registration process.

We defined four points around each kidney pole that were used as simulated, virtual surgical targets. The first dependent variable was the registration accuracy for these four points, which was operationalised as the mean target registration error (TRE) for these targets. The second dependent variable was the task completion time (TCT) that was required for identifying and recording the landmark points/fiducial positions. Regarding the second study objective, we recorded the difference between the TRE after the completed primary registration and the TRE after the completed secondary registration.

Eighteen participants took part in our study. The participants were medical students in their fourth and fifth year of training. Participants’ age ranged from 21 to 27 years (median \(=\) 23.5 years). Twelve participants reported having between 0.5 and 14 h (median \(=\) 3 h) of previous experience with laparoscopy (either in clinical applications or in simulators or trainers). We administered some laparoscopic training tasks to mitigate the different levels of prior experience (see Study procedure). Participants were paid 20 EUR for participation.

The surgical site was simulated with a partially occluded phantom. For this phantom, we retrieved a computed tomography imaging dataset of a healthy, adult kidney from a public database [17]. We segmented the parenchymal surface using 3D Slicer [12] and exported a triangle mesh surface model. This model served as the virtual model to be displayed in our AR environment. We prepared a printed phantom of this model with the deposition modelling method. The resulting rigid phantom had a length of 112 mm from pole to pole (original scaling). The phantom was equipped with an adapter in order to spatially track it from outside the simulated surgical site (Fig. 4a).

The phantom was placed inside a simulated laparoscopic workspace. This workspace was created using a cardboard box that occluded the simulated surgical site. The site could be accessed with our tracked laparoscope and pointing tool through six holes in the box (Fig. 4b). The organ motion that would occur in real surgery was simulated by varying the holes through which the workspace was accessed. When the simulated surgical target was on the upper pole (to the participant’s left), holes one and three were used during the primary registration and holes two and four were used for the secondary registration. When the simulated target was on the lower pole (to the participant’s right), holes three and five were used during the primary registration and holes four and six were used for the secondary registration.

Approximately half of the phantom was covered with a cloth in each registration procedure. The cloth reached from one of the kidney poles to the phantom adapter (Fig. 4c). The registration was conducted on the non-covered half of the phantom. This aimed to simulate the fact that not the entire renal surface would be revealed during intraoperative dissection. Figure 4c also displays the simulated adhesive markers that were applied in our study. The resulting overall study set-up is shown in Fig. 5.

We collected participants’ demographic information before the main experiment. Participants were asked to complete two laparoscopic training tasks to practise the particular hand-eye coordination and spatial understanding that are required in laparoscopic interaction. We applied a self-built version of the “bean drop” and “checkerboard drill” tasks [23]. These two tasks require the targeted, coordinated motion of the laparoscope and a laparoscopic tool but are not more complex than necessary for our task (e.g. suturing or cutting tasks). Each task was performed once by every participant. The training performance was not measured or recorded.

Following this training, participants conducted the registration process for the first time with step-by-step instructions from the experimenter. They then conducted a second training trial without explicit instructions but with the opportunity to ask questions. After all questions had been answered, the experimenter exchanged the targeted kidney pole (by moving the cloth). Finally, participants performed the registration process in a test trial in which we recorded the required data. This concluded the experiment.

We conducted one-sided paired t tests for the TRE and TCT. The tests compared data for the initial landmark-based registration (prior to surface-based refinement) and the secondary registration with the alternative hypotheses:It is inherent in our concept that the TRE will systematically increase between the refined primary registration and the secondary registration because the latter builds on the former. We, therefore, did not perform significance tests for this difference but rather identified confidence intervals to provide an estimate for the magnitude of the accuracy loss during this step. Modified post hoc tests were conducted as reported in the Results section.

Generally, the surface-based registration step is conducted to refine the landmark-based initial alignment. It is generally expected to increase registration accuracy [24, 27]. However, data analysis showed that six of our participants produced a higher TRE during surface-based refinement than during the initial landmark-based registration. We believe that this registration error increase during what is intended as a registration refinement step is likely to be caused by a number of errors that are described in Discussion section below. These errors are unlikely to be encountered by experienced surgeons, i.e. the intended user population for systems like this. We, therefore, excluded these six participants for a post hoc exploratory and descriptive analysis of the TRE development. The resulting reduction is shown descriptively in Fig. 7.

Our results indicate that the two-step registration solution can improve registration speed and may be able to improve accuracy for laparoscopic AR applications in time-critical surgical phases. One limitation of our results is the unusually low overall accuracy of the surface-based primary registration. We see two potential reasons for this: firstly, we used pivot-calibrated standard laparoscopic graspers as an optically tracked tactile pointer to record the required point clouds. We chose this instrument because it is readily available in the operating room. However, it is somewhat flexible and bends easily under mechanical load. This considerably affects the tooltip tracking as that is based on a rigid pivot calibration. An interesting follow-up objective of our work may lie in the measurement and quantification of this deformation and its contribution to the overall registration error. To our knowledge, this has not been previously reported in the literature. Secondly, our participants had very limited experience with handling laparoscopic tools. We anecdotally observed that several participants accidentally recorded some points after the tooltip had slipped off the phantom surface. Participants also applied high pressure when tracing the instrument across the phantom surface, which increased the issue of instrument deformation. Moreover, it was difficult for some participants to keep the tooltip rather than the side of the tool on the surface because of the typically constrained tooltip motion. It seems unlikely that experienced laparoscopic surgeons would experience these specific difficulties.

Inaccuracies in the surface-based primary registration are passed on to the secondary registration. This is because the fiducial location in the virtual model is conducted after and, thereby, based on the primary registration. Thus, there is a discrepancy between the physical fiducial’s position on the kidney and the fiducial’s recorded position on the virtual model. This “fiducial storage error” is added to the fiducial localisation error that occurs during the secondary point acquisition. We were able to quantify the impact of the resulting accuracy loss (fiducial storage error plus fiducial localisation error) with a TRE growth of approximately 0.47–4.23 mm. We, therefore, believe that absolute accuracy could be considerably improved by modifying the means of surface acquisition.

While this article represents a successful proof-of-concept evaluation for our two-step registration method, it does not yet demonstrate clinical applicability or benefit. The obvious follow-up question for our results is whether the improvements that our method brings are sufficient to make AR support feasible during the time-critical phase of resection site repair. Specifically, three questions arise: firstly, is the added registration task with an estimated duration of 40 s during a time-critical phase justified by the clinical benefit? That is, can the resection site repair either be accelerated enough to compensate for the additional 40 s or does the additional information make the process more safe and effective? Future work should also examine whether this can be further accelerated by supporting the user in the fiducial acquisition. For example, the fiducials could be detected and highlighted in the video stream. Secondly, is the registration (with improved surface acquisition accuracy) sufficiently accurate to provide meaningful information about the position of risk structures? Finally, our participants’ experience does not reflect the skill level of the experienced surgeons that would use the system in a real application. The different levels of experience may influence users’ abilities to recognise landmarks/fiducials due to a better understanding of the surgical site and to record those landmarks/fiducials due to a higher skill level by using the laparoscopic tools. The third question is, therefore: Which accuracy levels would experienced surgeons achieve with this approach? These questions remain to be answered in future follow-up work.

Another important scope limitation is that we did not consider the effects of organ deformation during the tumour resection. Organ deformation in abdominal AR registration is a major limiting challenge and an active field of research [1, 21]. Promising concepts exist in the literature to mitigate this by applying biomechanical models to the virtual content and, thereby, simulating the physical organ’s deformation. One approach [20] informs a biomechanical model via fiducial marker locations and is, therefore, promising for our application and marker-based concept. However, to our knowledge, current biomechanical models [20, 30] assume that the kidney is deformed but structurally intact. In our application, however, the kidney is additionally deformed from its preoperative state by removing an unknown tissue volume. While some data have been published on the surface deformation caused by a single straight-line incision [1], a biomechanical model for our application would also have to consider the intrarenal structure deformation that is caused by the removal of a tissue sample. While this requires further research, a deformation study for the liver [16] has shown that intraoperative deformation is very limited on a local scale. Thus, within the area of the four fiducials and resection site, rigid registration may even prove to be sufficient.

The two conditions that we compared in our study were measured in a fixed order. This may have led to training effects between the two stages of the registration process. Specifically, participants were more familiar with the surgical object (in our case, the phantom) during the secondary registration than during the primary registration. A part of the fiducial acquisition acceleration may be attributed to this fact. However, this prior familiarisation with the surgical site is realistic and, therefore, does not affect the validity of our results.

Overall, registration accuracy in a clinical setting may be higher due to better surface acquisition methods, or it may be lower due to organ deformation. Thus, the absolute TRE values from our study are of limited external validity. However, the effects we found indicate that our concept may be a viable approach for AR support during the resection site repair phase of LPN/RPN.

It should be noted that this article presents a registration concept for the resection bed but does not address the issue of visualising relevant anatomical information. This poses a separate challenge because the exact resected volume is unknown at this point during the surgery. Further work is required to address this, but one potential approach may be visualisations that are based on instrument locations (e.g. [25]) rather than the permanent bulk display of anatomical structures (e.g. [7]).

This article discusses the proposed registration method in the context of resection wound repair in LPN/RPN. Further research is required to assess its suitability in other laparoscopic oncological resections. Moreover, the general two-step concept may be suitable for even more image-guided surgery applications in which the registration process is conducted under time pressure and in which the opportunities for intraoperative imaging or preoperative fiducial placing are limited.