Automatic matching of surgeries to predict surgeons’ next actions

https://doi.org/10.1016/j.artmed.2017.03.007Get rights and content

Highlights

  • A method for automatic matching of surgeries to predict surgeons’ next actions is proposed.

  • The method is based on the optimal registration of a partial sequence to complete reference sequences.

  • The method is evaluated on neurosurgical procedures.

Abstract

Objective

More than half a million surgeries are performed every day worldwide, which makes surgery one of the most important component of global health care. In this context, the objective of this paper is to introduce a new method for the prediction of the possible next task that a surgeon is going to perform during surgery.

Material and Method

We formulate the problem as finding the optimal registration of a partial sequence to a complete reference sequence of surgical activities. We propose an efficient algorithm to find the optimal partial alignment and a prediction system using maximum a posteriori probability estimation and filtering. We also introduce a weighting scheme allowing to improve the predictions by taking into account the relative similarity between the current surgery and a set of pre-recorded surgeries.

Results

Our method is evaluated on two types of neurosurgical procedures: lumbar disc herniation removal and anterior cervical discectomy. Results show that our method outperformed the state of the art by predicting the next task that the surgeon will perform with 95% accuracy.

Conclusions

This work shows that, even from the low-level description of surgeries and without other sources of information, it is often possible to predict the next surgical task when the conditions are consistent with the previously recorded surgeries. We also showed that our method is able to assess when there is actually a large divergence between the predictions and decide that it is not reasonable to make a prediction.

Introduction

In the USA alone, 1000 new surgeries will have started within the next 10 min. This highlights how central surgeries have become for global health care. To support and assist surgical teams, Operating Rooms (ORs) have undergone tremendous changes. One of the targeted goals is the development of context-aware systems [1] that continuously monitor the activities performed in the ORs in order to provide an accurate and reliable support. The key challenge in developing these new methods is to process the data coming from sensors and real-time detection systems, in order to provide useful information and support decision making. This is extremely challenging because OR environments are very diverse, surgical interventions are very variable with specific patients, and different surgeons might have different levels of expertise. The richness and complexity of the data that is collected calls for new artificial intelligence methods [2] to support pre-, peri- and post-surgery (before, during and after). In this context, predictive data mining techniques [3] have long proven to be extremely relevant.

The field of Surgical Process Modeling (SPM) [4] targets the development of new methods that leverage from OR activities monitoring. In this field, several methods have already been proposed to automatically detect surgical activities. These methods rely either on manual annotations by an observer [5], [6] or on sensors present in the OR (e.g., camera) [7], [8]. For example, the task performed by a surgeon can be automatically inferred by combining RFID chips on instruments (for identification) with accelerometers [9].

With the richness of the data comes the difficulty of analysing it, because of its complexity. For example, two surgeons performing the same surgery on the same patient might exhibit a very different course of specific actions, while being surgically very similar: they might use the same technique, have the same patient outcome, etc. However, from the low-level point of view (the sequence of low-level tasks like cut, suture, etc.), these surgeries will look very different from each other.

Extracting useful high-level knowledge from this low-level data has been one of the research themes targeted by the field of SPM [4], [10]: the objective is to understand surgeries to improve the quality of care. The above-mentioned sensors capture the surgical tasks performed in real-time, which opens the door to using artificial intelligence methods to provide real-time information to the surgical team.

This paper tackles the prediction of possible surgeons’ subsequent actions, using low-level information alone. Predicting surgeons’ possible next actions is critical for OR management: it can be used to provide useful real-time information to the surgical team (e.g., nurses, anesthetist, junior surgeon), while allowing the surgeon to focus on more demanding tasks. For example, the nurses will be able to prepare the tool that is going to be used next, thus ensuring a smooth transition between the activities of the surgeon. Because predicting the next surgical task is central, such a prediction system will also be a keystone to the development of many other systems. For example, while the relative importance of the different factors that cause surgical error is unknown [11], technical skills acquisition are shown to correlate with a reduction of patient complications [12]. Thus, performing the right action at the right moment in surgery can greatly influence patient outcome. A study on patterns of technical error among surgical malpractice [13] highlighted that most technical errors occur in routine operations with experienced surgeons. One of the recommendation of the study is to focus surgical safety research on improving decision-making and performance in routine operations. This is why working on systems helping the surgeon to take action-oriented decisions is critical in the OR.

The data captured in the OR have a specific granularity level. A granularity level is defined as the level of abstraction at which the surgical procedure is described. MacKenzie et al. [14] were the first to propose a model of the surgical procedure that consists of different levels of granularity: the procedure, the step, the substep, the task, the subtask and the motion. Later, Lalys and Jannin [4] introduced a terminology consisting of phases defined as the major types of events occurring during surgery. Each phase is composed of several steps. A step is considered to be a sequence of activities used to achieve a surgical objective. The data used in this paper captures the activity of both hands for three different elements: used instrument, performed action and targeted anatomical structure [15]. Learning to predict the next activity of the surgeon from such low-level information is extremely challenging, because the next surgical action depends upon high-level information (such as phase of the surgery, technique used, patient-specific information, so-far reaction of the patient to the surgery, etc.), while a surgery is represented by a series of actions like “cut the skin with a scalpel”.

Intuitively, our approach matches the on-going surgery to every surgery of a reference set of surgeries, and uses the next actions that have been performed in the reference set of surgeries to draw a prediction about the next action that will be performed in the current surgery. Our proposed approach includes the three following features:

  • 1

    Optimal registration of a partial surgery: We propose a new method to optimally register the on-going surgery (partial surgery) to any complete pre-recorded surgery. Our approach is based on the Dynamic Time Warping similarity measure [16], which is consistent with surgical processes [5].

  • 2

    Voting for high-confidence prediction: Using the optimally registered reference set of surgeries, we use voting to draw a high-confidence prediction about the next action that is going to be performed by the surgeon.

  • 3

    Detecting when to predict with high-confidence: Using the agreement rate among multiple predictors, we are able to detect when to perform a prediction and when it is not possible to draw an accurate prediction.

  • 4

    Weighting the prediction according to sequence similarity: Using the relative similarity between the on-going surgery and the set of pre-recorded surgery as weights, we are able to improve the prediction accuracy by giving more importance to similar surgical behaviors.

Our framework was assessed using two clinical datasets of Lumbar Disc Herniation surgeries (LDH) and Anterior Cervical Disectomy surgeries (ACD). The first dataset contains 24 LDH surgeries performed by multiple surgeons and was recorded at the Neurosurgery Department of a first site, named site A. The second dataset contains 18 ACD surgeries and was recorded at the Neurosurgery Department of a second site, named site B. We show that our method outperformed the state of the art on both datasets by providing a prediction with a 95% accuracy more than 85% of the times.

This article is an extended version of the article that was presented at the 15th Conference on Artificial Intelligence in Medicine in Europe [17]. In this extended version, we have improved the time complexity of the main algorithm from Θ(l · k) to O(l · k) and improved our methodology with a new weighting technique for the predictions. We have also extended the validation of our work, by studying and comparing our method on a second clinical dataset. This paper is organized as follows. Section 2 introduces the related work in both surgical process modeling and prediction systems in health care and surgery. In Section 3, we present our method for high-confidence prediction of the next surgical activity that is going to be performed. In Section 4, we present experiments conducted to demonstrate the quality and performance of our approach compared to the state of the art. Finally, we conclude this work and describe future research in Section 5.

Section snippets

Related work

In this section, we briefly introduce the existing methods to record and recognize surgical activities (Section 2.1), initial required step in our application. We then present some examples of prediction systems in health care and for surgery application (Section 2.2).

High-confidence prediction of the next surgical activity

We detail our approach in this section. We start by presenting our method for optimal sub-sequence matching in Section 3.1. In Section 3.2, we then show how to construct a discriminative model of the to-be-performed surgical actions; we also detail how to decide upon the situations in which we believe that uncertainty is too high to draw a high-confidence prediction.

Clinical data

We evaluated our framework on two datasets composed of two types of surgical procedures: Lumbar Disc Herniation (LDH) surgery and Anterior Cervical Discectomy (ACD) surgery. Fig. 3 presents an extract of the LDH surgeries and Fig. 4 an extract of the ACD surgeries. The legends illustrate the most common actions in the respective dataset. The white spaces correspond to times when the surgeon was not performing any action.

The LDH dataset is composed of 24 lumbar disc herniation surgeries recorded

Conclusion

This work shows that it is possible to predict the next surgical task accurately. Our predictions are drawn from a low-level description of surgeries, without other source of information, and assume that the current surgery is consistent with the ones constituting the training set. Our contributions include (1) a definition of the objective function for the registration of a partial sequence to a complete reference sequence, (2) an efficient algorithm, based on DTW, to optimally minimize the

Supplementary materials

Acknowledgments

This work was supported by the Australian Research Council under award DE170100037. This material is based upon work supported by the Air Force Office of Scientific Research, Asian Office of Aerospace Research and Development (AOARD) under award number FA2386-16-1-4023.

References (40)

  • N. Bricon-Souf et al.

    Context awareness for medical applications

    Med Appl Artif Intell

    (2013)
  • F. Lalys et al.

    Surgical process modelling: a review

    Int J Comput Assist Radiol Surg

    (2013)
  • F. Lalys et al.

    Automatic phases recognition in pituitary surgeries by microscope images classification

    Information processing in computer-assisted interventions; vol. 6135 of LNCS

    (2010)
  • C. Meissner et al.

    Sensor-based surgical activity recognition in unconstrained environments

    Minim Invasive Ther Allied Technol

    (2014)
  • B.J. Dlouhy et al.

    Surgical skill and complication rates after bariatric surgery

    N Engl J Med

    (2014)
  • S.E. Regenbogen et al.

    Patterns of technical error among surgical malpractice claims: an analysis of strategies to prevent injury to surgical patients

    Ann Surg

    (2007)
  • L. MacKenzie et al.

    Hierarchical decomposition of laparoscopic surgery: a human factors approach to investigating the operating room environment

    Minim Invasive Ther Allied Technol

    (2001)
  • N. Mehta et al.

    Sequence and task analysis of instrument use in common laparoscopic procedures

    Surg Endosc

    (2002)
  • H. Sakoe et al.

    Dynamic programming algorithm optimization for spoken word recognition

    IEEE Trans Acoust Speech Signal Process

    (1978)
  • G. Forestier et al.

    Optimal sub-sequence matching for the automatic prediction of surgical tasks

    AIME 15th conference on artificial intelligence in medicine, vol. 9105

    (2015)
  • Cited by (0)

    View full text