Surgical reporting for laparoscopic cholecystectomy based on phase annotation by a convolutional neural network (CNN) and the phenomenon of phase flickering: a proof of concept

As surgeons worldwide are legally obligated to provide prompt, detailed, thorough, and complete documentation of surgical procedures (surgical reporting) [1, 2], each approach facilitating an easy to create, yet comprehensive documentation is noteworthy.

Literature search revealed several studies showing that unstructured and handwritten operative notes are often poor in quality, but interventions like training with the intent to enhance structure, did result in significant improvement. [3, 4] By now, most surgeons make use of templates for documentation as this can verifiably improve quality. [5] In addition, structured operative notes with a predefined wording empower automated information extraction (IE) for economic and scientific applications apart from a bare medical significance. [6] An obvious approach to foster a reliable yet applicatory surgical reporting is the use of computer science.

Recent developments paved the way to stunning new applications. The wide field of Machine Learning (ML), especially the use of convolutional neural networks (CNN) has taken video, audio, and speech recognition a giant step further. [7] For instance, machine learning enables the extraction of adverse events from narrative surgical reports today [8]. The logical consequence is to construct applications that allow to create even comprehensive surgical documentation based on this technique. The most significant modality regarding ML in surgery is the image and video recognition as it facilitates the detection of phases and instruments. [9] EndoNet was one of the first CNNs enabling multiple recognition tasks in videos of laparoscopic cholecystectomies [10]. Its annotations were based on 8 phases (P0−P7) specified in the Cholec80 data set. The authors mentioned a possible application in real-time surgical scheduling and automatic indexing of video databases. However, the information gained from highly reliable phase annotation by CNNs can not only be used for real-time applications but also for the much less time-critical generation of operative notes. In 2020, Czempiel et al. introduced a CNN called TeCNO which incorporates the timely course of a laparoscopic cholecystectomy producing a smoother phase recognition. [11] As our research group was significantly involved in the development and training of this project contributing medical expertise and recordings of surgeries, we now apply the resulting CNN to a specific medical issue. Phase annotation thereby seems to be a proper approach to the challenge of structuring a surgical procedure. [12] Admittedly, this technique seems artificial or even arbitrary in some cases as different surgeons may adhere to a divergent order of surgical steps. Nevertheless, many critical events as ‘clipping an artery’ or ‘dissection of the gallbladder’ are milestones that must be completed to reach the intended surgical result, and allow a robust phase definition. Even classic handwritten surgical notes are mainly structured on the basis of these critical steps according to our experience in daily clinical practice. In this original research article we propose a first proof of concept allowing partly automated surgical reporting for laparoscopic cholecystectomy by a CNN output. We use phase annotations of 15 cholecystectomies delivered by TeCNO after a training with 52 surgeries recorded in our surgical department, and try to retrieve information from these datasets that could serve as a basis for a reliable and easy to apply reporting tool. Moreover, we explore, how patterns of incorrectly assigned frames (‘phase flickering’) in the CNN’s annotation can be used to obtain even more information, e.g. about adverse events for an automated report. To evaluate validity of annotations and the subsequent operative notes, we compare the TeCNO results with that of an experienced surgeon (HE) and an algorithm-based annotation by a human scientist (HA). Finally, we propose a workflow to generate feasible operative notes based on the data obtained by the use of a self-programmed reporting tool.

Based on the annotations achieved by TeCNO, we were able to provide a proof of concept for computer generated and manually completed surgical reporting in a preclinical scientific set up. The proposed workflow offers a standard surgical report to the surgeon and announces reliably, where possible adverse events or aberrance from standard course occurred. Thus, the surgeon only has to change the standard text if necessary. Otherwise, if a surgery is performed entirely in accordance with the standards, no further intervention by the surgeon is required. This principle of a documentation based on deviations from a standard course was already implemented in the approach of Just et al. by the use of automatically generated checklists which have to be completed by human interaction in real time. [19] In our case, during the surgery, no action by the surgeon or an observer is required. Moreover, we integrated an analysis of phase flickering and thus detected the occurrence of adverse events as bleeding and gallbladder perforation with a sensitivity of 67% for the number of FT groups and of 83% for the number of affected frames. The specificity for both parameters was estimated to be 1, suggesting that an adverse event is rather improbable if no flickering is detected. Regarding the annotation accordance of the CNN and the HE of 79.7%, there must be taken into account, that the original task was to assign one of the 8 phases to the video sequences. [11] As already mentioned, a human annotator will use another strategy than an CNN. He/she will first define the phase transitions and then will fill the frames between these timestamps with the annotation of the suitable phase, as assignment of each single frame to a certain phase would take many hours or even days for only one surgery. [20] Furthermore, the annotation of the CNN bases on an image recognition with respect to the temporal course. For this reason, in ambiguous constellations regarding time course and image content, the CNN may opt for a phase that does not make sense considering the original task but delivers additional information about intraoperative complications encoded in form of the above announced flickering. Therefore, the accordance of 79.7% cannot be seen as the only parameter to benchmark the performance of the CNN in that context. Although the assessment of ‘flickering’ at this time is only feasible to decide whether an adverse event or a rough aberrance from the standard procedure occurred or not, we estimate that the potential of a flickering analysis could go a lot further. With higher numbers of example videos in the inference group, statistical analysis of flickering patterns may be possible and could define certain types of complications or at least groups like ‘bleeding’, ‘perforation’, and ‘technical problems’ without having introduced them intentionally in the training process. In our proof of concept, we used the flickering solely to trigger an additional alert in the GUI’s header section.

A strength of our approach is the versatility of finally generated operative reports. While we exemplary proposed a narrative report in the last step of our workflow, even a more structured synoptic documentation is possible as the annotated phase lengths and flickering parameters can be the basis for each kind of final output, depending on users’ requirements regarding data extraction or detailed description of single steps of an operation. [21] Even hybrid reports uniting narrative and synoptic virtues are possible.

In our study, the laparoscopic cholecystectomy served as a highly standardized and approved procedure model for machine learning. [22] Of course a less structured surgery for example in an emergency setting may be much more challenging. But even if our approach becomes solely available for standard procedures as cholecystectomy or laparoscopic appendectomy in daily clinical use, it could reduce the documentation effort significantly. Certainly, our results confirm that it is mandatory to expand the basis for computer aided surgical documentation by further parameters, for example driven by vital sign sensors as well as other patient and procedure specific data, similar to the requirements of workflow recognition. [23] However, mere phase annotation seems not capable to reproduce the course of a surgery detailed enough to comply with the aim of retracing an entire procedure, as postulated in the introduction section. [1] A sage combination with other very promising approaches as real-time text recognition and keyword-augmented sequence prediction could even open up more complex and unpredictable procedures for automated reporting. [24]

While machine learning in terms of phase and instrument recognition proceeds at a breathtaking pace with accuracies of up to 92%, practical use cases in a daily clinical context are not yet fully uncovered. [9] The computer aided creation of surgical documentation is doubtless a promising application. Thus, next steps in development of an automated surgical reporting system are to analyze more surgeries in a prospective study design, and to identify more descriptors that contribute to a sustainable and rich in content documentation. Moreover, a common definition of what kind of information a surgical report should essentially contain is urgently needed, even to achieve international comparability and scientific assessment. Last but not least, the ideal design of final output is not thoroughly defined. Actually, a synoptic report seems to offer more advantages than a narrative text regarding information density and structure. [25] Nevertheless, we may not forget that a surgical documentation is primarily created to be read by human medical professionals with the intent to facilitate best patient care, and must not forfeit legibility as price for structure and machine interpretability.