Interactive visual exploration of surgical process data

Surgical procedures of the same type usually follow a certain workflow, consisting of different events with a certain temporal extent. To enable consistent naming of such events, corresponding terminology is commonly defined. This terminology may describe the process on different levels of granularity, e.g., it is typical to refer to high-level events during surgery as phases [5]. Individual phase lengths and phase occurrence patterns vary between surgeries and are, therefore, subject to further analysis. Additional terminology has developed for more low-level events than phases, like steps and actions [6, 7], introducing hierarchical relations between the events. Neumuth et al. refer to the visualization of surgical workflows as one of multiple levels of abstraction of surgical procedures [8] and proposed a timeline-based and logic-oriented visualization of such event-based information [4]. Other techniques to represent surgical workflow models include hidden Markov model graphs and line charts [9, 10]. Additionally, time-oriented data like phase annotations are often represented as sequences of bars [8, 10]. However, these techniques do not scale well if larger numbers of procedures or events should be displayed. Nor do they provide solutions for performing combined analysis of data coming from different sources, like phase and instrument annotations, or device measurements. We aim to allow such investigations in our work. In the notion of Lalys and Jannin [5], we facilitate “Analysis” in the form of “Display” to “Explore qualitatively and illustrate results,” and support the “Aggregation - Registration” of surgical procedure data.

The formalization of surgical procedures as sequential lists of events is referred to as surgical process modeling [5]. Neumuth et al. distinguish between individual surgical process models describing a single procedure and generic surgical process models [11]. The latter are referred to as “mean” interventional models computed from individual process models. Such a generic model can be useful in statistical assessment, visualization, and quantification of deviation from the mean procedure. However, the work does not provide any visual tools in this direction.

Focusing more on automated analysis, techniques from sequence analysis can be used to compare and evaluate surgical procedures [12]. In general, visualizations do not play a central role in surgical process mining, yet. Accordingly, Rojas et al. call for a better “visualization of the process models and the results obtained” [13].

There exist datasets related to the scenario from Section 1, covering information like event annotations, instrument presence, or device measurements. (Below, we write the different event types italicized.)

The dataset SARAS consists of four procedure recordings with approximately 27 thousand frames in total. In them, 21 different action classes were annotated [14]. In contrast, MISAW comprises video data, kinematic data, and workflow annotations [6]. The annotations consist of hierarchically ordered events: phases > steps > activities. In total, 27 sequences were recorded from laparoscopic surgeries. Another dataset, provided by Charriére et al., contains 50 videos of cataract surgeries [15], where each frame is annotated with one out of 19 different surgical steps. Moreover, Wagner et al. collected 33 videos from laparoscopic surgeries [7], including annotations of seven phases, four surgical actions, presence of 21 instruments, and surgical skill classifications.

We used the following two datasets. HeiCo comprises 30 videos from three types of laparoscopic surgeries, namely proctocolectomy surgeries as well as rectal and sigmoid resections [10]. It also includes annotations of the surgical phases, the presence of instruments, and measurements from medical devices. The phases, instruments, and devices are the same across all three surgery types. In addition, we used Cholec80, which contains 80 videos of laparoscopic surgeries, including phase and instrument annotations [16].

We selected HeiCo and Cholec80 as they are open source and cover a variety of data sources and types. Hence, our solution can be applied to other datasets of the same variety. Moreover, Cholec80 with its 80 procedures allows us to test whether our solution can also scale to larger datasets. Note that we excluded hierarchical data from our investigations to narrow the scope.

To introduce the goals, we use the following terminology (see Fig. 1 for examples).

We identified the following three main analysis goals:

Synchronize information (\(G_S\)). The first goal is to synchronize the information from different procedures and from different data categories. For this, the time-dependent information from the different data entities needs to be temporally aligned within each procedure. It also requires options to align the information from different procedures. This presents a particular challenge since procedures can differ in their execution, raising the question of which point in one procedure corresponds to which point in another procedure.

Compare information (\(G_C\)). The second goal covers the comparison between different procedures for the same data entity, but also the comparison across data entities. In the first case, multiple approaches are desirable: New insights can be gained by comparing procedures one vs. one, one vs. many, and many vs. many. To facilitate such comparisons, means of aggregation, filtering, and selection need to be provided. Moreover, for comparisons across procedures or data entities, the data needs to be synchronized first (\(G_S\)).

Understand the workflow (\(G_W\)). The third goal is to better understand the surgical workflows in practice. This should be possible at the level of individual procedures as well as groups of procedures. An improved understanding of the workflows can help experts to form new hypotheses from the data, e.g., by learning how the average procedures are executed and comparing them to exceptional ones, or understanding the usage patterns of the surgical instruments. Such investigations rely on means for synchronization (\(G_S\)), aggregation, and comparison (\(G_C\)).

The design contains a reference view at the very top (see Fig. 2) to allow the users to refer to the individual procedures at any time.

Procedure reference view. Each procedure is represented by a row of bars (see Fig. 3). The length of each bar represents a time-independent value for the corresponding procedure. Here, we only include information calculated across each procedure, but other information could be presented in this view, too, like surgical skill classifications [7] or the similarity between procedures [12].

We propose one column in the center consisting of one main view for each of the time-dependent data categories (see Fig. 2). For each category, we provide a view with the temporal axis in the horizontal direction and the different data entities in the vertical direction. The main views are positioned such that their temporal axes are aligned to allow a synchronized inspection (\(G_S\)) and comparison (\(G_C\)) of the data categories and entities. On each of the temporal axes, the information from the individual procedures is aligned at the alignment point, which is represented as a vertical line and defaults to the beginning of the first phase (see Fig. 4). We now present the views we propose for the different data categories.

Phases main view. The topmost view is used for the data category that divides the procedures into the most coarse events, i.e., the phase annotations in case of HeiCo [10]. The phase annotation data is representative of data categories whose data entities are conceptually ordered while the entities contain binary information for each procedure. For each entity, i.e., phase, we map the information of how many procedures were in the given phase throughout the surgery to a horizontally connected set of areas (see Fig. 4, left). The higher the area, the more procedures were in the phase at that time. Thus, the information from different procedures is aggregated and can be compared (\(G_C\)).

To represent the conceptual order of the phases, we color the areas based on a sequential colorblind-safe scale. The “NOOP” phase, which represents an exceptional phase that does not normally occur during the procedure, was given a bright red color. If two subsets of procedures should be compared, the areas corresponding to one set are kept colored and the fill color of the other set of areas is changed to gray (e.g., see Fig. 5). More details on how such subsets can be selected are given in “Interaction Techniques” Additionally, the abbreviated phase name labels on the left are fully revealed when hovering over them.

Instruments main view. The second view is used for the instrument annotations. The data entities for the instruments do not have an inherent order but, again, contain binary information for each procedure. Hence, the view is constructed in the same way as the phases view, except that only a single main fill color is used for the areas (see Fig. 5).

The first form refers to each procedure individually. Device values are often measured at a higher frequency (e.g., 25 per second) than there are pixels in a single row of a common computer screen. Hence, multiple measured values need to be represented in the width of a single pixel. We do so by extracting the minimum (min), maximum (max), and mean value of all values corresponding to a single pixel. This way, we can depict a single procedure through the development of its min, max, and mean values over time. We map the mean value to a dark blue line and draw the line over a bright blue area of which the upper edge represents the max values and the lower edge the min values (compare Fig. 6). We selected the min/max solution over more precise representations of the distribution since it is the least complex to understand.

The second form of aggregation is performed across all procedures, not only a single one, to represent their average developments over time. To do so, we calculate the min, max, and mean value for all frames corresponding to a single pixel across all procedures. The result can be represented like for a single procedure (see Fig. 6).

To improve the readability, individual device rows can be expanded for a closer inspection (see Fig. 7). Moreover, if two subsets of procedures should be compared, the visual representations of one of the sets are kept colored while the others are colored in gray.

In addition to the main views, we propose a column of side views containing one view for each data category (see Fig. 2) to represent the time-independent information for each of the data entities. One goal was to facilitate the identification of procedures of interest, e.g., those which stayed particularly long in a certain phase, did not use a certain instrument, or recorded extreme values for a certain device (\(G_C\)). Additionally, the views should support the analysis of the relations between different data entities (\(G_W\)). Accordingly, we selected a set of histograms showing summarized information, but other data could be displayed there, too. Each histogram shows how the procedures are distributed regarding a certain value derived for the corresponding data entity. For our selected datasets, we used the following derived information.

Phases side view. The histograms depict how much time the different procedures have spent in each phase, and how often the procedures stepped into each of the phases (see Fig. 4, right).

Instruments side view. The histograms show for how long each instrument was used in the different procedures, and how often the instruments were changed in them (see Fig. 5, right).

Devices side view. We include one histogram for the devices, depicting the average value the procedures had across the surgery (see Fig. 6, right).

We included various interaction techniques to adjust the views. Most of them can be reached via the interaction toolbar (see Fig. 2).

Alignment. The users can adjust the alignment point at which the time-dependent information should be aligned. The point can be set to either the first or the last occurrence of any of the phases. Accordingly, all procedure recordings in the main views are shifted along the time axis to be aligned at the point where the corresponding phase first/last occurred in them (e.g., see Fig. 5). This way, the information of all data entities from all data categories can be synchronized (\(G_S\)) and compared (\(G_C\)).

Scale The x-scales of the histograms from the side views can be set to one of two modes. The scales can either be fitted to the minimum and maximum value of each individual histogram (see Fig. 5) or to the minimum and maximum value across all histograms from the same column (see Fig. 4). In the first case, it is easier to compare the information within a single histogram, and in the second case, it is easier to compare the information from different data entities to investigate, e.g., which phases took longer overall (\(G_C\)).

Zoom The users can zoom in and out of the time-dependent main views. The zooming is synchronized between the views (\(G_S\)). This is particularly useful for high-frequent data like device measurements (see Fig. 7).

Highlight Highlighting allows users to select subsets of the procedures to compare them against the remaining procedures. To do so, the users can draw yellow highlight constraint boxes in any of the reference, main, or side views (e.g., see Fig. 5). Subsequently, the visual representations of all procedures whose values lie in the given box are kept colored while the visual representations of those which do not fulfill the constraint are colored gray. This highlighting is applied synchronously across all views (\(G_S\)). It allows comparing subgroups of procedures based on their values (\(G_C\)), providing a deeper insight into the workflows the different procedures followed (\(G_W\)).

Filter To compare specific subsets of procedures or even one procedure against another (\(G_C\)), users have the option to filter out procedures based on their values. Once a highlighted subset was selected, the users can either remove all other non-highlighted procedures or remove the highlighted procedures themselves to drill deeper into the data. Filtering can also be used to remove outliers that otherwise skew aggregated information.

Auxiliary In addition to the interaction techniques above, there are minor techniques to support the analysis. The layout of the application can be customized by extending, shrinking, or hiding the views of entire data categories or individual entities to compare information of interest more easily (\(G_C\)). Additionally, the investigation of the procedure reference view is supported by allowing users to reorder the procedures according to the depicted values, making correlations easier to detect (\(G_C\)).

The participants were given 13 tasks (summarized in Table 1) to solve using the application. Fig. 8 (right) shows an abstraction of which aspects were covered by which tasks. (Note that even though some main goals implicitly include others (e.g., \(G_W\) includes \(G_C\) and \(G_S\)), we only state the goals in the figure which were explicitly part of each task.) Our goal with the tasks was to evaluate whether experts are able to analyze and compare synchronized, post-operative surgical data to improve their understanding of the underlying workflows. Some consecutive tasks were also designed to reflect a sequence of analysis steps building on each other that might occur in a real-world scenario.

Results. The average correctness for each task, i.e., the percentage of participants who could solve it correctly, is depicted in Fig.8 (left). The overall average correctness was 78.8%. While the participants struggled with tasks 2 and 4, they could solve almost all tasks correctly later on.

We asked the participants whether they could verify existing hypotheses or come up with new ones using the application. In addition, we asked them for potential improvements of the application.

Results The open feedback showed that the participants could generate new hypotheses using the application. To give an example for a possible hypothesis: For a certain set of procedures, a specific phase was particularly difficult, indicated by a higher occurrence of the exceptional NOOP phase during that phase. A participant followed this investigation, mentioning in the free feedback that, for a selected subset, they found that “[p]rocedure[s] that experience diff[i]culties often only do so in one phase.” Another hypothesis proposed and validated by a participant was that during the “NOOP [phase], the gas pressure changes” for the sigmoid resection surgeries [10]. In addition, the participants found correlations between the duration of different phases as well as constraints as to when certain instruments are used. It turned out that the participants barely had any pre-existing hypotheses before the evaluation, so they focused on investigating those that came up during the evaluation. Regarding potential improvements, they said that the application would benefit from being more self-explanatory, e.g., by using more annotations. They also suggested some minor improvements of the interface, such as more precise descriptions of the x-axes in the summary histograms, or a simplification of certain interactions. Also, even for the larger set of 80 procedures from Cholec80 [16], none of the participants experienced performance issues.

We presented the participants the questions from the System Usability Scale (SUS) [17] to evaluate the usability of the application. The SUS relies on ten statements covering different usability aspects, each followed by a 5-point Likert scale question asking for the agreement of the participant with the statement. It yields an overall value between 0 (worst) and 100 (best).

Results The questionnaire yielded a score of 57.73. We give below the two statements for which our application received the best feedback on average. At that, 1 corresponds to “strongly disagree” and 5 to “strongly agree”:The two questions for which the feedback was on average least in favor of our application are the following:All other statements received feedback that was on average between the highest and the lowest rating stated above.