Intrapapillary capillary loop classification in magnification endoscopy: open dataset and baseline methodology

We focus on the problem of classifying video frames as normal/abnormal. These frames are extracted from the magnification endoscopy (ME) recording of a patient. To the best of our knowledge, we introduce the first IPCL normal/abnormal open dataset¹ containing ME video sequences correlated with histopathology. Our dataset contains 68K video frames from 114 patients.

For a small and representative sample of 158 frames (IPCL types A, B1, B2, B3), we ask 12 senior clinicians to label them as normal/abnormal and report the inter-rater agreement as Krippendorff’s \(\alpha \) coefficient [8], achieving 76.6%. We also draw a comparison between raters and our gold standard histopathology results, achieving an average accuracy across raters of 94.7%.

We propose a novel convolutional network (CNN) architecture to solve the binary classification task with a particular focus on the explainability of predictions. Our proposed method achieved an average accuracy of 91.7%. In addition to a global classification estimation, our novel design produces activation maps and class scores at every resolution of the convolutional pyramid. The network has to explain where it is looking at prior to the generation of a class prediction. Looking at the activation maps for the abnormal class, we have observed that the network is looking at IPCL patterns when predicting abnormality. No conclusive evidence has been found that it is paying attention to large deep submucosal vessels to detect normal tissue. We believe that this baseline method may serve as a reference for future benchmarks on both video frame classification and explainability in the context of ESCN detection.

Patients attending for endoscopic assessment to two early squamous cell neoplasia (ESCN) referral centres in Taiwan (National Taiwan University Hospital and E-Da Hospital) were recruited with consent. Patients with oesophageal ulceration, active oesophageal bleeding or Barrett’s oesophagus were excluded. Gastroscopies were performed by two expert endoscopists (WLW, HPW), either under conscious sedation or local anaesthesia. An expert endoscopist was defined as a consultant gastroenterologist performing \(>50\) early squamous cell neoplasia (ESCN) assessments per year. All endoscopies were performed using an HD ME-NBI GIF-H260Z endoscope, with Olympus Lucera CV-290 processor (Olympus, Tokyo, Japan). A solution of water of simethicone was applied via the endoscope working channel to the oesophageal mucosa, in order to remove mucus, food residue or blood. This allowed good visualization of the oesophageal mucosa and microvasculature, including IPCLs.

Initially, a macroscopic assessment was made of the suspected lesion in an overview, with the borders of the lesion delineated by the endoscopist. The endoscopist then identified areas within the borders of the lesion on which to undertake magnification endoscopy. The IPCL patterns were interrogated using magnification endoscopy in combination with narrow-band imaging (ME-NBI). Magnification endoscopy was performed on areas of interest at \(80-100\)x magnification. Using the JES IPCL classification system, the IPCL patterns were classified by the consensus of three expert endoscopists (WW, HPW, RJH) as type A, B1, B2, B3, in order to give a prediction of the worst-case histology for the whole lesion. The entire lesion was then resected by either endoscopic mucosal resection (EMR) or endoscopic submucosal dissection (ESD). Resected specimens were formalin-fixed and assessed by an expert gastrointestinal histopathologist. As is the gold standard the worst-case histology was reported for the lesion as a whole, based on pathological changes seen within the resected specimen. Similarly to abnormal lesion areas, type A recordings (normal, healthy patients) were obtained by visual identification of healthy areas, magnification endoscopy, visual confirmation of normal vasculature and IPCL patterns, and biopsy to confirm the assessment.

Our IPCL dataset comprises a total of 114 patients (45 normal, 69 abnormal). Every patient has a ME-NBI video (30fps) recorded following protocol in “Correlating imaged areas with histology” section. Raw videos can present some parts where NBI is active. In this dataset, only ME subsequences are considered. All frames are extracted and assigned to the class normal or abnormal depending on the histopathology of the patient. They are quality controlled one-by-one (running twice over all the frames) by a senior clinician with experience in the endoscopic imaging of oesophageal cancer. Frames that are highly degraded due to lighting artifacts (e.g. blur, flares and reflections) up to the point where it is not possible (for the senior clinician) to make a visual judgement of whether they are normal or abnormal are marked as uninformative and not used. This curation process results in a dataset of 67742 annotated frames (28,078 normal, 39,662 abnormal) with an average of 593 frames per patient. For each fold, patients (not frames) are randomly split into 80% training, 10% validation (used for hyperparameter tuning), and 10% testing (used for evaluation). The statistics of each individual fold are presented in the supplementary material.

Let \(\{{\hat{y}}_{f,p}\}_{f=1}^{F_p}\) be the set of estimated probabilities for the frames f (out of \(F_p\)) belonging to patient clip p. Then, the estimated probability of abnormality for p is computed as an average of frame probabilities:Similarly to frame predictions, a threshold (\(p=0.5\)) is applied to obtain a class label for p. As per our data collection protocol (see “Correlating imaged areas with histology” section), magnification endoscopy clips contain either normal or abnormal tissue. Hence, a correlation between \(P(X={\hbox {abnormal}}|\{{\hat{y}}_{f,p}\}_{f=1}^{F_p})\) and histopathology is expected. The analysis of clip classification errors facilitates the identification of worst cases, singling out patient-wide mistakes from negligible frame prediction errors.

Explaining network predictions is of particular interest to draw a comparison between image features that clinicians employ in their clinical practice and those that might exist but be unknown to them. Conversely, adding attention to those image features that are known to be relevant but are not used by the network could potentially improve its performance. In the context of ESCN detection, this leads to investigate whether the network is actually looking at deep submucosal vessels and IPCL patterns to predict abnormality. The answer to this question typically comes in the form of a heatmap, with those parts relevant to the classification being highlighted.

Our baseline model (ResNet-18) may be formalized as \(f_{\varvec{\theta }} = r(h(g(T_{{\varvec{x}}})))\) where \(T_{{\varvec{x}}} = T_{\varvec{\theta }}({\varvec{x}}) \in {\mathbb {R}}^{H \times W \times K}\) is the feature tensor obtained after processing \({\varvec{x}}\) at the deepest pipeline resolution, K represents the number of feature channels, \(T_{{\varvec{x}}}(k)\) is a matrix that represents the feature channel with index k, and g, h, and r represent the global average pooling (GAP), fully connected (FC), and final scoring convolution layers, respectively.

The FC layer h represents a challenge for explainability, as relevance is redistributed when gradients flow backwards, losing its spatial connection to the prediction being made [11]. Hence, inspired by [18], we stripped out the fully connected layer of 1000 neurons from the baseline model (ResNet-18), connecting the output of the GAP directly to the neurons that predict class score (those in layer r) and setting their bias to zero. Formally, this leads to \(f_{\varvec{\theta }} = r(g(T_{{\varvec{x}}}))\), the output of the network before softmax beingwhere \(w_{k, c} \in {\hat{\varvec{\theta }}}\), and \({\hat{\varvec{y}}}^{(c)}\) is the score predicted for class c. Following this approach, a heatmap per class can be generated obviating the GAP layer during inference, simply computingThese heatmaps called class activation maps (CAMs) [18] keep a direct spatial relationship to the input, which is relevant for visual explanations. Although the architecture proposed in [18] requires removing the GAP layer to produce the CAMs, (2) can be reformulated asin which case the CAMs are embedded within the network pipeline as a \(1\times 1\) convolution (as we have already shown in [3]). This leads to \(f_{\varvec{\theta }} = g(r(T_{{\varvec{x}}}))\). We refer to this architecture as ResNet-18-CAM (as for the baseline, LR is set to \(5e-3\) and decayed by 0.5 every 10K iterations until 45K iterations). The performance of this network is shown in Table 2. Although the accuracy of ResNet-18-CAM is comparable to the baseline network (ResNet-18), ResNet-18-CAM conveniently computes a heatmap per class as part of the network processing. However, the explainability in the context of our classification problem remains very challenging due to the low resolution of the heatmaps produced.

In the computer vision field, images tend to display one or a few large objects. This is, however, not the case in medical images such as the magnification endoscopy ones used to classify IPCL patterns. Due to their low resolution, it is very challenging to understand what the network is looking at, as abnormal microvasculature in an endoscopic image is not localized only in a single spot. In our clinical problem, two types of features could be expected to be highlighted, submucosal vessels and IPCLs, which represent endoscopic markers for ESCN [7, 12, 13]. The procedure to generate the CAM proposed in [18] employs the deepest feature maps as inputs to produce the attention heatmaps. For our input images of \(256\times 256\) pixels, these feature maps have a resolution of \(8\times 8\) pixels, leading to very low-resolution CAMs (also \(8\times 8\) pixels ). This hinders the explanatory capability of the heatmaps, as small capillaries are the main clinically discriminating feature. It is of interest to know whether they are being looked at to predict abnormality. A trivial solution would be to reduce the depth of the network, but this could potentially hamper the learning of abstract features and decrease performance. In addition, the optimal amount of resolution levels for the given task to balance accuracy and explainability is a hyperparameter that would need to be tuned. Instead, we propose an alternative path modelling \(f_{\varvec{\theta }}({\varvec{x}})\) aswhere \(f_{\varvec{\theta _t}}\) represents the function that processes the input at resolution t, and whose output tensor has a width and height downsampled (strided convolution) by a factor of 0.5 with regards to its input tensor. In this formulation, given an \({\varvec{x}}\) of size \(256\times 256\) pixels and \(t=5\), the output of \(f_{\varvec{\theta _5}}\) is \(8\times 8\) pixels.

Given (5), let \(T_{{\varvec{x}}, t}\) be the output tensor produced by \(f_{\varvec{\theta _t}}\). Then, similarly to (4), we propose to generate a class score prediction at each resolution t as followsand final class scores are obtained as a sum over scores at different resolutions:As indicated by (6), prior to generating a class prediction, a CAM at resolution t is produced. This heatmap contains both positive and negative contributions from the input image towards class c. However, for the sake of heatmap clarity, we consider only the positive contributions towards each class when generating our CAMs. That is, we want to see what part of the image contributes to normality/abnormality, as opposed to what part of the image does not contribute to normality/abnormality. Thus, our CAMs are generated as followswhere \(z^{+} = \max (0, z)\). A loss based just on this final score alone would not force the network to produce meaningful CAMs at every resolution level. Therefore, we also propose to deeply supervise the side predictions in our proposed loss:where \({\varvec{x}}\) is the input image, y the ground truth class label, \({\hat{\varvec{\theta }}}\) the network parameters, and \(\{ {\hat{\varvec{y}}}^{(c)}_t \}_{c=1,t=1}^{C, T}\) represent the score predictions for each class c at resolution t. Both \({\mathcal {L}}_f(\cdot )\) and \({\mathcal {L}}_s^t(\cdot )\) are denoted \({\mathcal {L}}_f\) and \({\mathcal {L}}_s^t\) for a simplified notation. \({\mathcal {L}}_f\) is defined aswhere \(\sigma (\cdot )_{c}\) represents the softmax function for class index c. \({\mathcal {L}}_s^t\) is the side loss for the prediction at each different resolution t, defined as:In addition to the network generating CAMs at every resolution prior to generating the scores as part of the prediction pipeline, the combined loss \({\mathcal {L}}\) proposed allows for the validation of the accuracy at each resolution depth of the network. We refer to the architecture that implements the model in (5) with embedded CAMs at different resolutions following (6) and loss (9) as ResNet-18-CAM-DS (see Fig. 2).