Smart Endoscope—Firmware Complex for Real-Time Analysis and Recognition of Endoscopic Videos

Endoscopic research is a minimally invasive medical procedure that allows examination of body cavities, including gastric tract. It is the most reliable method for revealing early stages of gastric mucosa malignant neoplasms, and pre-cancer diseases that increase chances for cancer development [2, P.88].

First flexible tract endoscopes appeared in early 1960s, and an ability to transform optical signal into electric impulses allowed displaying images, and storing them in analog, and later—digital form. In late 1980s—early 1990s, with the expanded availability of computers, and the development of programming languages and environments allowed development of information systems for search and classification of gastric surface epithelial neoplasms, referred to as CADs (computer-aided diagnosis systems). CADs produce additional diagnostic data or preliminary diagnosis (either posterior or real time), or series of marked images used by a specialist to select a case that best matches the observed one. Recent decade was marked with an increase of interest directed toward CADs [10, P.73], which can be related to accumulation of an extensive set of image and video files, transforming visual examination, analysis, interpretation, and classification of that data into an extremely labor-intensive process. Besides that, developments in the area of wireless capsule endoscopy made the task of creating automated image analysis and diagnostic methods extremely relevant.

Medic performing an endoscopic examination has to possess substantial experience and remain focused during the whole procedure. Endoscopic image recognition accuracy greatly depends upon the specialist qualification, and is usually considered to be between 80 and 100% [5, P.174; 1, P.A507]. Modern CADs come close to visual examination by this criterion; however, they require further improvement before implementation into clinical practice [6, P.15; 7, P.350; 4, P.7130]. It is necessary to refine the algorithms used to select, analyze, and classify characteristic image elements while medical community reaches consensus on the topic of visual criteria of endoscopic image assessment [3, P.471; 8, P.526; 11, P.17].

The purpose of research was to develop and implement methodology of making diagnostic solutions based upon high-resolution endoscopic images in a CAD system in order to improve quality of diagnosis for stomach and colon oncological diseases.

Research goals included the following:

An algorithm used to analyze microstructure of stomach or colon, which consists of capillaries, grands, and their excretory ducts, includes the following stages:

Skeleton of the capillaries is built using Zhang–Suen algorithm [12, P.237]. Image of the pits is obtained by subtracting image of capillaries from the part of an initial image limited by an area interest. At each skeleton branch, secants that are perpendicular to the branches were drawn with equal intervals. Intensity profiles are built along the secants, and used to determine thickness of pits and capillaries. Distortion degree for pit and capillary boundaries is characterized by a relative intensity drop that is calculated using formula (1)where Max is the intensity of a central pixel; and LeftMin and RightMin are the closest local minimums located left and right of a central maximum.

In order to perform recognition the interest area is divided into squares with preset size. For each square six parameters are calculated: C ₁—ratio of an average pit thickness to average capillary thickness; C ₂—standard deviation of pit thickness; C ₃—average value for relative intensity drop in secant profiles for the pits (distortion), calculated using (1); C ₄—share of single-node clusters; C ₅—share of two-node clusters (rods); C ₆—number of end points. Coefficient that characterizes mucosa within a given square is calculated using the following formula:where B _i is a coefficient equal to 0 if C _i  ≥ C _i2, and equal to 1 if C _i  < C _i2; and p _i and C _i2 are threshold values determined by an expert.

The range of K _a values is broken into four equal intervals. Depending upon the value of K _a coefficient, the boundaries of square area are color coded, and serve as local indicators of status for the parts of mucosa (Fig. 2a).

In order to characterize the overall state of a part of gastric or colon mucosa, presented on an image, as a whole, the value of C ₁ for each square is additionally compared with expert-defined threshold value C ₁₀. Squares with C ₁ ≥ C ₁₀ are marked as key ones. For eight squares connected with key one, the following pairs of conditions are checked: (a) \(C_{2} \ge \widetilde{N}_{20}\) AND \(\widetilde{N}_{1} \ge \widetilde{N}_{11}\), and (b) \(C_{3} \ge \widetilde{N}_{30}\) AND \(\widetilde{N}_{1} \ge \widetilde{N}_{11}\). The squares, for each at least one pair of conditions (a or b) is true, together with key square form the risk zone R. Image can contain several risk zones. Indicative coloring of risk zones is performed based upon the results of comparing values of C ₁, C ₂, and C _3, averaged for a zone, with threshold values set by an expert. After examination, the boundaries of risk zone squares get colored with indicative colors (Fig. 2b).

Endoscopic images were also processed using GoogLeNet neural network. Training set of endoscopic images contained 200 neoplasm cases, and 300 cases where no neoplasms were present. All endoscopy images were loaded into a web atlas (http://​endoscopy.​siams.​com). Histologic composition was specified for each image in accordance with WHO classification of gastric tumors of 2010, considering Vienna classification of gastrointestinal epithelial neoplasia of 2002 [9, P.52]. Medical practitioners applied web atlas tools to loaded histological images from the training set in order to select interest areas to be examined with different methods including histology. Initial image description in web atlas, created by medical professionals, contained additional data on regularity of mucosa microstructure. No univocal correspondence was determined between the binary regularity criterion, and binary neoplasm criterion, the following model for qualitative assessment of the changes was proposed:where K _n is a value for relative coefficient of gastric mucosa change, obtained using neural network; p ₁, p ₂—weights, K _c is a binary neoplasm criterion, and K _r is a binary criterion of regularity.

Table 1 presents interpretation of the model. Examples of endoscopic image fragments with assessment variants are presented on Fig. 3(a–d).

On the images marked up by medics fragments were cut from selected interest areas using sliding window. Empirically selected window size was 135 × 135 pixel. For each selected fragment, a value of relative change coefficient K _n was assigned. That resulted in a training set containing more than 30,000 images, equally distributed among four classes corresponding with values of K _n . Neural network was additionally trained to remove glares, recognize background, distorted fragments, and blood. According to cross-validation of neural network use results, the accuracy of assessing K _n value with regard to initial markup was equal to 99%. Example of an image marked up using neural network is presented on Fig. 3. Markup is an overlapping regular grid, with corresponding calculated value of K _n and color marker. For each endoscopic image, the relative shares were calculated for the cells of each color. Descriptor of an endoscopic image is a four-dimensional vector D ₄ that represents calculated shares.

When using neural network the accuracy of assessing K _n , obtained using cross-validation with regard to initial markup When using neural network the accuracy of assessing K _n , obtained using cross-validation with regard to initial markup was equal to 93%.

Complex architecture is presented on Fig. 4, it is distributed and depends upon user roles. Real-time mode operator (role 1) uses the complex during the endoscopic examination, so he has to use the complete complex functionality of autonomous mode. Hardware of an autonomous module (module 1 on Fig. 4) is based upon the high-capacity graphic card, and includes wide-screen sensor display used to visualize recognition results and hints. Signal from endoscope recorder is transmitted to one of the graphic cards of an autonomous module that interacts with local server (module 2) using https protocol. During the operation of an autonomous module, marked up images are uploaded to local server.

Deferred mode operator (role 2) can change image markup, set values of image attributes, and place image to a training set using web interface. Images placed into training set can be reached by remote operator of a local server (role 3) through the web atlas mode. In order to supplement training set, selected image fragments transmit to a central server (module 3) using https. Training set on a central server is obtained by combining image collections from local servers. Central server operator (role 4) controls image quality.

Neural network training is performed using an external cluster (module 4) in order to not hamper work of a central server operator Neural network training takes several hours, and is regularly performed once in several days. Trained neural network represented by a complex mathematical model is translated from central to local servers, and then—to autonomous modules, in order to revalidate recognition results.

The research allowed transformation of visual assessment criteria provided by endoscopists into a set of formal characteristics describing microstructure of gastric and colon mucosa. That was an important step in development of “Smart endoscope”—firmware complex implementing automated endoscopic image analysis and recognition algorithms in real-time mode. Combination of the neoplasm recognition approaches described above allowed obtaining experimental accuracy comparable with accuracy of specialist analysis.

Research results allow hoping that future wide use of CAD systems in endoscopy will allow decreasing analysis subjectivity, improve quality of diagnostics, and cut research cost and time. Use of CADs as pathology classification tool for education and diagnostics will facilitate improving qualification of young specialists, and distribution of expert knowledge that becomes a base for the system intellectual and analytic kernel.