Arthemis: Annotation software in an integrated capturing and analysis system for colonoscopy

https://doi.org/10.1016/j.cmpb.2007.07.011Get rights and content

Abstract

Colonoscopy is an endoscopic technique that allows physicians to inspect the inside of the human colon. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. In current practice, the entire colonoscopic procedure is not routinely captured. Software tools providing easy access to important contents of videos that are digitally captured during colonoscopy are not available. Hence, it is very time consuming to review an entire video, locate important contents, annotate them, and extract the annotated contents for research, teaching, and training purposes. Arthemis, a software application, was developed to facilitate this process. For convenient data sharing, Arthemis allows annotation according to the European Gastrointestinal Society for Endoscopy (ESGE) Minimal Standard Terminology (MST), an internationally accepted standard for digestive endoscopy. Arthemis is part of our integrated capturing and content analysis system for colonoscopy called Endoscopic Multimedia Information System (EMIS). This paper presents Arthemis as a component of EMIS, the design and implementation of Arthemis, and key lessons learned from the development process.

Introduction

Colonoscopy is currently the preferred screening modality for prevention of colorectal cancer—the second highest cause of cancer-related deaths behind lung cancer in the US [1]. During a colonoscopic procedure, a flexible endoscope (with a tiny video camera at the tip) is gradually advanced under direct vision via the anus and rectum into the most proximal part of the colon (signified by the appearance of the appendiceal orifice or the terminal ileum). Next, the endoscope is gradually withdrawn. The video camera generates a sequence of images of the internal mucosa of the colon. These images are displayed on a monitor for real-time manual analysis by the endoscopist. Diagnostic and therapeutic operations such as polyp removal can be performed during the procedure. In current practice, videos of the entire colonoscopic procedure are not routinely captured for post-procedure review or analysis. A few snapshot images maybe taken to document the procedure or unusual findings.

Over 14 millions colonoscopic procedures are performed annually [2]. Some evidence of variations in quality of colonoscopic procedures among different physicians was recently reported [3], [4]. To measure in some detail what exactly is achieved during colonoscopy, ESGE proposed taking snapshot images at recommended positions during colonoscopy [5]. A number of indirect markers of quality have also been proposed [6]. These include duration of the withdrawal phase and the percentage of patients with polyps detected during screening colonoscopy. Despite intensive research in medical imaging in recent years, research on automatic content analysis of colonoscopy videos has been minimal. Computer-based methods were proposed to identify polyps in endoscopic images using texture features, which include Color Wavelet Covariance analysis [7], [8], and Texture Spectrum classification using Neural Network Classifier [9], [10]. We found no other existing algorithms and tools proposed to automatically document a colonoscopic procedure and derive objective measurements of quality of the procedure. Motivated by this fact, we have been developing Endoscopic Multimedia Information System (EMIS) that has the abilities to transparently capture the entire colonoscopic procedure into a colonoscopy video file, analyze the captured video for important contents, provide efficient access to these contents, and derive quality measurements (e.g., withdrawal time, visualization of the appendiceal orifice) of the procedure [11].

Arthemis,1 a software application, is the only component in EMIS that directly interacts with end users to provide easy access to important contents of the captured video and allow annotation on these contents. Endoscopists and medical students can use Arthemis to gain knowledge from studying colonoscopic procedures operated by experienced endoscopists or to evaluate the quality of the procedures. Hence, graphical user interface and functional components are important design factors. Guided by comments and suggestions from the endoscopist, Arthemis provides the following key features: (1) play, pause, or jump to a specific frame in a colonoscopy video to facilitate selection of images for annotation; (2) preview a video at a fast rate up to 8 times the normal playback speed; this feature reduces the time taken to review and annotate an entire procedure; (3) play important segments (e.g., segment showing polypectomy) determined by our analysis software [12]; (4) import endoscopic images for annotation; (5) annotate images using ellipse and free-hand-draw tools and associate the European Gastrointestinal Society for Endoscopy (ESGE) Minimal Standard Terminology (MST) terms with the annotated ellipse figure; (6) zoom in to get a closer look at details in the captured image or zoom out to view a larger portion of the image; (7) record annotated figures in an APRO (Arthemis Project) format based on Extensible Markup Language (XML) for later retrieval; (8) create video clips and corresponding APRO files with relevant annotation from the original colonoscopy video; and (9) authenticate user licenses with encrypted messages. These features post several challenges during the design and implementation process.

Arthemis is implemented mostly in Java with some features implemented in C using Microsoft DirectShow and a third-party software library for MPEG decoding and encoding [13]. Arthemis provides two unique functionalities: (1) annotation ability using ESGE MST and (2) ability to view automatically detected segments of colonoscopy videos. MST terminology is an internationally accepted standard for digestive endoscopy, which was proposed to enable electronic data exchange of the results of endoscopy examinations. To the best of our knowledge, no other software tools provide the two aforementioned abilities.

The rest of this paper is organized as follows. Section 2 introduces some background on EMIS and Arthemis as a user-interface component of EMIS. In Section 3, we briefly discuss enabling technologies used in the development of Arthemis. Section 4 discusses the design and implementation of Arthemis. In Section 5, we conclude the paper with the description of the current status of the software and experience gained during the software development process.

Section snippets

Background

This section provides a brief discussion on existing annotation software and overview of EMIS.

Enabling technologies

Here we discuss key technologies used in the development of Arthemis. Java was chosen to implement Arthemis due to the language familiarity of the development team. Standard Java provides two major GUI toolkits: AWT (Abstract Window Toolkit) and Swing. AWT includes a platform-independent Java API to wrap the native GUI widgets of the various operating systems. However, it only provides a very limited set of widgets for building graphical user interfaces (GUIs). At best, AWT is only good for

Design considerations and system description

The first prototype of Arthemis was completed in February 2005 to get an initial feedback from the endoscopist in terms of the user interface and functionalities. The current version was completed in November 2006. The current design is based on user feedback regarding the first prototype. The required features have already been discussed in Section 1. In this section, we only discuss important features of the current design.

Discussion and conclusion

Arthemis has gone through several revisions in order to satisfy the user's requirements in terms of functionality, user-interface, and performance of fast playback. As a result, Arthemis is starting to become a useful tool for endoscopic research and education. Training in recognition of endoscopic findings at present is limited to verbal descriptions combined with one or a few pictures of a representative lesion. Rarely if ever is a single video shown when discussing protruding lesions within

References (33)

  • S. Hwang et al.

    Automatic measurement of quality metrics for colonoscopy videos

  • Y. Cao et al.

    Computer-aided detection of diagnostic and therapeutic operations in colonoscopy videos

    IEEE Trans. Biomed. Eng.

    (2007)
  • Mainconcept,...
  • Normedia cdrom minimal standard terminology for databases in digestive endoscopy,...
  • Provision 1.5 video image management system for video endoscopes,...
  • Given imaging,...
  • Cited by (19)

    • Using artificial intelligence to improve adequacy of inspection in gastrointestinal endoscopy

      2020, Techniques and Innovations in Gastrointestinal Endoscopy
      Citation Excerpt :

      AI software also has been created using training sets of inside-the-patient and outside-the-patient images. The value of detailed, manually annotated training and testing video file corpora cannot be overstated: they are crucial for development of any AI system; they can be reused if newer AI methods become available, and the video files can be re-annotated for additional features if new features need to be discovered [24]. Examples of features that can be annotated range from simple to complex.

    • Advanced Systems to Assess Colonoscopy

      2010, Gastrointestinal Endoscopy Clinics of North America
      Citation Excerpt :

      Inside the project folder reside all the annotations, the individual frames that are annotated, and any clips generated from the video file (Fig. 5). All annotations are stored in a single extensible markup language (XML) file with extension, APRO (for Arthemis PROject; the original code name of the project14). All images are stored in an IMG folder.

    • Study of healthcare annotation systems

      2021, International Journal of E-Health and Medical Communications
    • Study of Healthcare Professionals’ Interaction in the Patient Records Based on Annotations

      2020, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus

    This work is partially supported by the US National Science Foundation Grant No. IIS-0513777, IIS-0513809, and IIS-0513582, the Mayo Clinic, and Grow Iowa Values Fund.

    View full text