Cloud-Based Benchmarking of Medical Image Analysis

Systematic evaluation has had a strong impact on many data analysis domains, for example, TREC and CLEF in information retrieval, ImageCLEF in image retrieval, and many challenges in conferences such as MICCAI for medical imaging and ICPR for pattern recognition. With Kaggle, a platform for machine learning challenges has also had a significant success in crowdsourcing solutions. This shows the importance to systematically evaluate algorithms and that the impact is far larger than simply evaluating a single system. Many of these challenges also showed the limits of the commonly used paradigm to prepare a data collection and tasks, distribute these and then evaluate the participants’ submissions. Extremely large datasets are cumbersome to download, while shipping hard disks containing the data becomes impractical. Confidential data can often not be shared, for example medical data, and also data from company repositories. Real-time data will never be available via static data collections as the data change over time and data preparation often takes much time. The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of these problems and has been applied in the VISCERAL project. In EaaS, the data are not moved but remain on a central infrastructure. In the case of VISCERAL, all data were made available in a cloud environment. Participants were provided with virtual machines on which to install their algorithms. Only a small part of the data, the training data, was visible to participants. The major part of the data, the test data, was only accessible to the organizers who ran the algorithms in the participants’ virtual machines on the test data to obtain impartial performance measures.

This chapter gives a brief overview of the VISCERAL Registration System that is used for all the VISCERAL Benchmarks and is released as open source on GitHub. The system can be accessed by both participants and administrators, reducing the direct participant–organizer interaction and handling the documentation available for each of the benchmarks organized by VISCERAL. Also, the upload of the VISCERAL usage and participation agreements is integrated, as well as the attribution of virtual machines that allow participation in the VISCERAL Benchmarks. In the second part, a summary of the various steps in the continuous evaluation chain mainly consisting of the submission, algorithm execution and storage as well as the evaluation of results is given. The final part consists of the cloud infrastructure detail, describing the process of defining requirements, selecting a cloud solution provider, setting up the infrastructure and running the benchmarks. This chapter concludes with a short experience report outlining the encountered challenges and lessons learned.

This chapter describes the ethical and privacy aspects of using medical data in the context of the VISCERAL project. The project had as main goals the creation of a benchmark for organ segmentation, landmark detection, lesion detection and similar case retrieval. The availability of a large amount of imaging data was extremely important for the project goals, and thus, we present an analysis of the procedures that were followed for getting access to the data from IRB (internal review board) approval to data extraction and usage. This chapter details the requirements stated by medical ethics committees in three partner countries that supplied data. The exact procedure from request to data distribution is explained. The specific requirements of each data provider (each from a different country) are described in detail. The final data collection was made available in anonymized form in the Microsoft Azure cloud with the restriction of having it on servers that are located inside the European Union.

This chapter describes the annotation of the medical image data that were used in the VISCERAL project. Annotation of regions in the 3D images is non-trivial, and tools need to be chosen to limit the manual work and have semi-automated annotation available. For this, several tools that were available free of charge or with limited costs were tested and compared. The GeoS tool was finally chosen for the annotation based on the detailed analysis, allowing for efficient and effective annotations. 3D slice was chosen for smaller structures with low contrast to complement the annotations. A detailed quality control was also installed, including an automatic tool that attributes organs to annotate and volumes to specific annotators, and then compares results. This allowed to judge the confidence in specific annotators and also to iteratively refine the annotation instructions to limit the subjectivity of the task as much as possible. For several structures, some subjectivity remains and this was measured via double annotations of the structure. This allows the judgement of the quality of automatic segmentations.

In the VISCERAL project, several Gold Corpus datasets containing medical imaging data and corresponding manual expert annotations have been created. These datasets were used for training and evaluation of participant algorithms in the VISCERAL Benchmarks. In addition to Gold Corpus datasets, the architecture of VISCERAL enables the creation of Silver Corpus annotations of far larger datasets, which are generated by the collective ensemble of submitted algorithms. In this chapter, three Gold Corpus datasets created for the VISCERAL Anatomy, Detection and Retrieval Benchmarks are described. Additionally, we present two datasets that have been created as a result of the anatomy and retrieval challenge.

This chapter provides an overview of the metrics used in the VISCERAL segmentation benchmarks, namely Anatomy 1, 2 and 3. In particular, it provides an overview of 20 evaluation metrics for segmentation, from which four metrics were selected to be used in VISCERAL benchmarks. It also provides an analysis of these metrics in three ways: first by analysing fuzzy implementations of these metrics using fuzzy segmentations produced either synthetically or by fusing participant segmentations and second by comparing segmentation rankings produced by these metrics with rankings performed manually by radiologists. Finally, a metric selection is performed using an automatic selection framework, and the selection result is validated using the manual rankings. Furthermore, this chapter provides an overview of metrics used for the Lesion Detection Benchmark.

While a growing number of benchmark studies compare the performance of algorithms for automated organ segmentation or lesion detection in images with restricted fields of view, few efforts have been made so far towards benchmarking these and related routines for the automated identification of bones, inner organs and relevant substructures visible in an image volume of the abdomen, the trunk or the whole body. The VISCERAL project has organized a series of benchmark editions designed for segmentation and landmark localization in medical images of multiple modalities, resolutions and fields of view acquired during daily clinical routine work. Participating groups are provided with data and computing resources on a cloud-based framework, where they can develop and test their algorithms, the submitted executables of which are then run and evaluated on unseen test data by the VISCERAL organizers.

Health providers currently construct their differential diagnosis for a given medical case most often based on textbook knowledge and clinical experience. Data mining of the large amount of medical records generated daily in hospitals is only very rarely done, limiting the reusability of these cases. As part of the VISCERAL project, the Retrieval benchmark was organized to evaluate available approaches for medical case-based retrieval. Participant algorithms were required to find and rank relevant medical cases from a large multimodal dataset (including semantic RadLex terms extracted from text and visual 3D data) for common query topics. The relevance assessment of the cases was done by medical experts who selected cases that are useful for a differential diagnosis for the given query case. The approaches that integrated information from both the RadLex terms and the 3D volumes (mixed techniques) obtained the best results based on five standard evaluation metrics. The benchmark set up, dataset description and result analysis are presented.

Automatic segmentation of anatomical structures in CT scans is an essential step in the analysis of radiological patient data and is a prerequisite for large-scale content-based image retrieval (CBIR). Many existing segmentation methods are tailored to a single structure and/or require an atlas, which entails multistructure deformable registration and is time-consuming. We present a fully automatic atlas-free segmentation of multiple organs of the ventral cavity in contrast-enhanced CT scans of the whole trunk (CECT). Our method uses a pipeline approach based on the rules that determine the order in which the organs are isolated and how they are segmented. Each organ is individually segmented with a generic four-step procedure. Our method is unique in that it does not require any predefined atlas or a costly registration step and in that it uses the same generic segmentation approach for all organs. Experimental results on the segmentation of seven organs—liver, left and right kidneys, left and right lungs, trachea, and spleen—on 20 CECT scans of the VISCERAL Anatomy training dataset and 10 CECT scans of the test dataset yield an average DICE volume overlap similarity score of 90.95 and 88.50%, respectively.

In this chapter, we introduce an automatic multiorgan segmentation method using a hierarchical-shape-prior-guided level set method. The hierarchical shape priors are organized according to the anatomical hierarchy of the human body, so that the children structures are always contained by the parent structure. This hierarchical approach solves two challenges of multiorgan segmentation. First, it gradually refines the prediction of the organs’ position by locating and segmenting the larger parent structure. Second, it solves the ambiguity of boundary between two attaching organs by looking at a large scale and imposing the additional shape constraint of the higher-level structures. To improve the segmentation accuracy, a model-guided local phase term is introduced and integrated with the conventional region-based energy function to guide the level set propagation. Finally, a novel coherent propagation method is implemented to speed up the model-based level set segmentation. In the VISCERAL Anatomy challenge, the proposed method delivered promising results on a number of abdominal organs.

We propose a generic method for the automatic multiple-organ segmentation of 3D images based on a multilabel graph cut optimization approach which uses location likelihood of organs and prior information of spatial relationships between them. The latter is derived from shortest-path constraints defined on the adjacency graph of structures and the former is defined by probabilistic atlases learned from a training dataset. Organ atlases are mapped to the image by a fast (2+1)D hierarchical registration method based on SURF keypoints. Registered atlases are also used to derive organ intensity likelihoods. Prior and likelihood models are then introduced in a joint centroidal Voronoi image clustering and graph cut multiobject segmentation framework. Qualitative and quantitative evaluation has been performed on contrast-enhanced CT and MR images from the VISCERAL dataset.

This paper presents a pipeline which uses a multiatlas approach for multiorgan segmentation in whole-body CT images. In order to obtain accurate registrations between the target and the atlas images, we develop an adapted feature-based method which uses organ-specific features. These features are learnt during an offline preprocessing step, and thus, the algorithm still benefits from the speed of feature-based registration methods. These feature sets are then used to obtain pairwise non-rigid transformations using RANSAC followed by a thin-plate spline refinement or NiftyReg. The fusion of the transferred atlas labels is performed using a random forest classifier, and finally, the segmentation is obtained using graph cuts with a Potts model as interaction term. Our pipeline was evaluated on 20 organs in 10 whole-body CT images at the VISCERAL Anatomy Challenge, in conjunction with the International Symposium on Biomedical Imaging, Brooklyn, New York, in April 2015. It performed best on majority of the organs, with respect to the Dice index.

As part of their daily workload, clinicians examine patient cases in the process of formulating a diagnosis. These large multimodal patient datasets stored in hospitals could help in retrieving relevant information for a differential diagnosis, but these are currently not fully exploited. The VISCERAL Retrieval Benchmark organized a medical case-based retrieval algorithm evaluation using multimodal (text and visual) data from radiology reports. The common dataset contained patient CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) scans and RadLex term anatomy–pathology lists from the radiology reports. A content-based retrieval method for medical cases that uses both textual and visual features is presented. It defines a weighting scheme that combines the anatomical and clinical correlations of the RadLex terms with local texture features obtained from the region of interest in the query cases. The visual features are computed using a 3D Riesz wavelet texture analysis performed on a common spatial domain to compare the images in the analogous anatomical regions of interest in the dataset images. The proposed method obtained the best mean average precision in 6 out of 10 topics and the highest number of relevant cases retrieved in the benchmark. Obtaining robust results for various pathologies, it could further be developed to perform medical case-based retrieval on large multimodal clinical datasets.

Text- and content-based retrieval are the most widely used approaches for medical image retrieval. They capture the similarity between the images from different perspectives: text-based methods rely on manual textual annotations or captions associated with images; content-based approaches are based on the visual content of the images themselves such as colours and textures. Text-based retrieval can better meet the high-level expectations of humans but is limited by the time-consuming annotations. Content-based retrieval can automatically extract the visual features for high-throughput processing; however, its performance is less favourable than the text-based approaches due to the gap between low-level visual features and high-level human expectations. In this chapter, we present the participation from our joint research team of USYD/HES-SO in the VISCERAL retrieval task. Five different methods are introduced, of which two are based on the anatomy–pathology terms, two are based on the visual image content and the last one is based on the fusion of the aforementioned methods. The comparison results, given the different methods indicated that the text-based methods outperformed the content-based retrieval and the fusion of text and visual contents, generated the best performance overall.