A Novel Public MR Image Dataset of Multiple Sclerosis Patients With Lesion Segmentations Based on Multi-rater Consensus

Quantified volume and count of white-matter lesions based on magnetic resonance (MR) images are important biomarkers in several neurodegenerative diseases. For a routine extraction of these biomarkers an accurate and reliable automated lesion segmentation is required. To objectively and reliably determine a standard automated method, however, creation of standard validation datasets is of extremely high importance. Ideally, these datasets should be publicly available in conjunction with standardized evaluation methodology to enable objective validation of novel and existing methods. For validation purposes, we present a novel MR dataset of 30 multiple sclerosis patients and a novel protocol for creating reference white-matter lesion segmentations based on multi-rater consensus. On these datasets three expert raters individually segmented white-matter lesions, using in-house developed semi-automated lesion contouring tools. Later, the raters revised the segmentations in several joint sessions to reach a consensus on segmentation of lesions. To evaluate the variability, and as quality assurance, the protocol was executed twice on the same MR images, with a six months break. The obtained intra-consensus variability was substantially lower compared to the intra- and inter-rater variabilities, showing improved reliability of lesion segmentation by the proposed protocol. Hence, the obtained reference segmentations may represent a more precise target to evaluate, compare against and also train, the automatic segmentations. To encourage further use and research we will publicly disseminate on our website http://​lit.​fe.​uni-lj.​si/​tools the tools used to create lesion segmentations, the original and preprocessed MR image datasets and the consensus lesion segmentations.