Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals
Introduction
World Health Organization (WHO) criteria for response assessment [1], [2] and, more recently, the Response evaluation criteria in solid tumors (RECIST) [3], [4] are imaging-based guidelines that are widely implemented in clinical trials and also used in clinical practice to determine patient outcomes. In these criteria, tumor response assessment is based upon measured change in tumor diameter(s) determined on cross-sectional modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). However, the response cut-off values (e.g., diameter decreases ≥30% are considered as partial responses by RECIST) were developed by evaluating the measurement error of antiquated response assessment modalities used during the 1970s and early 1980s (i.e., physical palpation or plain X-ray measurements) [5], [6]. For 30 years, despite the ubiquitous use of sectional imaging techniques in response assessment and the remarkable progress that has been made in these technologies and image analysis algorithms, limited data have been published on quantifying measurement variability of these new technologies and techniques for response assessment.
Recently, a same-day repeat CT study reported the measurement variability in non-small cell lung cancer (NSCLC) on thin slice CT images and its potential impact on response assessment [7], [8]. The study found much lower magnitudes of measurement variability/error compared to the cut-off values set by the RECIST and WHO criteria to define tumor response. Indeed, with the latest imaging techniques and tumor size measurement tools, it should be possible to quantify much smaller tumor changes earlier during the treatment, and thus provide earlier assessment of patient outcome.
Volumetric techniques have begun to demonstrate advantages over uni-dimensional (1D) measurement in the response assessment of NSCLC tumors treated with targeted therapies and in the discovery of tumor biomarkers [9], [10]. With the growing number of computer algorithms available for facilitating tumor volume (VOL) calculation, and as clinical trials begin to incorporate tumor VOL as an exploratory endpoint, it is now critical to optimize and validate the use of computer-aided volumetric technique in such trials.
Understanding the magnitude of measurement variability is essential for establishing a new volumetric response method and re-evaluating conventional 1D and 2D methods to better detect tumor changes with therapies, especially novel targeted therapies that may not produce tumor changes at the same magnitude or speed on radiographic images as observed in those treated with classic cytotoxic chemotherapies. Another critical issue for clinical trials that are incorporating volumetric response method is the proper (optimal) use of imaging acquisition/reconstruction parameters (e.g., slice interval). Can we use the current standard slice interval (thickness) of 5 mm to adequately measure tumor volumes? Would a thinner slice interval of 2.5 mm or 1.25 mm provide more reliable uni-, bi-dimensional and volumetric measurements for assessing therapy response? Determining optimal/uniform CT imaging acquisition parameters is an urgent task that would help improve interpretation of measurement results obtained especially from those early phase multicenter clinical trials in which a small number of patients are enrolled.
This study was therefore designed to (1) quantify the level of intra-/inter-reader variability in 1D, 2D, and VOL measurements of solid tumors in the lungs, liver and lymph nodes, the three most frequently involved metastatic sites, on the three most commonly used CT slice intervals (5, 2.5 and 1.25 mm), and (2) assess the agreement between manual and computer-assisted methods (CAM) for tumor measurement.
Section snippets
Clinical imaging data
This study was conducted as a prospective exploratory project utilizing data acquired according to protocol-scheduled tumor assessments. This ancillary, HIPAA compliant study was approved by the institutional review board, with informed patient consent waived.
CT imaging scan data collected from 30 patients, enrolled between May 2007 and June 2009 in several Phase I or II oncology clinical trials evaluating systemic therapies in melanoma, lymphoma, lung, pancreatic, and colorectal cancer
Statistical analysis
Analyses were performed with SAS software (version 8.1; SAS Institute, Cary, NC). Separate analyses were performed to obtain the CAM and manual, intra- and inter-reader components of variability for 1D, 2D and CAM VOL measurements at each slice interval. Linear mixed-effects analysis of variance (ANOVA) models [12] were fitted to the natural logarithm of the tumor lesion measurements for each patient. Patient and lesion were included as random effects in the model to account for the repeated
Results
One hundred eighteen (118) lesions in 30 patients were selected for the study. Overall, lesion diameters (averaged over the three slice intervals) ranged from 6.1 mm to 80.1 mm, with a median of 18.4 mm.
Reader 2 carried out 708 measurements (118 lesions × 3 slice intervals × 2 manual measurements) and reader 1, who performed the repeat measurements, carried out 1416 measurements (708 measurements × 2). Four out of 708 (0.6%) of reader 2's manual measurements were missing. Given their small number and
Discussion
Variability in tumor VOL measurements due to CT scanner, dose, imaging acquisition and reconstruction parameters and segmentation algorithm has previously been studied mainly with a focus on pulmonary nodules detected in CT lung cancer screening programs and routine clinical care for the purpose of non-invasive diagnosis [16], [17], [18], [19]. However, in those clinical settings, nodules are usually small (<2 cm with an average size <10 mm) and image spatial resolutions are high (≤1 mm in z-axial
Conclusions
We assessed the intra- and inter-reader variability in 1D, 2D, and VOL measurements of solid tumors in the lungs, liver and lymph nodes made both manually and by CAM on CT scans reconstructed at three slice intervals of 5, 2.5 and 1.25 mm that cover a potential slice interval range of today's and future clinical trials and clinical practice. Our findings showed that, compared to 5 mm and 1.25 mm slice intervals, 2.5 mm may provide the best viewing and the most robust computation conditions for
Role of the funding source
The study sponsors provided funding for the research; they did not have any role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the paper for publication.
Conflict of interest
No conflict of interest has been declared.
Acknowledgements
This work was in part supported by Grant R01CA125143 from the National Cancer Institute (NCI) and a research grant from AstraZeneca. The content is solely the responsibility of the authors and does not necessarily represent the funding sources. The authors would like to thank Caroline Shepherd for her help in editing the manuscript.
References (24)
- et al.
New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)
European Journal of Cancer
(2009) - et al.
Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets
Translational Oncology
(2009) - et al.
Application of coefficient of variation in reliability-based mechanical design and manufacture
Journal of Materials Processing Technology
(2001) - et al.
Variability in size measurement of renal masses smaller than 4 cm on computerized tomography
Journal of Urology
(2006) - WHO handbook for reporting results of cancer treatment. Geneva (Switzerland). World Health Organization Offset...
- et al.
Reporting results of cancer treatment
Cancer
(1981) - et al.
New guidelines to evaluate response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada
Journal of the National Cancer Institute
(2000) - et al.
The effect of measuring error on the results of therapeutic trials in advanced cancer
Cancer
(1976) - et al.
Studies in variation associated with the measurement of solid tumors
Cancer
(1980) - et al.
Evaluating variability in tumor measurements from same-day repeat CT scans in patients with non-small cell lung cancer
Radiology
(2009)
Variability of lung tumor measurements on repeat computed tomography (CT) scans taken within 15 minutes: implications for care and clinical research
Journal of Clinical Oncology
A pilot study of volume measurement as a method of tumor response evaluation to aid biomarker development
Clinical Cancer Research
Cited by (74)
On-cloud decision-support system for non-small cell lung cancer histology characterization from thorax computed tomography scans
2023, Computerized Medical Imaging and GraphicsCorrelation of Radiomics with Treatment Response in Liver Metastases
2023, Academic RadiologyLung nodule diagnosis and cancer histology classification from computed tomography data by convolutional neural networks: A survey
2022, Computers in Biology and MedicineClinical outcome of Yttrium-90 selective internal radiation therapy (Y-90 SIRT) in unresectable hepatocellular carcinoma: Experience from a tertiary care center
2022, Liver ResearchCitation Excerpt :Multiple factors including patient's preference, clinician's decision as well as the patient and disease status might introduce selection bias to recommend Y-90 SIRT for these HCC patients. Tumor volume, which may affect clinical outcomes, was not evaluated in the present study due to technical difficulties and high inter-observer variability in such determination.30,35 Instead, we selected the largest tumor size criteria to evaluate the impact of tumor burden on survival.
Hierarchical classification of early microscopic lung nodule based on cascade network
2024, Health Information Science and Systems
- 1
Current address: The Second Hospital of Hebei Medical University, Shijiazhuang, Hebei 050000, China.
- 2
Department of Radiology, Columbia University Medical Center, 710 West 168th Street, New York, NY 10032, USA.
- 3
North Middlesex University Hospital, London, Greater London N18 1QX, United Kingdom.
- 4
Flat 3, 113 Old Tiverton Road, Exeter, Devon EX4 6LD, United Kingdom. Tel.: +44 01392 21539/625 516730.
- 5
AstraZeneca, Statistics and Informatics, Clinical Information Science, 90F169-1, East Wing, Parklands, Alderley Park, Macclesfield, Cheshire SK10 4TG, United Kingdom.
- 6
AstraZeneca, Personalised Healthcare and Biomarkers, Alderley Park, Macclesfield SK10 4TG, United Kingdom.
- 7
Department of Radiology Columbia, University Medical Center, 180 Fort Washington Avenue, New York, NY 10032, USA.
- 8
AstraZeneca, Oncology Global Medicine Development - 41, 11G Mereside, Alderley Park, Macclesfield, Cheshire, SK10 4TG, United Kingdom.