Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals

doi:10.1016/j.ejrad.2013.02.018

European Journal of Radiology

Volume 82, Issue 6, June 2013, Pages 959-968

https://doi.org/10.1016/j.ejrad.2013.02.018 Get rights and content

Abstract

Objective

Understanding magnitudes of variability when measuring tumor size may be valuable in improving detection of tumor change and thus evaluating tumor response to therapy in clinical trials and care. Our study explored intra- and inter-reader variability of tumor uni-dimensional (1D), bi-dimensional (2D), and volumetric (VOL) measurements using manual and computer-aided methods (CAM) on CT scans reconstructed at different slice intervals.

Materials and methods

Raw CT data from 30 patients enrolled in oncology clinical trials was reconstructed at 5, 2.5, and 1.25 mm slice intervals. 118 lesions in the lungs, liver, and lymph nodes were analyzed. For each lesion, two independent radiologists manually and, separately, using computer software, measured the maximum diameter (1D), maximum perpendicular diameter, and volume (CAM only). One of them blindly repeated the measurements. Intra- and inter-reader variability for the manual method and CAM were analyzed using linear mixed-effects models and Bland–Altman method.

Results

For the three slice intervals, the maximum coefficients of variation for manual intra-/inter-reader variability were 6.9%/9.0% (1D) and 12.3%/18.0% (2D), and for CAM were 5.4%/9.3% (1D), 11.3%/18.8% (2D) and 9.3%/18.0% (VOL). Maximal 95% reference ranges for the percentage difference in intra-reader measurements for manual 1D and 2D, and CAM VOL were (−15.5%, 25.8%), (−27.1%, 51.6%), and (−22.3%, 33.6%), respectively.

Conclusions

Variability in measuring the diameter and volume of solid tumors, manually and by CAM, is affected by CT slice interval. The 2.5 mm slice interval provides the least measurement variability. Among the three techniques, 2D has the greatest measurement variability compared to 1D and 3D.

Introduction

World Health Organization (WHO) criteria for response assessment [1], [2] and, more recently, the Response evaluation criteria in solid tumors (RECIST) [3], [4] are imaging-based guidelines that are widely implemented in clinical trials and also used in clinical practice to determine patient outcomes. In these criteria, tumor response assessment is based upon measured change in tumor diameter(s) determined on cross-sectional modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). However, the response cut-off values (e.g., diameter decreases ≥30% are considered as partial responses by RECIST) were developed by evaluating the measurement error of antiquated response assessment modalities used during the 1970s and early 1980s (i.e., physical palpation or plain X-ray measurements) [5], [6]. For 30 years, despite the ubiquitous use of sectional imaging techniques in response assessment and the remarkable progress that has been made in these technologies and image analysis algorithms, limited data have been published on quantifying measurement variability of these new technologies and techniques for response assessment.

Recently, a same-day repeat CT study reported the measurement variability in non-small cell lung cancer (NSCLC) on thin slice CT images and its potential impact on response assessment [7], [8]. The study found much lower magnitudes of measurement variability/error compared to the cut-off values set by the RECIST and WHO criteria to define tumor response. Indeed, with the latest imaging techniques and tumor size measurement tools, it should be possible to quantify much smaller tumor changes earlier during the treatment, and thus provide earlier assessment of patient outcome.

Volumetric techniques have begun to demonstrate advantages over uni-dimensional (1D) measurement in the response assessment of NSCLC tumors treated with targeted therapies and in the discovery of tumor biomarkers [9], [10]. With the growing number of computer algorithms available for facilitating tumor volume (VOL) calculation, and as clinical trials begin to incorporate tumor VOL as an exploratory endpoint, it is now critical to optimize and validate the use of computer-aided volumetric technique in such trials.

Understanding the magnitude of measurement variability is essential for establishing a new volumetric response method and re-evaluating conventional 1D and 2D methods to better detect tumor changes with therapies, especially novel targeted therapies that may not produce tumor changes at the same magnitude or speed on radiographic images as observed in those treated with classic cytotoxic chemotherapies. Another critical issue for clinical trials that are incorporating volumetric response method is the proper (optimal) use of imaging acquisition/reconstruction parameters (e.g., slice interval). Can we use the current standard slice interval (thickness) of 5 mm to adequately measure tumor volumes? Would a thinner slice interval of 2.5 mm or 1.25 mm provide more reliable uni-, bi-dimensional and volumetric measurements for assessing therapy response? Determining optimal/uniform CT imaging acquisition parameters is an urgent task that would help improve interpretation of measurement results obtained especially from those early phase multicenter clinical trials in which a small number of patients are enrolled.

This study was therefore designed to (1) quantify the level of intra-/inter-reader variability in 1D, 2D, and VOL measurements of solid tumors in the lungs, liver and lymph nodes, the three most frequently involved metastatic sites, on the three most commonly used CT slice intervals (5, 2.5 and 1.25 mm), and (2) assess the agreement between manual and computer-assisted methods (CAM) for tumor measurement.

Section snippets

Clinical imaging data

This study was conducted as a prospective exploratory project utilizing data acquired according to protocol-scheduled tumor assessments. This ancillary, HIPAA compliant study was approved by the institutional review board, with informed patient consent waived.

CT imaging scan data collected from 30 patients, enrolled between May 2007 and June 2009 in several Phase I or II oncology clinical trials evaluating systemic therapies in melanoma, lymphoma, lung, pancreatic, and colorectal cancer

Statistical analysis

Analyses were performed with SAS software (version 8.1; SAS Institute, Cary, NC). Separate analyses were performed to obtain the CAM and manual, intra- and inter-reader components of variability for 1D, 2D and CAM VOL measurements at each slice interval. Linear mixed-effects analysis of variance (ANOVA) models [12] were fitted to the natural logarithm of the tumor lesion measurements for each patient. Patient and lesion were included as random effects in the model to account for the repeated

Results

One hundred eighteen (118) lesions in 30 patients were selected for the study. Overall, lesion diameters (averaged over the three slice intervals) ranged from 6.1 mm to 80.1 mm, with a median of 18.4 mm.

Reader 2 carried out 708 measurements (118 lesions × 3 slice intervals × 2 manual measurements) and reader 1, who performed the repeat measurements, carried out 1416 measurements (708 measurements × 2). Four out of 708 (0.6%) of reader 2's manual measurements were missing. Given their small number and

Discussion

Variability in tumor VOL measurements due to CT scanner, dose, imaging acquisition and reconstruction parameters and segmentation algorithm has previously been studied mainly with a focus on pulmonary nodules detected in CT lung cancer screening programs and routine clinical care for the purpose of non-invasive diagnosis [16], [17], [18], [19]. However, in those clinical settings, nodules are usually small (<2 cm with an average size <10 mm) and image spatial resolutions are high (≤1 mm in z-axial

Conclusions

We assessed the intra- and inter-reader variability in 1D, 2D, and VOL measurements of solid tumors in the lungs, liver and lymph nodes made both manually and by CAM on CT scans reconstructed at three slice intervals of 5, 2.5 and 1.25 mm that cover a potential slice interval range of today's and future clinical trials and clinical practice. Our findings showed that, compared to 5 mm and 1.25 mm slice intervals, 2.5 mm may provide the best viewing and the most robust computation conditions for

Role of the funding source

The study sponsors provided funding for the research; they did not have any role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the paper for publication.

Conflict of interest

No conflict of interest has been declared.

Acknowledgements

This work was in part supported by Grant R01CA125143 from the National Cancer Institute (NCI) and a research grant from AstraZeneca. The content is solely the responsibility of the authors and does not necessarily represent the funding sources. The authors would like to thank Caroline Shepherd for her help in editing the manuscript.

References (24)

E.A. Eisenhauer et al.
New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)
European Journal of Cancer
(2009)
H.X. Barnhart et al.
Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets
Translational Oncology
(2009)
X. He et al.
Application of coefficient of variation in reliability-based mechanical design and manufacture
Journal of Materials Processing Technology
(2001)
S. Punnen et al.
Variability in size measurement of renal masses smaller than 4 cm on computerized tomography
Journal of Urology
(2006)
WHO handbook for reporting results of cancer treatment. Geneva (Switzerland). World Health Organization Offset...
A.B. Miller et al.
Reporting results of cancer treatment
Cancer
(1981)
P. Therasse et al.
New guidelines to evaluate response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada
Journal of the National Cancer Institute
(2000)
C.G. Moertel et al.
The effect of measuring error on the results of therapeutic trials in advanced cancer
Cancer
(1976)
P.T. Lavin et al.
Studies in variation associated with the measurement of solid tumors
Cancer
(1980)
B. Zhao et al.
Evaluating variability in tumor measurements from same-day repeat CT scans in patients with non-small cell lung cancer
Radiology
(2009)

G.R. Oxnard et al.

Variability of lung tumor measurements on repeat computed tomography (CT) scans taken within 15 minutes: implications for care and clinical research

Journal of Clinical Oncology

(2011)

B. Zhao et al.

A pilot study of volume measurement as a method of tumor response evaluation to aid biomarker development

Clinical Cancer Research

(2010)

Cited by (74)

On-cloud decision-support system for non-small cell lung cancer histology characterization from thorax computed tomography scans
2023, Computerized Medical Imaging and Graphics
Non-Small Cell Lung Cancer (NSCLC) accounts for about 85% of all lung cancers. Developing non-invasive techniques for NSCLC histology characterization may not only help clinicians to make targeted therapeutic treatments but also prevent subjects from undergoing lung biopsy, which is challenging and could lead to clinical implications. The motivation behind the study presented here is to develop an advanced on-cloud decision-support system, named LUCY, for non-small cell LUng Cancer histologY characterization directly from thorax Computed Tomography (CT) scans. This aim was pursued by selecting thorax CT scans of 182 LUng ADenocarcinoma (LUAD) and 186 LUng Squamous Cell carcinoma (LUSC) subjects from four openly accessible data collections (NSCLC-Radiomics, NSCLC-Radiogenomics, NSCLC-Radiomics-Genomics and TCGA-LUAD), in addition to the implementation and comparison of two end-to-end neural networks (the core layer of whom is a convolutional long short-term memory layer), the performance evaluation on test dataset (NSCLC-Radiomics-Genomics) from a subject-level perspective in relation to NSCLC histological subtype location and grade, and the dynamic visual interpretation of the achieved results by producing and analyzing one heatmap video for each scan. LUCY reached test Area Under the receiver operating characteristic Curve (AUC) values above 77% in all NSCLC histological subtype location and grade groups, and a best AUC value of 97% on the entire dataset reserved for testing, proving high generalizability to heterogeneous data and robustness. Thus, LUCY is a clinically-useful decision-support system able to timely, non-invasively and reliably provide visually-understandable predictions on LUAD and LUSC subjects in relation to clinically-relevant information.
Correlation of Radiomics with Treatment Response in Liver Metastases
2023, Academic Radiology
To assess differences in radiomics derived from semi-automatic segmentation of liver metastases for stable disease (SD), partial response (PR), and progressive disease (PD) based on RECIST1.1 and to assess if radiomics alone at baseline can predict response.
Our IRB-approved study included 203 women (mean age 54 ± 11 years) with metastatic liver disease from breast cancer. All patients underwent contrast abdomen-pelvis CT in the portal venous phase at two points: baseline (pre-treatment) and follow-up (between 3 and 12 months following treatment). Patients were subcategorized into three subgroups based on RECIST 1.1 criteria (Response Evaluation Criteria in Solid Tumors version 1.1): 66 with SD, 69 with PR, and 68 with PD on follow-up CT. The deidentified baseline and follow-up CT images were exported to the radiomics prototype. The prototype enabled semi-automatic segmentation of the target liver lesions for the extraction of first and high order radiomics. Statistical analyses with logistic regression and random forest classifiers were performed to differentiate SD from PD and PR.
There was no significant difference between the radiomics on the baseline and follow-up CT images of patients with SD (area under the curve (AUC): 0.3). Random forest classifier differentiated patients with PR with an AUC of 0.845. The most relevant feature was the large dependence emphasis's high and low pass wavelet filter (derived gray level dependence matrix features). Random forest classifier differentiated PD with an AUC of 0.731, with the most relevant feature being the surface-to-volume ratio. There was no difference in radiomics among the three groups at baseline; therefore, a response could not be predicted.
Radiomics of liver metastases with semi-automatic segmentation demonstrate differences between SD from PR and PD.
Semiautomatic segmentation and radiomics of metastatic liver disease demonstrate differences in SD from the PR and progressive metastatic on the baseline and follow-up CT. Despite substantial variations in the scanners, acquisition, and reconstruction parameters, radiomics had an AUC of 0.84–0.89 for differentiating stable hepatic metastases from decreasing and increasing metastatic disease.
Lung nodule diagnosis and cancer histology classification from computed tomography data by convolutional neural networks: A survey
2022, Computers in Biology and Medicine
Lung cancer is among the deadliest cancers. Besides lung nodule classification and diagnosis, developing non-invasive systems to classify lung cancer histological types/subtypes may help clinicians to make targeted treatment decisions timely, having a positive impact on patients’ comfort and survival rate. As convolutional neural networks have proven to be responsible for the significant improvement of the accuracy in lung cancer diagnosis, with this survey we intend to: show the contribution of convolutional neural networks not only in identifying malignant lung nodules but also in classifying lung cancer histological types/subtypes directly from computed tomography data; point out the strengths and weaknesses of slice-based and scan-based approaches employing convolutional neural networks; and highlight the challenges and prospective solutions to successfully apply convolutional neural networks for such classification tasks. To this aim, we conducted a comprehensive analysis of relevant Scopus-indexed studies involved in lung nodule diagnosis and cancer histology classification up to January 2022, dividing the investigation in convolutional neural network-based approaches fed with planar or volumetric computed tomography data. Despite the application of convolutional neural networks in lung nodule diagnosis and cancer histology classification is a valid strategy, some challenges raised, mainly including the lack of publicly-accessible annotated data, together with the lack of reproducibility and clinical interpretability. We believe that this survey will be helpful for future studies involved in lung nodule diagnosis and cancer histology classification prior to lung biopsy by means of convolutional neural networks.
Clinical outcome of Yttrium-90 selective internal radiation therapy (Y-90 SIRT) in unresectable hepatocellular carcinoma: Experience from a tertiary care center
2022, Liver Research
Citation Excerpt :
Multiple factors including patient's preference, clinician's decision as well as the patient and disease status might introduce selection bias to recommend Y-90 SIRT for these HCC patients. Tumor volume, which may affect clinical outcomes, was not evaluated in the present study due to technical difficulties and high inter-observer variability in such determination.30,35 Instead, we selected the largest tumor size criteria to evaluate the impact of tumor burden on survival.
Whereas Yttrium-90 selective internal radiation therapy (Y-90 SIRT) was shown to improve local tumor control in non-Asian population, the efficacy of this therapy for Asian population in real-world setting remains poorly detailed. We aimed to determine outcomes and identify predictors of response in hepatocellular carcinoma (HCC) patients treated by Y-90 SIRT.
We retrospectively enrolled 52 HCC patients receiving Y-90 SIRT at our tertiary center between 2014 and 2019. Overall survival (OS), progression free survival (PFS), and predictive factors were determined by Kaplan–Meier method and Cox-proportional hazard analysis.
Of the 52 patients (81% male, mean age 64.9 years), 71% and 29% were classified as Barcelona Clinic Liver Cancer stage C and B HCC, respectively; 63% had portal vein thrombosis, and 35% had objective tumor response defined by the modified Response Evaluation Criteria in Solid Tumors (mRECIST) criteria. OS and PFS were 11.0 and 2.4 months, respectively. Two patients were successfully down-staged and further underwent surgical resection. Multifocal lesion, alpha-fetoprotein (AFP) ≥200 ng/mL, and Eastern Cooperative Oncology Group (ECOG) score ≥1 were significantly associated with poor survival, with adjusted hazard ratio (95% confidence interval) of 7.7 (2.0–29.8), 5.4 (2.0–14.7), and 3.1 (1.0–9.6), respectively (all in P < 0.05).
Y-90 SIRT is an effective treatment for the local tumor control of HCC without serious adverse events. Single lesion, AFP level and ECOG status were predictors of response.
Hierarchical classification of early microscopic lung nodule based on cascade network
2024, Health Information Science and Systems
Top 10 Tips for Writing Materials and Methods in Radiology: A Brief Guide for Authors
2024, Radiology

View all citing articles on Scopus

¹: Current address: The Second Hospital of Hebei Medical University, Shijiazhuang, Hebei 050000, China.

²: Department of Radiology, Columbia University Medical Center, 710 West 168th Street, New York, NY 10032, USA.

³: North Middlesex University Hospital, London, Greater London N18 1QX, United Kingdom.

⁴: Flat 3, 113 Old Tiverton Road, Exeter, Devon EX4 6LD, United Kingdom. Tel.: +44 01392 21539/625 516730.

⁵: AstraZeneca, Statistics and Informatics, Clinical Information Science, 90F169-1, East Wing, Parklands, Alderley Park, Macclesfield, Cheshire SK10 4TG, United Kingdom.

⁶: AstraZeneca, Personalised Healthcare and Biomarkers, Alderley Park, Macclesfield SK10 4TG, United Kingdom.

⁷: Department of Radiology Columbia, University Medical Center, 180 Fort Washington Avenue, New York, NY 10032, USA.

⁸: AstraZeneca, Oncology Global Medicine Development - 41, 11G Mereside, Alderley Park, Macclesfield, Cheshire, SK10 4TG, United Kingdom.

View full text

Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals

Abstract

Objective

Materials and methods

Results

Conclusions

Introduction

Section snippets

Clinical imaging data

Statistical analysis

Results

Discussion

Conclusions

Role of the funding source

Conflict of interest

Acknowledgements

European Journal of Cancer

Translational Oncology

Journal of Materials Processing Technology

Journal of Urology

Reporting results of cancer treatment

Cancer

New guidelines to evaluate response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada

Journal of the National Cancer Institute

The effect of measuring error on the results of therapeutic trials in advanced cancer

Cancer

Studies in variation associated with the measurement of solid tumors

Cancer

Evaluating variability in tumor measurements from same-day repeat CT scans in patients with non-small cell lung cancer

Radiology

Variability of lung tumor measurements on repeat computed tomography (CT) scans taken within 15 minutes: implications for care and clinical research

Journal of Clinical Oncology

A pilot study of volume measurement as a method of tumor response evaluation to aid biomarker development

Clinical Cancer Research