INTRODUCTION

Colorectal cancer (CRC) remains the third most commonly diagnosed cancer and the second most common cause of cancer-related deaths in the United States.1 Colonoscopy can reduce CRC-related mortality by up to 90 %, and is cost-effective.2–6

Despite clear benefits, screening colonoscopy is underutilized, and rates of non-adherence with scheduled colonoscopy and flexible sigmoidoscopy are high, ranging from 5 to 40 %.7–12 Consistent identification of patients at risk for non-adherence with scheduled colonoscopy is a critical first step towards improving adherence. A number of studies have identified predictors of non-adherence with colonoscopy scheduling and completion.9–11,13–17 However, no study has described a validated multivariable model for predicting non-adherence with scheduled colonoscopy. Furthermore, no automated method to assess a patient’s prior record of non-adherence has been developed, an essential step for any implementation strategy involving large patient populations.

Natural Language Processing (NLP) systems—which enable the accurate extraction and interpretation of information from free text electronic documents—may facilitate the incorporation of clinically important predictive factors that are not readily available in structured databases. NLP has been used to facilitate numerous assessments of health care quality and safety.18–22

This project had two major aims: 1) to develop an automated method utilizing NLP that evaluates patients’ prior non-adherence with health care services; 2) to use this adherence metric and readily available clinical, demographic, and scheduling data to develop and validate a prediction model that accurately identifies individuals at high risk for non-adherence with scheduled colonoscopy.

METHODS

Study Design

Our study included a discovery phase and a validation phase. The discovery phase consisted of a matched case–control analysis. Using results from this analysis, we built a logistic regression model for predicting non-adherence with scheduled diagnostic or screening colonoscopy. We then validated the model’s ability to predict adherence with scheduled colonoscopy in a distinct, randomly selected patient cohort.

Per standard practice at this institution, all patients scheduled to undergo outpatient colonoscopy received a phone call reminder about their upcoming colonoscopy approximately 1 week prior. Patients who scheduled their appointment within 7 days of the procedure date did not always receive this reminder.

This project was designed and carried out from 2009 to 2011 at an academic medical center (AMC) in the Northeastern United States and was approved by this AMC’s Institutional Review Board (IRB).

Patients

We used IDX, the gastroenterology department’s ambulatory scheduling database (IDX systems, Burlington, VT; USA), to identify all patients with scheduled outpatient colonoscopies between 1 December 2009 and 15 October 2010 (“index colonoscopy”). The discovery cohort included 110 patients from this list who were non-adherent with scheduled diagnostic or screening colonoscopy (“cases”) and 220 adherent patients (“controls”), matched for procedure date. The validation cohort consisted of 1,200 randomly selected individuals with scheduled outpatient colonoscopies at our institution between 1 December 2010 and 31 November 2011 (“index colonoscopy”). Exclusion criteria included: 1) cancellation of scheduled colonoscopy; 2) receipt of inpatient colonoscopy in place of outpatient colonoscopy; 3) matching of a control to two cases (in such instances, the control was not double-counted); 4) primary residence outside of the United States; and 5) incomplete chart review.

The Non-Adherence Ratio and Non-Adherence Query

The Queriable Patient Inference Dossier,23 or QPID, is a natural language search engine developed at our institution. QPID allows the development and deployment of regular-expression based NLP queries, which allows users to search free text in a patient’s Electronic Medical Record (EMR) for specific words, phrases, data, and combinations thereof. Using QPID, we developed a unique chart review variable, which we named the Non-Adherence Ratio (NAR).24 The NAR includes an automated assessment of a patient’s prior non-adherence with health care services, called the Non-Adherence Query (NAQ). The NAQ searches a patient’s EMR for phrases that clinicians commonly use to document non-adherence, including “no show,” “did not present,” “failed to attend,” and “missed appointment.” The NAQ also looks for pre-specified combinations of these phrases. The development and validation of the NAQ and NAR are described in eMethods 1 (see electronic supplementary material).

Discovery Cohort

Data Collection

Chart reviews were performed by a research assistant and two physicians who were blinded to case/control status. Chart reviewers used standardized methods and definitions. Chart review variables were chosen based on prior evidence of an association with colonoscopy non-adherence, a plausible association with colonoscopy non-adherence, and ease of consistent, objective, measurement.8–10,25,26 Chart review variables included: age, gender, insurance status (Private, Medicare, Mass Health, Medicaid, or None), education level, personal history of bright red blood per rectum (BRBPR) or positive fecal occult blood test (FOBT), history of CRC in a first-degree relative, history of colon polyps, and history of psychiatric illness. History of psychiatric illness was defined as documentation in the medical record of a mood, anxiety, and/or psychotic disorder. Education level was stratified into “no more than high school” education and “college or graduate school” education (e.g., anyone with at least some college education). Data about the number of missed endoscopies prior to the index colonoscopy, the index colonoscopy booking date, and the procedure date were obtained from IDX. A missed endoscopy was defined as non-attendance at a scheduled outpatient esophagogastroduodenoscopy, endoscopic ultrasound, endoscopic retrograde cholangiopancreatography, flexible sigmoidoscopy, or colonoscopy. Chart review data were collected and managed using REDCap electronic data capture tools (Vanderbilt University; Nashville, TN).27

Statistical Analysis

Univariate and unconditional multivariate logistic regression analyses both with and without the NAR were performed using SAS Analytics Software (version 9.3, SAS Institute Inc.; Cary, North Carolina). Model building was guided by clinical judgment and results of univariate and multivariate regression analyses of potential predictors. The non-adherence model’s output, called the “Non-Adherence Score,” was a decimal, which, when converted to a percentage, represented the predicted probability of non-adherence with scheduled colonoscopy. With α = 0.05 and β = 0.20, we calculated that we would need 107 cases and 214 controls to correctly identify a variable associated with a 100 % difference in the odds of colonoscopy adherence. When setting our validation cohort’s size, we attempted to balance the benefits of a large sample size with the significant time commitment required to perform chart reviews. We considered a two-tailed p value of 0.05 or less to be statistically significant.

Validation Phase

Data Collection

Chart reviews were performed for each patient in the validation cohort to gather information for demographic and clinical variables included in the discovery analysis (listed above). Data on booking date, scheduling date, and number of missed endoscopies were obtained from IDX. NAR and wait time were calculated for each patient.

Statistical Analysis

We compared no-shows and attendees in the discovery and validation cohorts using chi-square tests for categorical variables and t-tests for differences of means for continuous variables. Positive predictive value (PPV) and negative predictive value (NPV) for the model were estimated (Note: PPV refers to the likelihood of non-adherence with colonoscopy, while NPV refers to the likelihood of adherence). Area under the receiver operator characteristic (AUC) curve and net reclassification index were used to compare models with and without NAR. Each patient’s Non-Adherence Score was assessed using the non-adherence model designed in the discovery cohort (with discovery cohort regression coefficients and intercept). Patients whose non-adherence scores were higher than a threshold value (identified below) were deemed to be at high risk for non-adherence. Sensitivity analyses assessed how the population no-show rate influenced the model’s PPV and NPV, and the model’s performance in the validation cohort after excluding patients with a prior history of CRC and/or endoscopy.

RESULTS

Characteristics of Discovery Cohort

The final discovery cohort included 318 patients: 107 cases and 211 controls (Table 1). Twelve patients were excluded from the discovery cohort analysis (eFigure 1a, see electronic supplementary material). Cases were significantly more likely than controls to have public health insurance (OR 3.11, p < 0.001 for Medicare vs. Private; OR 3.66, p < 0.001 for Mass Health, Medicaid, or None vs. Private), a history of BRBPR or a (+) FOBT (OR 1.86, p = 0.03), and a history of psychiatric illness (OR 2.25, p < 0.001) (Table 2). Forty-one percent and 66.8 % of cases and controls attended college, respectively (OR 0.35 for college attendance, p < 0.001). The mean NAR for no-shows was 7.2 %, compared with 2.0 % for controls (p < 0.001) (Table 1). A 10 % increase in the NAR was associated with a 297 % increase in non-attendance (p < 0.001) (Table 2). Missing one prior endoscopy appointment was associated with a 160 % increase in no-show risk (p < 0.001).

Table 1. Demographic and Clinical Characteristics of Patients in the Discovery and Validation Cohorts
Full size table
Table 2. Results of Univariate and Multivariate Logistic Regression Analysis of Predictors of Colonoscopy Adherence in the Discovery Cohort
Full size table

Characteristics of Validation Cohort

The final validation cohort included 1114 patients: 89 no-shows and 1,025 adherent patients. Eighty-six patients were excluded from the initial 1,200 patient validation cohort (eFigure 1b, see electronic supplementary material). No-shows were more likely than attendees to have public insurance (Medicare insurance: 24.7 % vs. 21.9 %; Mass health or Medicaid: 23.6 % vs. 9.9 %; Table 1). Approximately 50 % of no-shows and 26 % of attendees had no more than a high school education. A prior history of colon polyps was noted in 25.8 and 40.7 % of no-shows and attendees, respectively, while 19.1 % of no-shows and 29.8 % of attendees had a history of BRBPR or (+) FOBT. The mean NAR was 5.9 and 1.8 % for non-adherent and adherent patients, respectively.

Comparison of Discovery and Validation Cohorts

No-shows in the validation cohort were more likely to have a history of polyps, and less likely to have missed a prior endoscopy (OR 0.54, p < 0.001) (Table 1). Moreover, attendees in the validation cohort were significantly more likely to have attended college, and less likely to have a history of polyps (OR 3.57, p < 0.001), BRBPR or a (+) FOBT (OR 1.93, p < 0.001), CRC in a first degree relative (OR 2.39, p < 0.001), or psychiatric illness (OR 1.44, p < 0.05) (Table 1).

Development of Non-Adherence Model

Five variables were significantly associated with colonoscopy non-adherence in multivariate regression analysis: 1) male gender (OR 1.81, p = 0.04); 2) history of psychiatric illness (OR 1.90, p = 0.03), 3) NAR (OR 2.79 for 10 % increase in NAR, p < 0.001); 4) wait time in months (OR 1.28 for each month, p = 0.006); and 5) number of prior missed endoscopies (OR 1.78 per missed endoscopy, p = 0.03) (Table 2). The final non-adherence model included these five variables and education level (Table 3). Education level was added to this model based on clinical judgment, and observed associations between this variable, CRC screening, and colonoscopy adherence.28–30 The final model’s AUC in the discovery cohort was 75.5 % (95 % CI 70–80.2 %) (Fig. 1a). Logistic regression coefficients for these variables, the model coefficient, and the final model equation are presented in Table 3. The model intercept yields a non-adherence score equal to the predicted probability of non-adherence in a population with the same prevalence as our discovery cohort, but the absolute magnitude of the non-adherence score does not affect its operating characteristics.

Table 3. Coefficient Estimates, Standard Errors, and Odds Ratios for Final Non-Adherence Model Variables in Discovery Cohort
Full size table
Figure 1.
figure 1

a Receiver Operating Characteristic (ROC) curves for prediction of colonoscopy adherence in discovery cohort using non-adherence prediction model. b Receiver Operating Characteristic (ROC) curves for prediction of colonoscopy adherence in validation cohort using non-adherence prediction model. In a–b, the ROC curve labeled “With NAR” represents the ROC for the full, six variable non-adherence model while the ROC curve labeled “Without NAR” depicts the ROC for the non-adherence model after removal of the NAR. NAR = Non-Adherence Ratio.

Full size image

Validation of Non-Adherence Model

The non-adherence model’s AUC in the validation cohort was 70.2 % (CI: 64.4–76.1 %) (Fig. 1b). At a Non-Adherence Score of 0.46, the model’s sensitivity (Sn) and specificity (Sp) were 33 and 92 %, respectively. At this threshold, the model correctly classified 29 of 89 no-shows, and 940 of 1,025 attendees, yielding a PPV and NPV of 26 and 94 %, respectively (Table 4). Assuming a constant Sn of 33 % and Sp of 92 %, an increase in the population non-adherence rate from 8 % (the rate in the validation cohort) to 20 % resulted in a PPV of 50 %, and an NPV of 85 % (eTable 1, see electronic supplementary material). Removing the NAR from the model reduced the AUC from 70.2 to 64.3 % (Difference = 5.9 %, p < 0.001). In sensitivity analyses, the model’s AUC was 70 % (95 % CI: 64–76 %) in validation cohort patients with no prior history of CRC (N = 1084), and 73 % (95 % CI: 65–80 %) in validation cohort patients with no prior history of endoscopy or CRC (N = 487).

Table 4. Predictive Accuracy of Non-Adherence Model in the Validation Cohort at Different Non-Adherence Scores
Full size table

In multivariable regression analyses in the validation cohort, three model variables significantly predicted no shows: education level (OR 0.55 for college education, 95 % CI 0.33–0.91), NAR (OR 2.70 per 10 % increase, 95 % CI 1.85–3.93), and number of missed endoscopies (OR 4.04 per endoscopy, 95 % CI: 1.11–14.7) (eTable 2, see electronic supplementary material). Having Masshealth, Medicaid, or no insurance predicted an increased risk of non-adherence, and a history of colon polyps significantly reduced the likelihood of non-adherence.

In eTable 3 (see electronic supplementary material), we describe four hypothetical patients with a predicted Non-Adherence Score of at least 0.46 in order to highlight “real life” examples of individuals who would be considered to be at high risk for non-adherence using this model.

DISCUSSION

We developed and validated a six variable model for predicting non-adherence with scheduled colonoscopy. Two of the strongest predictors of non-attendance were the NAR and prior non-attendance with scheduled endoscopy. The NAR is a novel parameter that we designed and measured using an NLP system. Calculating the NAR through this automated and systematic approach can help to prospectively identify individuals who are at high-risk for colonoscopy non-adherence. Removing the NAR from the model significantly reduced its validated AUC.

These findings expand upon previous work by Turner et al., who identified an association between non-attendance with lower endoscopy and health care appointment non-adherence.9 A history of clinic appointment non-attendance has also been shown to predict future clinic appointment non-attendance, hospital readmission for congestive heart failure, and HIV viral load in patients on antiviral treatment.18,31,32 Thus, appointment-keeping behavior is an important predictor of adherence with many health care services.9

Measuring adherence with scheduled appointments across different departments requires access to a central scheduling database, which many clinicians and clinical support staff do not have. Our study demonstrates that NLP systems can enable real-time evaluation of a patient’s history of health service adherence using readily available unstructured data. We are not aware of any studies investigating a role for NLP in measuring appointment adherence.19–21,32–40 Thus, our work highlights a novel method using NLP for this purpose.

Although others have identified a relationship between appointment keeping behavior and endoscopy attendance, we demonstrate that such information can be incorporated into a non-adherence prediction model—a critical step towards improving health care quality. The NAR can be calculated quickly in real time by QPID using basic regular expression search technology that has been widely available since the 1960s and is common to virtually all existing NLP systems. So, this approach can be generalized.41–43 More broadly, this work highlights one way in which health information technology may facilitate process and quality improvement work.44,45

This model’s low sensitivity, high specificity, and high NPV make it well suited to help medical centers more effectively deploy resources to improve colonoscopy adherence. For example, “high-risk” patients, as identified by the model, could be assigned to patient navigators, who have been shown to significantly improve colonoscopy adherence.46–48 Moreover, while sensitivity and specificity are inherent characteristics of a model, its PPV and NPV depend upon the population-level incidence of the outcome of interest. Our model’s low PPV (25 %) may reflect the low baseline non-adherence rate in our population (8 %). Using a 20 % baseline non-adherence rate, as reported in previous studies, the model’s estimated PPV improves significantly (50 %), with only a modest reduction in the NPV.7,9,10,15,49 Moreover, the model is equally effective at predicting non-adherence by patients with no prior history of endoscopy, who may be at higher risk of colonoscopy non-adherence.9

Changes in the model’s NPV and PPV have implications for the allocation of resources, like patient navigators, that may improve colonoscopy adherence. For example, a higher model PPV increases the likelihood that a navigator will be appropriately assigned to a patient who otherwise would have no-showed. However, if the NPV drops, the model may also misclassify high-risk patients as low in risk. Nonetheless, in systems with limited resources for improving colonoscopy adherence, focusing on patients with higher non-adherence scores, which correspond to higher PPVs, will maximize effective resource utilization.

The differences between the discovery and validation cohorts may help to explain why only three model variables significantly predicted non-adherence in the validation sample. These differences likely stem from how the cohorts were constructed (i.e., randomly chosen cases with matched controls for the discovery sample and randomly selected cohort of all-comers for the validation sample), rather than from systematic differences between the populations from which they were derived. Indeed, the endoscopy suite’s patient population, colonoscopy referral and scheduling systems, reminder policies, and indications for colonoscopy did not change from 2009 to 2011. While chart reviewers were trained to use a consistent coding system, we cannot rule out minor variation between reviewers in the operationalization of certain variables.

Our work adds to existing literature about colonoscopy non-adherence in other ways. Higher education level has previously been associated with higher rates of CRC screening.50 Associations between endoscopy non-adherence, male gender, psychiatric illness, and appointment wait time have all been reported previously.17,22,26

This study has several limitations. We did not investigate certain variables that may be associated with non-attendance, including socio-economic status, ethnicity, indication for colonoscopy, colonoscopy referral source, or patient-specific barriers to attendance.10,14,16,25,32 Furthermore, because few patients in this study were uninsured, on Medicaid, or lacked college education, our findings may not extend to these populations. Likewise, because the NAR relies on data from prior interactions with a health system, it may have limited utility for predicting adherence for patients who are new to a health system, or rarely see a physician. In addition, our findings may not be generalizable to other health systems. Nonetheless, the significant differences between our discovery and validation cohorts, and our sensitivity analyses, provide some evidence that our model will perform adequately in other populations. Finally, our discovery analysis may not have been powered to detect certain clinically relevant predictors of adherence, or to precisely bound odds ratios for predictors that were identified. Further research is necessary to determine prospectively if implementation of this non-adherence model improves adherence with scheduled colonoscopy.

CONCLUSION

We developed and validated the first published model for predicting non-adherence with scheduled colonoscopy using data that are readily available in patient medical records and easily extractable with an NLP system. Our work confirms that a patient’s prior history of health care service non-adherence is a strong predictor of adherence with scheduled colonoscopy, and offers a novel tool that utilizes an NLP system to quickly and reliably assess a patient’s history of health care service adherence.