A framework for the estimation and reduction of hospital readmission penalties using predictive analytics

Many machine learning algorithms have been used for the creation of HRPS. Logistic regression (LR) is a regression model whose dependent variable is categorical. Boulding et al. [15] have used LR to predict readmission using patient satisfaction as independent variables. Greenwald et al. [16] also successfully used LR for predicting readmission using physical function, cognitive status, and psychosocial support. Support vector machines (SVM) are another binary linear classifier which attempt to maximize the margins of classification. Research by Braga et al. used SVM to predict readmission for intensive care patients, while Sushmita et al. have used SVM for prediction of all-cause readmission [9, 17]. Decision trees have also been successfully used for the prediction of patient readmission [18]. Naïve Bayes (NB) classifiers are simple probabilistic classifiers which use Bayes’ theorem to classify instances. NB assumes conditional independence between features. Although this assumption is often not true, NB remains a useful classifier and is often used in text classification [19]. Researchers using unstructured text as a data source have seen good results using NB for predicting readmission [10, 20].

HRPS utilizing machine learning often produce a binary classification. Algorithms use training data to create a model which will classify new instances as either readmission or non-readmission, based on the evidence provided. This may be problematic, as patients with a high risk of readmission may be assigned the same classification prediction as those whose readmission risk is considerably lower. If resources are available to send a home healthcare professional to a single patient, but there are two patients classified as potential readmissions, additional information is required. However, binary classification systems often do not inherently offer additional information such as probability of readmission. This shortcoming limits the use of a HRPS within a CDSS.

The most common evaluation metric of HRPS performance is c-statistic [6]. C-statistic is defined as the area under a receiver operating characteristic (ROC) curve. ROC curves are graphical plots that illustrate the performance of a binary classifier as its discrimination threshold is varied. The plot compares performance of the true positive rate (TPR) and false positive rate (FPR) for various thresholds of classification. A survey by Kansagara et al. [6] reviewed many HRPS using c-statistic as the primary performance metric and few perform better than 0.7. Small data samples are often cited as a primary reason for poor model performance. However, models created using The United Kingdom’s National Health Service (NHS), using millions of patients, [21] performed similarly to a model created using 1029 patients [22].

The current use of c-statistic as an evaluation metric for HRPS has shown to produce inconsistent results. Researchers have argued that c-statistic is often used inappropriately to measure the performance of classification systems [23]. Assumptions about misclassification cost and uniform distribution within classes may not be true and comparison of classification systems using c-statistic may produce incoherent results [24]. Although cost as a performance metric may offer a clear alternative to c-statistic, its use in HRPS has been limited.

Data sources for readmission models can be categorized into structured and unstructured data [25]. Structured data is generally stored in a relational database and contains information such as demographics and ICD-9 or ICD-10 codes. Current systems described by Kansagara et al. [6] commonly use structured data as the primary data source. The main advantage to using structured data is that once extracted from a system, little work needs to be done to convert the data into a form usable by supervised machine learning algorithms. Unstructured data is often represented in the form of clinical notes or discharge summaries. These are often written in natural language such as English and allow the medical professional to completely describe their thoughts regarding patient status. Systems using structured data may potentially lose information if there is no predefined input for an observation. However, unstructured data is often difficult to convert to a format usable by supervised machine learning algorithms. The field of natural language processing (NLP) can often be of assistance in structuring natural language to a usable format. Although unstructured data as a primary data source for hospital readmission systems has historically seen little adoption, advances in clinical NLP have helped increase adoption among researchers [10, 20, 26‐29].

Apache cTAKES is an open source Clinical NLP tool created and maintained by the Mayo Clinic [30]. Apache cTAKES annotates clinical notes using domain specific dictionaries and clinically trained NLP models. Core to cTAKES is the Unified Medical Language System (UMLS), a set of dictionaries and vocabularies maintained by the National Library of Medicine (NLM) [31]. UMLS allows cTAKES to annotate notes using vocabularies assembled by domain experts and enables cTAKES to separate clinical and non-clinical terms. Many diseases, symptoms, and medications have variations in spelling, abbreviations, and usage. UMLS provides a normalization ID known as the Concept ID (CID) which allows terms to be reduced to their base components. Many variants of the same feature will often confuse machine learning models and merging terms that have the same semantic meaning strengthens the model and increases performance. Figure 1 illustrates many lexical variants of asthma normalized to a single CID.

The data for this research consists of 1248 discharge summaries containing chronic obstructive pulmonary disease (COPD) as a primary diagnosis. Unstructured data was chosen as a primary data source, due to the ease of extraction from Electronic Health Record (EHR) systems. Discharge summaries and classification labels were available as Microsoft Word documents. Apache cTAKES was used to annotate discharge summaries and normalize medications, diseases, and symptoms to a UMLS CID. Figure 2 illustrates the most frequently occurring terms in this dataset. A total of 1018 UMLS normalized terms have been identified and extracted from this dataset using Apache cTAKES.

Annotations are converted to a bag-of-words representation, where each annotation in the corpus is a feature and the presence of an annotation in an instance is the value of that feature. Given the discharge summaries. The patient has asthma and diabetes and Jane Doe was found to have COPD and asthma, Table 1 illustrates the bag-of-words representation using medical terms discovered by cTAKES. In this example, the classification label is true for the first instance (the patient was readmitted within 30 days) and false for the second instance (patient was not readmitted within 30 days).

The bag-of-words representation of unstructured text is inherently unable to detect the presence of a missing value. A missing value in the context of a clinical note would imply a medical professional omitted text (either intentionally or intentionally). The value of a term which is present is be represented as the number one and the value of a term which is not present is represented as the number zero. The omission of a term in unstructured text is assumed to be purposeful and missing values, whether intentional or unintentional, are represented by the number zero.

Many HRPS use binary classification labels when predicting readmission [9, 14, 17, 32‐35]. However, staffing resources are often limited and a probability of readmission is potentially more useful. Sorting a group of patients by readmission probability allows patients with the highest readmission probability to be allotted the greatest number of resources. Though many supervised machine learning algorithms have the ability to coerce classification distribution, the NB machine learning algorithm naturally produces posterior probabilities without coercion [19]. Therefore, NB can be used to sort patients by likelihood of readmission. This is useful when resources are limited and an optimal subset of patients must be selected to minimize penalties.

Additional classifiers were considered, but early experimentation often resulted in models which classified all instances as positive. This may be due to the high cost of readmission or resulting coercion of posterior probabilities. Since these classifiers offered no useful predictive abilities, NB was chosen as the primary classifier.

CMS has made available the formulas for which HRRP penalties are calculated [2]. The cost consists of two primary components: Diagnostic Related Group (DRG) amount and excess readmissions ratio (ERR). DRG is calculated using many variables, including case mix index, labor share, wage index, non-labor share, cost of living adjustments, technology payments, and total number of Medicare cases. Few of these variables can be affected administratively and for modeling purposes considered unchangeable. ERR is defined as followsExpected readmissions rate is the target readmissions rate which CMS has assigned a given hospital. Using national readmission statistics and regression models, this is the rate which the average hospital would obtain, given a hospital’s patient demographics and disease distribution. Predicted readmissions rate is related to the actual readmissions rate obtained by a hospital. A risk adjustment is performed and predicted readmissions rate is the product of the actual rate and risk adjustment. Expected readmissions rate is considered constant as it is set by CMS. However, predicted readmissions rate can be lowered by lowering the actual readmission rate. The final penalty for a given DRG is calculated by CMS as follows

Table 2 defines the variables used in cost modeling.

ERR can be expanded and risk adjustment factored out. Risk adjustment is the fraction of actual readmissions between [0,1] for which a hospital is responsible.

Each hospital has a set of patients for which the ground truth readmissions status is known. This is due either to 30-days having lapsed or the patient having experienced readmission. These patients are denoted P and R respectively. By definition, \( \hat{\rho } = \frac{R}{P} \) and can be expanded in our equation.

When a new patient is added to our model and the ground truth readmission status is known to be a readmission, R increases by 1, P increases by 1, and the DRG increases by the cost of that patient. The expected rate ρ remains unaffected as this is set by CMS. The total cost is then calculated as follows.

As new patients enter the hospital, the ground truth readmission label will not be known for up to 30 days. However, an estimate can be calculated using the probability of readmission. Therefore, a new patient entering the current cost estimation model can be assigned a probability of readmission p _r .

Additional patients may be admitted before ground truth readmission status is known for previous patients. As patients enter the hospital, the model increases by N patients and the sum of readmission probabilities \( \sum\nolimits_{i = 1}^{N} {p_{r} } \). The final cost estimation for HRRP penalties is below:

ERR may be useful in some instances and is defined by:

The net cost of patient intervention must include the possibility that intervention may not work. Internal statistics regarding the effectiveness of patient intervention may be collected and is represented by p _s . Assuming N patients, of which we intervene for N _s , we can define \( ERR^{{(N_{s} )}} \) as follows:where (1 − p _s )N _s represents the probable number of readmissions that will still occur even though a home healthcare professional has been assigned. N _s can be increased iteratively until either ERR ≤ 0 or there are no additional patients to analyze. If ERR ≤ 0, no additional HRRP penalties are incurred and resources for preventing readmission can be diverted elsewhere. Figure 3 describes the process for finding the optimal number of patients in which to select for intervention.

Cost estimates may additionally incorporate the average cost of a home healthcare professional, represented as \( \bar{c}_{t} \). Assuming N _s interventions, the current total cost of intervention will be the following:This represents the total cost of intervention given N patients and N _s interventions. Since the patients have been sorted by decreasing order of readmission probability and number of patients in analysis is constant, it is possible to iteratively increase the number of patients to intervene until a minimum cost is achieved. When \( ERR^{{(N_{s} )}} \le 0 \), no additional cost savings are possible as CMS does not refund medical facilities for exceeding expected rates. No penalties will be incurred, but the cost of sending a home healthcare professional remains. If the cost of \( \bar{c}_{t} \) is very high or set of patients very unlikely to require readmission, a minimum cost where \( ERR^{{(N_{s} )}} > 0 \) is possible. This scenario indicates that patients are unlikely to need readmission and cost of intervention high. In this case, it is less expensive to pay HRRP penalties than to intervene. Due to the generally high cost of HRRP penalties, this scenario is rare.

An overview of the framework is shown in Fig. 4. Discharge summaries are gathered and sent to Apache cTAKES for annotation. Once annotated, features are extracted using the bag-of-words representation. This representation allows discharge summaries to be used with the NB machine learning algorithm, which predicts the probability of readmission. These probabilities are then sent to the MinCost algorithm which attempts to identify the costliest patients.