Resource-efficient fast prediction in healthcare data analytics: A pruned Random Forest regression approach

Random Forest (RF) has proven its effectiveness as a classification and a regression method in a variety of applications [10]. In [11], a new method termed CLUB-DRF was introduced to select diverse decision trees drawn from groups of similar trees (i.e. clusters of trees), to form a pruned Random Forest ensemble that is much smaller than the initial and traditional RF ensemble [7], and yet, performs at least as good as the original ensemble. Since a well established principle in ensemble classification and regression is that ensembles tend to perform better when the individual classifiers in the ensemble exhibit a high level of diversity [1, 8, 19, 31], we have adopted data clustering [17] to inject more diversity into an RF, and to prune the RF, leading to a faster inference. The premise is that grouping of similar classifiers in clusters according to their classification patterns, and then choosing a representative classifier (or more) of each cluster can result in a pruned and more diversified ensemble.

Since diversity can lead to better performance as previously described, in a nutshell, there are two levels of diversity already applied in an RF. The first level is when each decision tree is constructed using sampling with replacement from the training data. The samples are likely to have some diversity among each other as they were drawn at random. The second level is achieved by randomisation which is applied when selecting the best node to split on. The ultimate objective of our new method is to add a third level of diversity by injecting more diversity in an RF using clustering as described in Sect. 3.3.

In [11], CLUB-DRF has proved its effectiveness. With at least 92% or above pruning level, while retaining or outperforming the RF accuracy (before pruning). Since in predictive healthcare data analytics low accuracy can lead to improper diagnosis, which in turn can be both fatal and devastating, in this paper, we aim to target one domain only, namely, the health domain. Hence, we apply our proposed method CLUB-DRF on three medical data sets of three different diseases. It is worth noting that in this paper, it is the first time to apply CLUB-DRF for regression. Thus, the method has been modified by replacing a clustering technique (k-modes) that is used for categorical data by another that is tailored to operate on numerical data (k-means). This is due to the replacement of the type of output between the two approaches (i.e. categorical for classification, and numerical for regression).

This paper is organised as follows. Section 2 presents related work in the domain of healthcare data analytics. Overview and detailed description of our proposed new method are given in Sect. 3. Experiments and results are presented in Sect. 4. The paper is finally concluded with a summary and pointers to future work in Sect. 6.

Healthcare data analytics, also known as clinical data analytics, is the gathering and interpretation of data from a variety of sources (e.g. the electronic health record, billing claims, cost reports, and patient satisfaction surveys) to help organisations improve the quality of care, lower the cost of care, and enhance the patient experience. According to [26], several researchers have conducted studies which proved that, by utilising healthcare data analytics technologies, they were able to reduce mortality rates, healthcare costs, and medical complications at various hospitals. Moreover, healthcare analytics has the potential to reduce costs of treatment, avoid preventable diseases, predict outbreaks of epidemics, avoid preventable deaths, and improve the quality of life in general [28]. According to a recent Research and Markets report [27], healthcare analytics is expected to rise and reach a $34.27 billion industry by the end of 2022. To improve healthcare as described above, healthcare analytics became a hot area of interest and attracted attention from diverse disciplines such as databases, data mining, information retrieval, image processing, medical researchers, and healthcare practitioners [26]. In this paper, we will exploit one of data mining’s tasks known as regression to improve the performance of the traditional RF on medical data sets, both in terms of accuracy and regression speed.

The prediction of chronic kidney diseases by using Decision Tree (C4.5) algorithms was investigated by [5]. In terms of accuracy and execution time, the classifier used proved its performance. A new method that is based on linear regression to correct Partial Volume Effects (PVE) in Arterial Spin Labeling (ASL) MRI was developed by [3]. An online Stochastic Gradient Descent (SGD) algorithm with logistic regression is implemented by [23] using Apache Mahout to develop the best scalable diagnosis model. The proposed prediction model achieved 81.99% accuracy for training sample and 81.52% accuracy for testing sample.

The prediction of three diseases, namely, leukemia, lung cancer, and heart disease was investigated by [24]. To do this, the researchers used three different classifiers: Naive Bayes, C4.5, and Random Forest. According to the researchers, the proposed method has better accuracy, precision, recall and Fmeasure.

Using stepwise logistic regression models, Higdon et al. [13] developed a model based on three different counts of medications: outpatient, inpatient drug classes, and individual inpatient drug names. The model was used to rank the patient population for medical complexity. Thanks to this model, based on the number and type of medications, simple admission screens for predicting the complexity of patients were implemented.

The approach offered in this paper aims at applying a pruned RF regression approach on medical data sets of different diseases. In the pruned RF ensemble produced by our approach, clustering is used as the main diversity approach for selecting diverse trees that form the pruned RF ensemble. The following section provides greater insight about the proposed approach.

In this section, an experimental study of our approach on three medical regression data sets of three different diseases will be presented. These diseases are parkinson’s, diabetes, and breast-cancer. Basic information about these data sets are given in Table 1. The first data set was obtained from the University of Waikato Repository, and the last two from the University of California, Irvine (UCI) Machine Learning Repository. The experiments were conducted on a laptop running Windows 7 Enterprise, with a dual processor of 2.60 Ghz and 8 GB of RAM.

The diabetes data set contains the necessary attributes to predict the dependence of the level of serum C-peptide on the various other factors in order to understand the patterns of residual insulin secretion. In a scale of 1–4, the attributes in the the breast-cancer data set are used to predict the type of the breast-cancer. Using a scale range of bio-medical voice measurements from 42 people with early-stage Parkinson’s disease, the attributes in the parkinson’s data set are used to predict the clinician’s Parkinson’s disease symptom score on the Unified Parkinson’s Disease Rating Scale (UPDRS) scale.

To use the holdout testing method, which is the simplest type of cross validation, each data set was divided into sets: training and testing. Two thirds (66%) were reserved for training and the rest (34%) for testing. The selection of the two sets was done randomly using uniform distribution, where each instance has the same probability of being selected. The size of the Parent Random Forest (refer back to Fig. 1), which we called parentRF, was 500 trees; a typical setting for Random Forest [12]. This setting was chosen for two main reasons. First, the more trees we have, the more diverse ones we can get. Secondly, the more trees there are, the more unlikely the problem of empty clusters [25] does not surface.

The CLUB-DRF algorithm outlined in Algorithm 1 was implemented using the Python programming language utilising its machine learning library Scikit-learn. It was run 10 times on each data set where a new parentRF was created in each run. The average of the 10 runs for each resulted CLUB-DRF was calculated to produce the average for a variety of metrics including Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R Squared.

Results of our experiments on the three data sets diabetes, breast-cancer, and parkinson’s are displayed in the Tables 2, 3 and 4 respectively. In each of these tables, the second row displays performance metrics of the parentRF. Following that, performance metrics of CLUB-DRFs of sizes in the range 5–50, reflecting extreme pruning levels from 99 to 90% respectively, as previously described in Sect. 3.3, are listed. To show the effectiveness and efficiency of CLUB-DRF over the traditional RF, we also listed performance metrics for traditional RFs in the same range that were derived from the parentRF. Each performance metric is represented as a couple of numbers each enclosed between parentheses, where the first number is for CLUB-DRF, and the second for a traditional RF. If the first number for CLUB-DRF is highlighted in boldface, it means that CLUB-DRF outperformed its counterpart traditional RF of the same size.

Table 5 shows the inference time per instance (ITPI) in microseconds for the three data sets diabetes, breast-cancer, and parkinson’s. This refers to the time needed to predict all the instances in the testing data set divided by the number of instances. As in the previous tables, we have highlighted in boldface the first entry when CLUB-DRF achieved less ITPI than its counterpart traditional RF.

To test our CLUB-DRF method, the training and testing data sets should be prepared, as aforementioned. The training data set is fed as input to CLUB-DRF which constructs an initial RF of 500 trees as was outlined in Sect. 3.2. Pruned RF of sizes 5–50 are then produced by CLUB-DRF which are then used to predict the value of the target feature for each instance in the data set. Since CLUB-DRF generates several pruned forests (with different sizes), and reports the performance of each forest, it is recommended to pick the forest that has achieved the highest performance in terms of accuracy for deployment.

As depicted in Tables 2, 3 and 4, it is obvious from the key performance indicator MAE (Mean Absolute Error) that CLUB-DRF, regardless of its size, outperformed not only the parentRF, but also a traditional RF of the same size. Furthermore, our approach seems to be insensitive to the dimension and size of the data sets. As outlined in Table 1, the 3 data sets have different dimensions and sizes and yet, CLUB-DRF performed consistently well as was demonstrated in Tables 2, 3 and 4. An interesting observation in these tables is that CLUB-DRF performs even better as the number of dimensions and the data set size increase.

As for the he inference time per instance (ITPI) in Table 5, it is easy to see that in almost all cases, CLUB-DRF has outperformed the traditional RF. The interesting thing is that, not only CLUB-DRF performed better accuracy-wise as was demonstrated in Tables 2, 3 and 4, but it also achieved better ITPI as was demonstrated in Table 5.

The results show that R Squared reached 0.9 for CLUB-DRF in both the breast-cancer and parkinson’s data sets. This implies that 90% of the variance in the data has been explained. Contrasting with traditional RF, the best performing RF for the breast-cancer measured in R Squared is 0.72, and for the parkinson’s data set is 0.69. This clearly shows the superiority of CLUB-DRF in explaining the variance in the data. We argue that the diversity created through the clustering process is the main reason for this performance boost up. However, for the diabetes data set, RF seems not to be able to explain the variance in the data. For CLUB-DRF with the number of trees greater than or equal to 30, the model is able to explain some of the variance (6% when the number of trees is 40). It is worth noting that there are two observations in this case: (1) the diabetes data set is small (only 43 instances); and (2) CLUB-DRF has shown positive values for R Squared when the traditional RF was not able to show any case of a positive value.

In predictive healthcare data analytics applications, it is imperative for such applications to be as accurate as possible to minimise misdiagnosis which can be fatal sometimes. To ensure proper diagnosis of diseases, we presented in this paper a Random Forest regression-based prediction approach that not only is more accurate than the traditional Random Forest, but also runs faster due to the small size of the ensembles it produces. To achieve both accuracy and speed, we empirically validated the principle that diversity in ensembles can lead to better performance. To do this, we have used clustering as a diversity technique to further diversify the traditional RF, resulting in CLUB-DRF. Since trees in the original ensemble (parentRF) were clustered into groups of similar trees, and a representative was selected from each group, many redundant trees were eliminated. Hence, CLUB-DRF ensembles with extreme pruning levels reaching as high as 99% were produced. As was demonstrated in the results obtained in Sect. 4, these CLUB-DRF ensembles not only run faster due to their small size, but also perform at least as good as the parentRF mainly due to the high level of diversity in their constituent trees.

As future work, the application of the proposed method on other healthcare data sets will be applied for both regression and classification problems. Also the deployment of the pruned models on handheld devices like smartphones will be experimented. Due to the small size and fast prediction, personalised healthcare through deployment of these models can be an interesting prospect.