Using random forest and decision tree models for a new vehicle prediction approach in computational toxicology

Pharmaceutical drug toxicity is a major concern facing the drug discovery industry today, resulting in not only high drug attrition rates but increased developmental costs (Basavaraj and Betageri 2014), which may not be recovered if the drug does not make it to market. Of particular concern are the anticancer compounds for which it is estimated that 95 % of the attrition rates are related to toxicity concerns (Hutchinson and Kirk 2011), which is unsurprising given the nature of their modes of action as cytotoxics. Safe, effective use of anticancer drugs is often limited by their dose-dependent toxicities typically associating such drugs with a narrow therapeutic index (NTI) (Liang et al. 2013). In a study involving hospitalised patients using NTI drugs compared to non NTI drugs, it was shown that more drug-related problems (DRPs) were associated with NTI drugs, of which adverse drug reactions was one of the eight categories considered to be a DRP (Blix et al. 2010).

Toxicity prediction without experimental effort is a major area of study in the field of toxicology. The ability to predict the nature of the toxicity of a previously untested chemical structure is highly desirable to a drug discovery team deciding which chemical structures they should continue developing. The ability to make informed decisions can not only save time and money but also bring new drugs to market sooner, benefiting the patient population as a whole. Using historical in vivo and in vitro experimental data for in silico predictions is a challenging problem. Making such predictions involves soft computing approaches that use quantitative structure activity relationships (QSAR’s) to relate the activity (toxicity) of a chemical to its structure (Eldred and Jurs 1999; Eldred et al. 1999). Many useful computational models have been developed with varying success. Models ranging from studies on aquatic toxicity that utilise multiple linear regression and neural networks (Eldred et al. 1999) through to predictive hepatotoxicity (Cruz-Monteagudo et al. 2008; Low et al. 2011 and mutagenicity (Bakhtyari et al. 2013; Xu et al. 2012; Modi et al. 2012) that use k-nearest neighbour, support vector machines and random forests are frequently reported in literature.

In this study we report on a methodology developed to build QSAR classification models based on drug-vehicle toxicity relationships. As more data becomes publicly available new methodologies are required to extract the relevant information that enables successful predictive models to be built. Using data frequently reported in toxicity studies we utilise existing data and develop a methodology that is applicable across the wider domain which could be adapted for other similar predictive problems. Specifically we employ an area under the curve (AUC) approach to discriminate between the toxicity classes of different drug vehicles. A binary classification model is built using C4.5 decision tree (DT) and random forest (RF) algorithms as machine learning methods with predictive accuracies in the 70 and 80 % range respectively. That is, our models are able to predict which vehicle will provide the better toxicity relief to a particular chemical drug.

The remainder of this paper is arranged as follows: Sect. 2 provides some background information that briefly describes the purpose of drug vehicles and how they are able to reduce a drug’s toxicity. Section 3 describes our proposed approach to building classifiers based on our AUC methodology. Section 4 discusses the dataset we acquired, its content and the challenges faced when using it. Section 5 provides the experimental details of our work. Section 6 reports the results generated from our predictive models and provides the discussion to this work. Section 7 ends this study with our conclusion.

To overcome problems relating to the toxicity of a drug significant efforts are made. Some involve the use of drug vehicle formulations that can considerably attenuate this toxicity, allowing the safer use of a potent yet highly toxic compound. Drug vehicles behave as “carriers” that aid the administration and distribution of a drug through an organism and with considered selection can maximise the efficacy and minimise the side effects. For instance, studies have shown that vehicles can be selected to improve the solubility and/or permeability of a drug or to target the drug to a specific disease site (Loftsson 1998; Porter et al. 2007; Shin et al. 2009; Hans and Lowman 2002; Liu et al. 2011; Lee et al. 2003). Vehicles have also been designed and formulated to mitigate unwanted toxicities associated with drugs that are considered highly toxic (Huo et al. 2010; Uchino et al. 2005; Kelava et al. 2010). Vehicles offer toxicity relief in several ways. A vehicle that improves the solubility or permeability of a drug removes this rate limiting step (Savjani et al. 2012) in the drug’s absorption profile and so may provide some form of local toxicity relief, that is, toxicity at a local site, typically the site of administration. For instance, studies on the antifungal drug amphotericin B have shown that its precipitation upon infusion is related to its acute toxicity. Solubilisation of amphotericin B using polymeric micelles, has resulted in a reduced haemolytic activity of the drug (Yu et al. 1998). Similarly the use of cyclodextrins to enhance the gastrointestinal absorption of tacrolimus was shown to reduce renal and neural toxicity in rats (Arima et al. 2001).

A drug that is toxic to a particular organ may be formulated to avoid accumulation within that organ, or targeted to a specific disease site such as a tumour thus avoiding accumulation within other organs. This type of vehicle formulation involves the modulation of a drug’s plasma concentration, ensuring it remains below the toxicity threshold but above a level needed for efficacy. Vehicles that alter a drug’s pharmacokinetic properties can reduce toxicity by limiting the C \(_\mathrm{max}\) (peak plasma concentration) (Kim et al. 2001; Italia et al. 2007). Two drug vehicles may exhibit the same area under the plasma-drug concentration curve for the same drug, but the vehicle associated with a higher C \(_\mathrm{max}\) value may produce a toxic outcome compared to a vehicle which produces a flatter and more prolonged plasma drug concentration profile (Fig. 1).

For a drug to produce observable toxicity effects it must exceed a toxicity threshold. The use of vehicle formulations can significantly alter the pharmacokinetic profile of a drug to prevent it exceeding this toxicity threshold. Figure 1 conceptualises this using the hypothetical drug X. While vehicle A produces a spike in the plasma-drug concentration curve and exceeds the toxicity threshold, vehicle B produces a flatter, prolonged plasma-drug concentration curve which remains below the toxicity threshold. Vehicle B therefore does not result in any measurable toxicity outcome but can still provide therapeutic benefit.

An alternative method for modulating a drug’s toxicity is with the aid of co-administrative agents. Such agents can be other drugs, protein extracts, vitamins or oils. They work not by altering a drug’s pharmacokinetic profile but instead by countering or moderating the drug’s toxic manifestation and providing some form of toxicity relief. For example this relief may come in the form of antioxidant benefits related to these co-administrative agents. Vitamins C and E are known to provide nephroprotective effects to cisplatin induced kidney injury in mice (Ajith et al. 2007; Maliakel et al. 2008). Other co-administrative agents may behave as deactivators of toxicity pathways that are initiated upon drug administration. Acetaminophen is a globally used analgesic that results in hepatic injury at high doses. It is not the parent drug but rather its reactive metabolite N-acetyl-p-benzoquinone imine (NAPQI) that results in hepatic injury (James et al. 2003; Hodgman and Garrard 2012). Whilst at low acetaminophen doses, NAPQI is deactivated via conjugation to endogenous glutathione, at high doses glutathione becomes depleted leaving the reactive metabolite NAPQI to bind covalently to hepatic proteins resulting in hepatotoxicity. In humans the most commonly associated cytochrome P450 enzymes that result in NAPQI production are CYP2E1, CYP1A2 and CYP3A4, of which CYP2E1 accounts for the greatest transformation (Lee et al. 1996; Patten et al. 1993). Lee et al. investigated the co-administrative use of chlormethiazol, an inhibitor of CYP2E1, with high doses of acetaminophen. They demonstrated that chlormethiazol was able to prevent liver injury in mice resulting in the survival of mice administered a high dose (500 mg/kg) of acetaminophen. A greater than 50 % death rate was observed in mice that were not co-administered chlormethiazol (Lee et al. 1999).

We propose a methodology of extracting the relevant information from our dataset and processing this using an area under the curve approach to assign classification of vehicles. This approach can be used for a multitude of problems that compare the functional relationship of two variables for different objects (vehicles in our case). Section 2.1 discusses some ways in which toxicity is commonly measured. In our study, toxicity was measured as animal survival per dose administered. Given this we are able to extract dose-survival relationships for drugs that are administered using different vehicles.

For any vehicle V \(^{k}\) for any given drug we consider it represented by two arrays D \(^{k}\) and S \(^{k}\), which are of equal size, \(n, (n \ge 2)\) where \({D}^{{k}}=\left[ {{d}^{{k}}_{{1}} \ldots {d}^{{k}}_{{n}} } \right] \) and \(\text{ S }^{{k}}=\left[ {{s}^{{k}}_{{1}} \ldots {s}^{{k}}_{{n}}} \right] \), D \(^{k}\) is the dose amount array and S \(^{k}\) is the survival array: \({V}^{{k}}=\left\{ {{D}^{{k}},{ S}^{{k}}} \right\} \). Elements of array D \(^{k}\) are ordered ascendingly: \({d}_{{1}}^{{k}} <{d}_{n}^{{k}}\) where \(n > 1\). In this case the value \({s}_1^{{k}}\) represents the survival for the smallest dose amount of drug using vehicle V \(^{k}\), and the value \({s}_{{n}}^{{k}}\) represents the survival value for the largest dose amount of drug using vehicle V \(^{k}\).

For two different vehicles (V \(^{1}\) and V \(^{2})\) used to test the same drug a plot of S \(^{1}\) against D \(^{1}\) would produce a dose versus survival curve for vehicle V \(^{1}\) and likewise a plot of S \(^{2}\) versus D \(^{2}\) would produce a curve for vehicle V \(^{2}\). Calculating their respective area under the curves can then be used to compare the toxicity difference afforded by the vehicles. Whist the elements in the arrays for V \(^{1}\) may not be equivalent in length or value to the elements in the arrays for V \(^{2}\) it is important to ensure that the maximum number of data points spanning the largest possible area under the dose versus survival curve is used to compare the toxicity difference between the two vehicles.

For simplicity let us consider vehicles V \(^{1}\) and V \(^{2}\) and the arrays they represent. To maximise data comparison these arrays were subjected to further extrapolation and interpolation before AUCs were calculated.

For this study we used the in vivo screening dataset obtained from the National Institutes of Health’s Developmental Therapeutics Program (DTP) (DTP 2000). This dataset contained 2,724,199 records, detailing experimental in vivo screening results of 227,093 potential cytotoxic drug candidates. Their aim was to assess the ability of the drug candidate to reduce the size of a tumour cell line inoculated onto a host. Each drug was administered using a carrier in the form of a drug vehicle. There are 39 different vehicles used within the dataset for which some vehicles had been used more frequently than others (Fig. 6).

Whilst we do not consider our dataset to be “big data” as defined by several proposed definitions of big data, it does meet some of the criteria that are generally accepted (Laney 2001; Berman 2013; Hashem et al. 2015). Big data is not simply a large volume of data or a large structured database containing lots of records. There are several proposed definitions that currently exist to define big data of which the three V’s is the most prominent.

The three V’s define big data as:

Volume—this refers to the amount of the data being large and often derived from various sources.

The size of our dataset is indeed large, approximately 2.7 million records, and these data have been compiled from different sources given that there were several different laboratories (screeners) that produced these data. This gives rise to the complexity of how reliable the data from multiple sources can be, each of which will carry a different error rate associated with the data they generate making comparisons across sources challenging.

Variety—Variety means that the data come in different forms or collected via different means.

This is true to some extent with our dataset given that different laboratories (screeners) were involved in generating and reporting the data. However, the form in which these data have been compiled is the same across all screeners.

Velocity—Velocity is the speed at which the data are generated which means the dataset is dynamic due to the absorption of complementary data.

Our large dataset is not changing dynamically or increasing in size. Most of these data were generated from the early 1950s up until the late 1980s. It covers an era where computers were not widely used across industries as they are used today. A large proportion of the dataset we are working with was only available electronically due to a substantial effort made to translate all the data from a hard copy format into a digital one (DTP 2000). Our dataset does however carry the possibility of having complimentary data added to it. If similar data become available from alternative sources then there is no reason why some form of data integration cannot occur between the datasets. For our work no additional data were added to the original dataset we acquired.

Presented with some big data challenges our dataset containing approximately 2.7 million records with drug-vehicle relationships was investigated. Working with big data does not always lead to novel discoveries, but presents challenges of its own as discussed above. The main purpose of this study was to propose and develop a methodology what would enable the extraction of drug-vehicle comparisons in order to build classifiers.

Using the experimental methodology discussed above we ran several prediction models. Given the importance of the classifier selection threshold (see Sect. 5.1) which in turn is determined by the interpolation and extrapolation procedures discussed in 3.1 and 3.2 of our methodology section we decided to run our models using 3 different threshold settings of 30, 40 and 60 %. For each of these 3 different thresholds we built models on data that were interpolated only with no extrapolation (Interpolation only), interpolated with high dose extrapolation (Extrapolation high) and interpolated with extrapolation at the high and low doses (Extrapolation high=-low). The output of these models is shown in Table 2. From Table 2 we see that the RF model tends to outperform the DT on almost all occasions as expected (Diaz-Uriarte and de Andres 2006; Caruana and Niculescu-Mizil 2006).

We generally see better accuracy values for DT’s and RF’s as the threshold increases from 30 to 40 to 60 %, possibly suggesting that the amount of noise in the original dataset is high and for any signal to be observed the necessary threshold needs to be high. We did not observe any trends in performance as the data defining the AUC range are extrapolated at one end or both—though the number of compounds included in the model is consistently greater when extrapolating at the high end only.

The best predictions we produce come from a RF model at the 60 % threshold with data extrapolated at the high and low doses (Extrapolation high–low), which produces a balanced accuracy of 80.8 %. This model also has individual prediction accuracies for Saline and CMC of 80.6 and 81.0 % respectively showing that the predictive outcome of the individual classes is evenly distributed. With the threshold set so high, i.e. the class discriminator for AUCs set to 60 % for classes to be assigned, we only produce a dataset of 52 compounds to be modelled.

Although the random forest models tend to outperform the decision trees we gain some insightful scientific information from the decision trees. Specifically, collecting information on how the trees are built and what descriptors the branches split on can provide some interesting scientific interpretation for our results.

Doing this for the trees built in our models brings up some interesting findings. We find that from the 9 different decision trees built in our models (each iterated 10 times) there were five MACCS fingerprints that were used frequently to split the branches on. These five most common fingerprints were (position-description); 120-HETEROCYCLIC ATOM \(>\)1, 109-ACH2O, 115-CH3ACH2A, 137-HETEROCYCLE and 53-QHAAAQH.

In this study we have introduced a methodology that shows how classifiers can be built on scientific data for model predictions. We explain the steps of this methodology that is applicable across many scientific disciplines. We show how existing scientific data can be repurposed and mined for new knowledge discovery. An additional innovative feature of our methodology is the use or our proposed interpolation and extrapolation to increase the data points used for real world datasets.

For our experimental work we acquired a large dataset from the Developmental Therapeutics Program and showed our methodology to work well and built classifiers that produce accuracies of 80 %.

This work shows that the effects on toxicity of drugs by different vehicles can be modelled, and that well-performing, interpretable models can be built if sufficient data are available. To the best of our knowledge this is the first study building models in this field.

We consider for our future work, extending our current approaches in dataset preparation to non-linear interpolation and extrapolation to deal with survival curve complexity. We also intend to explore other vehicle relationships that are contained within our dataset.