Through the looking glass: evaluating post hoc explanations using transparent models

The “black box problem” of AI arises from the inherent complexity and sophisticated internal data representations of many modern machine learning algorithms [26]. Although more complex predictive models may produce more accurate results, this accuracy comes at the cost of human interpretability of algorithmic decisions [24]. XAI research attempts to provide human-understandable explanations for predictive models. This is necessary to ensure system quality and to facilitate informed human decision-making [8].

In this work, we consider “interpretability” to be the ability to provide meaning in terms understandable to a human and “explanations” to be the interface between the human and the predictive model [18]. Interpretability in machine learning is generally broken down into two categories: inherently interpretable predictive models and post hoc interpretation. Interpretable predictive models are those that are immediately interpretable by a human [18], though this often means that the models are simpler, and so may have reduced predictive power. This category includes simple decision trees and linear and logistic regression models, which are easily interpretable, and thus fully transparent.

Post hoc methods provide interpretations that are not inherent to the predictive model and are derived after its creation through an external mechanism [18]. Post hoc methods are usually applied to opaque models, which have complex internal mechanisms, such as a neural network, or are otherwise “black box”. A sufficiently complex tree-ensemble model or a model protected by IP agreements may also fall into this category [43]. Thus, post hoc interpretability allows for the use of more complex and accurate predictive models without compromising interpretability. However, since post hoc interpretation is external to the predictive model, there is no guarantee that the explanations provided are fully and correctly representative of the underlying model [43].

The purpose and scope of post hoc explanations often vary [29]. In this work, we assess four local (prediction–level) explanations, rather than global (whole–model) explanations. That is, when given an individual input \(x \in X\) of length n and a prediction function f, a local XAI method would return an explanation \(\phi (x)\), rather than \(\phi (f)\). In this work, we use four methods that generate local, post hoc explanations.

Moreover, we examine XAI methods that are model-agnostic. These methods typically use some explanation-generation mechanism external to the underlying predictive model, and/or use multiple mechanisms suited for examining multiple types of predictive models. We describe these in more detail below.

Evaluation methods for XAI can be categorised into three levels of evaluation [12]:The former two categories are also often referred to as cognitive metrics of evaluation, and functionally-grounded evaluations fall into the complementary category of computational metrics [26].

Evaluations of this last category are often the first step in determining the quality of an XAI method [12]. This includes evaluation of explanation fidelity with respect to the original predictive model. The term “fidelity” is generally defined as a measure of how faithful and accurate an explanation is to the underlying black box model [56]. Two characteristics of explanations are relevant to explanation fidelity [30, 59]: the completeness of an explanation in capturing the dynamics of the underlying model; and the correctness and truthfulness of the explanation with respect to the underlying model.

Explanation fidelity can severely impact the safety and usefulness of a predictive system. The authors of [21] find that, when the user has deep knowledge of the context, local explanations enabled the identification of prediction errors and the predictive model’s error boundaries. The study also finds that explanations affect the user’s perception of model correctness. Similarly, the authors of [47] find that model transparency affects causability—the ability of the user to understand what features affect the prediction. In turn, causability is found to determine the user’s trust in and perception of model performance. We must note that explanation fidelity does not play a direct role in causability or user perception; these are generally determined by explanation presentation and content [21, 58]. However, if a user’s knowledge or perception is based on incorrect or incomplete explanations of the model, it may engender trust in a false prediction, or mistrust in a correct prediction, leading to unsafe or incorrect usage of a predictive system. Thus, explanation fidelity is necessary for predictive model safety and usefulness.

Although a number of methods have been proposed to assess fidelity, they are often highly specific to particular XAI methods or datasets. Fidelity evaluation approaches fall into two general measures: measurements of external fidelity and internal fidelity [32]. External fidelity approaches are those that compare how often the decision implied by the explanation and the decision made by the underlying predictive model agree. For example, when evaluating surrogate models that approximate the underlying predictive model, this method determines the accuracy of the surrogate model’s predictions using the predictions of the underlying model as ground truth [46]. Explainable methods that produce decision rules are also often evaluated through this approach, and the predictions of the decision rule or decision rule sets are compared against the black box model’s predictions [19, 25, 33].

Evaluating external fidelity provides an assessment of consistency between the explanation’s predictions and the predictions of the underlying model, i.e. that the predictive model and the explanation mechanism can reach the same decision. However, this does not guarantee that the explanations are faithful to the computations, or “decision-making”, of the black box model, for example, weights given to each feature, or the decision path followed in a tree [32]. Such information may be necessary in order to change the predicted outcome to a desired outcome. For example, in the case of a risk prediction model, taking the most appropriate action to reduce the risk. Lack of full transparency can also hamper model inspection and diagnosis, which may be necessary to ensure that the model relies on appropriate features when computing a prediction [41]. Moreover, some explainable methods do not produce an output suitable for external fidelity evaluation approaches, including LINDA-BN and some optimisations of SHAP.

As such, we aim to develop a method to assess the internal fidelity of feature attribution explainable methods, where the fidelity of the explanation regarding the underlying model’s decision-making process is evaluated [32]. As noted in [6], given that the internal workings of a black box model are opaque and there is no “ground truth” of the model, it is near impossible to assess explanation correctness. Significant effort is required to assess the internal fidelity of an explanation or explainable method. Potential internal fidelity evaluation approaches include: It is important to note that the method in [32] is an evaluation of a surrogate model using a post hoc technique, rather than an evaluation of the post hoc technique. Thus, it is not suitable for this work. As another note, while the method of permuting features is relatively common in the literature, it is mostly applied to text [13, 38] or image data [16, 26, 44], in which case, the relevant features are often removed or blurred from the original input prior to encoding. This method has been applied to tabular data [4, 31, 37], typically by permuting rather than removing features at the input level [4, 37] or removing the feature as a whole from the dataset [31]. Moreover, we note that these works typically used few, standard benchmark datasets and/or limited predictive models. In our previous work, we adopted a local, perturbation method to evaluate explanations for models built on time series tabular data, which, however, provided poor results [49]. We theorise that the results of our previous work were likely influenced by the difficulty in choosing and permuting an appropriate subset of features given the complexity of time series data, and lack of understanding of model and XAI technique behaviour with respect to tabular data.

Tabular datasets are highly heterogeneous with respect to the quantity and type of features and may present challenges such as lack of locality and sparsity in data, as well as the need to manage mixed feature types [48]. Furthermore, there may be fewer correlations between features in tabular data than in other types of data such as images, as well as lack of semantic meaning and connection between features such as may be present in natural language data [7]. Although it is noted in [17] that simpler, glass-box models can generally provide accurate predictions for tabular data, the authors of the study also note that in some cases, particularly when the dataset is “noisy”, more complicated black box models may be necessary. These factors affect the predictive techniques that may be used [7], but, as we noted in [49], the characteristics of tabular data also does not allow for use of explanation evaluation methods developed for image and text data.

In this work, we use fully transparent, glass-box models to better understand interactions between data, model and XAI technique. This provides a “ground truth” for model decision-making, with which to assess explanations. Furthermore, we extend the approach used in [41], which relied on measuring only the completeness of the explanation in capturing a hard-coded, arbitrary number of features. Moreover, to the best of our knowledge, no work compares the performance of XAI techniques in different technical contexts to understand the factors that affect explanation quality. In particular, we aim to examine whether easily accessible, model-agnostic XAI methods, such as those described, work reliably and consistently in all technical contexts. Thus, in this work, we attempt to take a more holistic approach to evaluation, as described in Sect. 3.2.

We investigate properties of post hoc XAI methods by addressing the following questions:

In this work, we aim to investigate the internal fidelity of model-agnostic explanations, where the faithfulness of the explanation to the underlying model’s computational process is evaluated [32]. Given that the internal workings of a black box model are opaque, significant effort is required to assess the internal fidelity of an explanation or explainable method. In our previous work [49], we attempted to assess explanation fidelity for black box models trained on tabular data, by replicating the methods used to test the internal fidelity of explanations for models trained on image and text data. Our results suggested that explanation fidelity is poor when explaining tabular data, though we found that this may be a consequence of the evaluation method and the complexity of the datasets used. This lack of evaluation method for tabular data is notable given the volume and diversity of tabular data used in modern data analytics.

Thus, in this work, we evaluate explanations using fully transparent glass-box models. We choose models from which the relevance of a feature to a prediction can be determined, to provide a reference for evaluating the internal fidelity of post hoc explanations. As the internal workings of a glass-box model are apparent, it can be compared against an explanation to determine the fitness of the explainable method. It must be noted that high explanation fidelity for a relatively simple glass box method does not necessarily imply high fidelity for a more complicated black box model. As such, our aim in using this method is not to determine some “best” XAI method. Rather, we aim to more precisely understand the specific workings of each XAI method, how different types of predictive models affect explanation quality, and any patterns of quality that could inform the selection of XAI methods for a given setup and context. Thus, we focus on behaviours and characteristics of models and XAI algorithms. Moreover, given the ease of availability of model-agnostic XAI techniques, we also wish to determine the level of expertise needed to select an XAI technique for use.

In this work, we choose to examine four local, post hoc XAI methods of differing characteristics in order to answer the questions posed:We have chosen datasets and predictive models in such a way as to facilitate answers to these questions. Our experimental setup and choice of algorithms are introduced in 4.

In this work, the goal of our evaluation is to understand the strengths and limitations of several local, post hoc XAI mechanisms. In particular, we examine these mechanisms with respect to the characteristics of the underlying models and datasets. Thus, in this work, we use a range of datasets, XAI techniques and predictive models for a comprehensive evaluation. We evaluate four XAI techniques, all of which use different explanation-generation mechanisms, with three predictive models of different classes. We also selected datasets by considering a breadth of different characteristics, including the prediction problem, the volume of the data, types of variables present, and the number of features that can be encoded from the data. A detailed description of techniques and datasets used is presented in Sect. 4.

We evaluate three properties when measuring fidelity:The evaluation method for the first two properties use two subsets of features. The basic approach of this method is to compare the subset of features determined to be most relevant by the model (\(x_t \subseteq x\)) against the features determined to be the most relevant by the explanation (\(x_e \subseteq x\)). A similar approach was used in [41], where the recall metric was used to determine fidelity in the form of completeness as follows:Additionally, we also use the precision metric to capture the correctness of the explanation as follows:To evaluate the third property, we extract the importance ranking of features by the model (\(r_t\)) and the ranking of features by the explanation (\(r_e\)), where r is a reordering of x indicating feature importance. We then use a correlation measure to determine the similarity between \(r_t\) and \(r_e\), where a higher similarity would indicate higher correctness of the explanation’s ranking. We use Kendall’s Tau-B to measure rank correlation between \(r_t\) and \(r_e\).

As XAI methods, particularly methods relying on feature permutations, often produce unstable explanations [46, 50, 51], we generate five \(\phi (x)\) for each x, compute an adjusted or average explanation \(\varphi (x)\), and determine \(x_e\) and \(r_e\) based on \(\varphi (x)\).

These metrics are applied at the local level (i.e. when testing individual inputs), but can be averaged out at the dataset level, as in Sect. 5.

Note that, of the chosen predictive methods, only the decision tree model provides a natural subset to be used as \(x_t\). Similarly, only the ACV provides a subset of features as \(\phi (x)\). Thus, a heuristic is necessary to identify \(x_t\) and \(x_e\) for other methods. The procedure for determining this heuristic, as well as the results, are summarised in Sect. 4.4.

We focus our evaluations on tabular data, which has been relatively unexplored in the literature on explanation fidelity. In order to better understand the effect of dataset properties on the results, we chose datasets with specific characteristics, rather than datasets from any particular domain. The datasets all vary in terms of the variable types present, the number of variables present once the dataset is encoded as described below, the volume of the data, and, in the case of the regression datasets, the distribution of the target variable.

A total of 14 open source datasets are used, which include seven classification datasets: and seven regression datasets: A brief profile of each dataset is provided in Tables 1 and  2.

Some pre-processing was necessary for all datasets. All classification datasets were balanced through downsampling, to ensure parity between classes. Where a classification dataset had more than two prediction classes, the target variable was binarized before downsampling. All categorical variables in the datasets were one-hot encoded, and min-max scaling was applied to all numeric variables (excluding the target variable in regression datasets). A train-test split of 70–30 was used, and a sample of the testing set was used to evaluate the XAI methods.

In this work, we use predictive model types for which extracting \(x_t\) is unambiguous. As such, we choose to exclude techniques for which \(x_t\) cannot be easily derived, or are unclear. For example, attention mechanisms are an increasingly popular technique used to interpret models without post hoc methods. This is a form of input selection within some neural networks, which allows the model to focus on inputs most relevant to the output [23]. While attention mechanisms are sometimes used as explanation and have been shown to be successful in some domains [53], the validity of attention as explanation is still debated [20, 31, 54]. Thus, we choose not to use it in this work.

We use four different algorithms of three different classes to create predictive models. Firstly, we use a simple CART decision tree to create both classification and regression models, covering all datasets. For all classification datasets, we also create logistic regression and Naïve Bayes models, and for all regression datasets, we create linear regression models. In summary, we have conducted our evaluations using a total of 35 models: 21 classification models and 14 regression models. The accuracy of these models is shown in Tables 3 and 4.

Given their differences, the extraction of \(x_t\) from each type of model varies. We provide a summary of the used methods in Table 5.

We use and evaluate four XAI methods in this work, each of which use different underlying mechanisms. The format of the provided explanations also varies across the explainable methods. The method by which we extract \(x_e\) from each method is described in detail below. A summary of the XAI techniques used, the feature extraction method and the experiments that it is used for is provided in Table 6.

As specified, one of the goals of this work is to understand the effect of explanation mechanisms on explanation fidelity. Thus, we examine XAI techniques with different underlying theoretical principles and explanation-generation procedures. LIME uses linear, local surrogate models to determine feature importance [41], while SHAP is grounded in game theory and differs its approach to determining the contribution of a feature depending on the underlying model [27, 28]. LIME and SHAP are also among the most commonly used local, post hoc techniques for tabular data, and thus, important candidates for evaluation. Although LINDA-BN also uses local surrogate models to generate explanations, unlike LIME, it is grounded in probability theory and uses a Bayesian Network as surrogate model [34]. Moreover, unlike LIME and SHAP, LINDA-BN does not attempt to determine feature importance to the prediction, but examines the conditional dependence between all variables, including the features. Similarly, ACV is also not a feature attribution method, but attempts to identify which subsets of features are necessary for the prediction [5]. Thus, we choose LINDA-BN and ACV in contrast to the more “conventional” LIME and SHAP.

Thus, a certain percentile of the Top-K features must be chosen as subsets from methods that do not provide subsets. To determine the most appropriate percentile, we conducted tests to identify a percentile from \(0.05\ldots 0.5\) that would provide the most faithful explanations. For each combination of dataset, model and XAI technique, we extracted \(x_t\) and \(x_e\) using each of the ten percentiles and calculated the F1-score to determine the match between the subsets. As with the fidelity evaluation, this resulted in 126 experiments. In 38 experiments, the subsets produced for all percentiles were equally faithful, i.e. the top-K percentile had no impact on the fidelity of the subsets, and these results were set aside. Out of the remaining 88 experiments, 0.05 produced the most faithful subsets in the majority of experiments (see results in Fig. 3). Of these, in 49 experiments, 0.05 produced the most faithful subsets. In another 38, the subsets produced for all percentiles were equally faithful, i.e. the top-K percentile had no impact on the fidelity of the subsets. Thus, 0.05 was chosen to determine \(x_t\) and \(x_e\).

There are a number of notable observations to be made from these results.

LIME’s results are highly inconsistent across the different experimental settings. LIME explanations generally have high precision coupled with low-to-moderate recall scores, indicating that a large number of relevant features are missing from the subset, but also that few extraneous features are included. Feature ranking correlation is also generally moderate. There are no significant differences in LIME performance between classification and regression model. However, model type appears to affect results, and LIME generally performs poorly when explaining Naïve Bayes models. Dataset characteristics also affect LIME explanation quality, and explanations are more faithful for datasets with a larger proportion of categorical features.

SHAP explanations also change significantly under different experimental conditions. Feature subsets derived based on SHAP contributions show high precision in the results, but poor recall and generally low-to-moderate feature rank correlation. SHAP shows no difference in performance between classification and regression datasets, but does show poor performance when explaining logistic and linear regression models, and exceptional precision when explaining decision tree models. Dataset characteristics do not appear to affect SHAP’s explanation fidelity.

The faithfulness of LINDA-BN’s explanations appears to be related to model confidence in predictions. LINDA-BN’s explanations generally had correctness, with respect to both correctness of the subset and the correctness of the feature ranking. However, recall scores for LINDA-BN’s explanations were generally high. Closer inspection shows that the set \(x_e\) returned by LINDA-BN contains numerous features. Thus, simply by volume, this set includes many features that are also present in \(x_t\). This does not appear to be related to model type, or dataset characteristics, but the prediction probability associated with a prediction. This phenomenon was most common when explaining models trained on the Iris, Mushroom and Nursery datasets, all of which produced predictions with high prediction probabilities.

ACV shows little to no consistency in performance, especially when explaining the classification models. ACV explanations generally have low-to-moderate scores for all measures. In particular, ACV generally performs poorly when explaining linear regression models, but otherwise shows little consistency in performance.

We examine these phenomena further in Sect. 6.

There were notable differences in XAI technique performance for each of the three fidelity criteria examined. We applied the Wilcoxon Signed Rank test with a significance level of \(p<0.05\), to determine whether this difference in performance is statistically significant. Our results Tables 9 and 10) show that there is a significant difference in the performance of the four techniques. Additionally, we also compute the F1-score from recall and precision. We find that the differences between XAI techniques on precision and recall do not appear to be a trade-off, as the F1-scores are also distinct across the XAI techniques.

LINDA-BN performed best with respect to the completeness of the subsets. The low precision scores that generally pair with high recall scores, and close examination of LINDA-BN’s explanations suggest that this is likely because LINDA-BN’s feature “weights" are often similar. This is especially true when the model is confident regarding a prediction. The other XAI methods generally produce less complete explanations than LINDA-BN. However, the consistently low correlation between \(r_t\) and \(r_e\) suggests that LINDA-BN simply does not perform adequately as a feature attribution technique. Thus, while LINDA-BN generally has higher recall than the other methods, this cannot be said to be an advantage of LINDA-BN.

Feature ranking correlation across all XAI methods was poor. These results suggest that none of the XAI methods could fully capture the true importance of all features to the model. This is particularly noteworthy when considering LIME and SHAP as they were intended to generate feature attribution explanations, which is closely tied to feature ranking. ACV’s low feature ranking correlation suggests that the LEI scores, which were used to determine \(r_e\), are not fully aligned with the importance of the model. It must be noted that the LEI is suggested by the creators of ACV to show the necessity of each feature to the prediction [5], and thus, it may not fully align with its importance to the model.

Overall, we note significant differences in the performance of the four XAI methods. ACV was the most consistent method, though it generally performed poorly across all measures, indicating that it could not fully capture the features most important to the model. LINDA-BN’s explanation fidelity is more consistent across model types than that of LIME or SHAP, but generally does not provide precise explanations. While LIME and SHAP generally provide precise explanations, the explanations they offer may not be complete. Moreover, the characteristics of the underlying model and data appear to affect the faithfulness of LIME and SHAP.

There is a clear distinction between the precision and recall scores for almost all XAI techniques evaluated. In particular, precision results for LIME and SHAP are generally higher than recall. That is, while the features most highly weighted by LIME and SHAP do not necessarily include all relevant features (i.e. the explanation is not complete), in general, all features weighted highly by the explanation are relevant to the model (i.e. explanation correctness is high). Thus, a user applying LIME and SHAP explanations can be confident that highly weighted features are truly relevant to the model, but must also be aware that other factors could have impacted on the model prediction.

LINDA-BN generally has low precision, often paired with high recall. This is a characteristic of the permutation method of LINDA-BN. Because LINDA-BN was intended to determine the conditional dependence between variables within a small neighbourhood, the variance \(\epsilon \) of permutations is generally low. Thus, in a neighbourhood where small changes in feature values do not necessarily affect the model’s prediction, the Bayesian Network returned by LINDA-BN reflects this. When \(x_e\) is extracted from the network using our method, \(\textrm{Pr}(f(x) \mid x_i)\) is the same or similar for a large number of features, and a large set of features is returned as \(x_e\). Such a result indicates that a user can have strong confidence in the original model’s prediction [34]. However, a faithful feature attribution explanation cannot be extracted from LINDA-BN for such a prediction.

All XAI techniques generally show a low-to-moderate correlation between \(r_t\) and \(r_e\). This is particularly noteworthy when using LIME and SHAP, which explicitly attempt to explain the importance of each feature to the prediction. Combined with the findings regarding precision and recall, we can come to the conclusion that while a user of LIME and SHAP can be confident that highly ranked features are relevant, they must be aware that the ranking of features as a whole may not be correct. Moreover, neither can they assume that they have identified all relevant features.

It must also be noted that LINDA-BN and ACV were not intended to provide feature attribution explanations. Their internal explanation-generation mechanisms reflect this, and thus, it is not surprising that both show low correlation scores. We explore this further in Sects. 6.4 and 6.5.

It is important to note that SHAP values are not defined as the importance of a feature to the model. Rather, SHAP values indicate the contribution of that feature in moving the actual prediction away from the expected value of the prediction, i.e. the average value of the target variable in the training set. In our evaluations, we compared SHAP values against the importance of the feature to the model. In the case of decision trees, we found that SHAP values aligned closely with the model regarding feature importance, likely because position in the tree matters both to the prediction and TreeSHAP’s calculation of SHAP values [28]. On the other hand, LinearSHAP assumes that the model treats features in the dataset as conditionally independent, and calculates SHAP values as for a given feature \(x_i \in x\) as [27]:Therefore, SHAP values, when produced by LinearSHAP, do not fully align with model importance.

Given this, we suggest that SHAP may be more useful for supporting end user decision-making, rather than data engineers or data scientists investigating the quality of a model.

LINDA-BN was designed to aid in understanding the dependencies between variables and determining the truthfulness of a prediction. Thus, rather than returning a static explanation, it returns a Bayesian Network describing the relationships between features within a small neighbourhood. This network can be further queried and explored to generate more insights into a prediction [34]. Generalising this explanation to a simple feature attribution explanation over-simplifies the information that this Bayesian Network provides, and so should not be the sole purpose of LINDA-BN.

However, in some cases, approximating the impact of each feature on the target variable does provide useful insights, even when the results suggest that the explanation is not sound. For example, as previously noted, the feature attribution scores extracted from the network returned by LINDA-BN could be used to determine the certainty of an explanation. Therefore, we suggest that the greatest strength of LINDA-BN is not in attempting to extract a single type of explanation from the Bayesian Network, but in querying and exploring the relationships between the variables using various methods, including the one presented in this work.

It must be noted, however, that understanding LINDA-BN’s explanations requires statistical knowledge and an understanding of Bayesian Networks. While a technical expert may have this knowledge, a domain expert in another field may not. Thus, LINDA-BN’s use for an end-user must also be evaluated prior to implementation.

There are several notable insights that can be derived from the described results. Firstly, extensive hyperparameter optimisation was conducted when training the surrogate model for ACV, resulting in relatively high surrogate model accuracy with respect to the original model’s predictions. However, our results suggest that this high external fidelity does not translate into high internal fidelity, particularly at the local level.

Secondly, in cases where no feature subset meets \(\pi \), no A-SE explanations are returned by ACV. As we use the A-SE to derive both the M-SE (which we take as \(\phi (x)\)) and the LEI (which is used to determine \(r_e\)), in cases where no explanation meets the \(\pi \)-level, we get poor scores for all three measures.

Finally, it is important to note that, although we used ACV as a post hoc explanation method in this work, ACV can function as predictive model in its own right. If used in this way, ACV would no longer have to mimic a distinct and separate model and can provide faithful explanations for its own behaviour. So, while ACV may underperform as a local, post hoc method in comparison with LIME and SHAP, its application as a self-explaining predictive model can eliminate the need for post hoc explanations altogether, which is generally more desirable [43].

Based on the insights identified in this section, we consolidate the strengths and weaknesses of the XAI techniques in Table 11.

These strengths and weaknesses can assist a data scientist or a practitioner in choosing an XAI technique to use. While these consolidated findings are not recommendations or guidelines for use, to the best of our knowledge, this is the first known attempt at moving towards such guidelines. Future work performing benchmarks of XAI performance can be useful in producing more concrete recommendations for XAI use in practice and in data science.

There are a number of limitations associated with our work. Firstly, we have evaluated XAI methods using only tabular data. While evaluating explanation fidelity for this form of data is relatively under-explored in the literature, it is unclear how our findings in this work may be generalised for other dataset types. In particular, Naïve Bayes models are commonly used in text classification. It is currently unclear how well our findings regarding the fidelity of linear explanation-generation mechanisms would apply for Naïve Bayes models trained on text data, rather than tabular data.

Secondly, in this work, we evaluate explanations by using fully transparent predictive models. Thus, we focus on the behaviours of the models and XAI techniques applied, rather than the classes of the models. It is key to note that high explanation fidelity for the relatively simple models we have used in this work does not necessarily translate into high explanation fidelity for other, more complicated predictive models. However, the findings of this work can provide guidance in selecting an appropriate XAI method, if considering the model or dataset behaviours, rather than model class. For example, if a model is known to assume a nonlinear relationship between features and target variable, such as with Naïve Bayes models, linear surrogate models may not be able to accurately capture those relationships. It is noted in [6] that an adversarial party could manipulate choice of explainable technique or explanation generation parameters in order to produce an explanation of benefit to them—a method which may be difficult for an external auditor or examiner to detect. However, with a clearer understanding of how explanation generation mechanisms interact with predictive techniques, such manipulations can be detected.

However, this work and the fidelity evaluation method used within could serve as the first step in developing a fidelity evaluation method for other contexts. As noted earlier, current fidelity evaluation methods at the local level performed poorly when applied to more complex tabular data than those used in this work. Future work could use the findings of this work in order to develop a new fidelity evaluation method compatible with the more complex time series tabular data, and which can examine explanations of black box predictive models. Our findings in this work regarding the behaviour of XAI techniques under certain situations can be used to validate such a method. We also suggest that glass box methods can be used as a proxy for black box models in developing a new evaluation method.

Similarly, we must also note that the evaluation methods outlined in this work necessitates that explanations be provided or can be reduced to some form of feature attribution or ranking. Though we found that LINDA-BN and ACV generally showed poor fidelity, this is because we simplified or focused on one aspect of the explanation, rather than assessing the explanation as a whole. Therefore, we suggest that future work should also explore methods to evaluate the fidelity of other forms of explanations. This includes not only non-static, queryable explanations, such as LINDA-BN’s explanations, but also counterfactuals, among others.

Furthermore, we suggest that the findings of this work could be used to motivate and inform future XAI development. In this work, we have outlined the strengths and weaknesses of each technique and identified potential causes for these. Future XAI development could use the insights presented in this work to develop XAI techniques that address these weaknesses.

Finally, it is important to note that the findings in this work are only the first step in determining the suitability of an XAI technique for a specific situation. When evaluating the XAI techniques used, we consider only the technical context of the explanation, such as dataset characteristics and the type of model used. This fails to consider how an explanation might be understood or perceived by a user. For example, past user studies have demonstrated that feature attribution explanations are often not helpful for users performing specific tasks [42, 52, 58], instead suggesting the rule-based or counterfactual-based explanations are most useful [9, 42, 52]. Moreover, some explainable methods, such as LINDA-BN, produce an explanation that a user must have a certain level of statistical knowledge to accurately understand, and so may not be useful for an end user. Thus, the use of cognitive metrics, used to gauge explanation quality with respect to a human user [26], also becomes necessary given a context.