Benchmarking and survey of explanation methods for black box models

Feature importance is one of the most popular types of explanation returned by local explanation methods. For feature importance-based explanation methods, the explainer assigns to each feature an importance value which represents how much that particular feature was important for the prediction under analysis. Formally, given a record x, an explainer \(f(\cdot )\) models a feature importance explanation as a vector \(e = \{ e_1, e_2, \dots ,\) \(e_m \}\), in which the value \(e_i \in e\) is the importance of the \(i^{ th }\) feature for the decision made by the black-box model b(x). For understanding the contribution of each feature, the sign and the magnitude of each value \(e_i\) are considered. W.r.t. the sign, if \(e_i < 0\), it means that the feature contributes negatively to the outcome y; otherwise, if \(e_i > 0\), the feature contributes positively. The magnitude, instead, represents how great the contribution of the feature is to the final prediction y. In particular, the greater the value of \(|e_i|\), the greater its contribution. Hence, when \(e_i = 0\), it means that the \(i^{ th }\) feature is showing no contribution to the output decision. An example of a feature based explanation is \(e = \{ age = 0.8, income = 0.0, education = -0.2\}, y = deny \). In this case, age is the most important feature for the decision \( deny \), income is not affecting the outcome, and education has a small negative contribution.

LIME, Local Interpretable Model-agnostic Explanations (Ribeiro et al. 2016), is a local model-agnostic explainer that returns explanations as feature importance vectors. The main idea of lime is that the explanation may be derived locally from records generated randomly in the neighborhood of the instance that has to be explained. The key factor is that it samples instances both in the vicinity of x (which have a high weight) and far away from x (low weight), exploiting \(\pi _x\), a proximity measure able to capture the locality. We denote b as the black-box and x as the instance we want to explain. To learn the local behavior of b, lime draws samples weighted by \(\pi _x\). It samples these instances around x by drawing nonzero elements of x uniformly at random. This gives to lime a perturbed sample of instances \(\{z \in \mathbb {R}^d \}\) to fed to the model b and obtain b(z). They are then used to train the explanation model \(g(\cdot )\): a sparse linear model on the perturbed samples. The local feature importance explanation consists of the weights of the linear model. A number of papers focus on overcoming the limitations of lime, providing several variants of it. dlime (Zafar and Khan 2019) is a deterministic version in which the neighbors are selected from the training data by an agglomerative hierarchical clustering. ilime (ElShawi et al. 2019) randomly generates the synthetic neighborhood using weighted instances. alime (Shankaranarayana and Runje 2019) runs the random data generation only once at “training time”. kl-lime (Peltola 2018) adopts a Kullback–Leibler divergence to explain Bayesian predictive models. qlime (Bramhall et al. 2020) also consider nonlinear relationships using a quadratic approximation. As presented in the following, lime can also be used on other data types. In Fig. 3 (upper part) are reported examples of lime¹² explanations relative to our experimentation on adult (a/b) and german (c/d).¹³ We predicted the same record using LG and CAT, and then we explained it. Interestingly, for adult (plots a/b), lime considers a similar set of features as important (even if with different values of importance) for the two models: on 6 features, only one differs. A different scenario is obtained applying lime on german (plots c/d): different features are considered important by the two classifiers. However, the confidence of the prediction between the two classifiers is quite different: both of them predict the output correctly, but CAT has a higher value, suggesting that this could be the cause of differences between the two explanations.

SHAP, SHapley Additive exPlanations (Lundberg and Lee 2017), is a local model-agnostic explanation method computing features importance by means of Shapley values,¹⁴ a concept from cooperative game theory. The explanations returned by shap are additive feature attributions and respect the following definition: \(g(z') = \phi _{0} + \sum _{i=1}^{M}\phi _{i}z'_{i}\), where \(z' \approx x\) as a real number, \(z' \in [0,1]\), \(\phi _i \in \mathbb {R}\) are effects assigned to each feature, while M is the number of simplified input features. shap retains three properties: (i) local accuracy, meaning that g(x) matches b(x); (ii) missingness, which allows for features \(x_i = 0\) to have no attributed impact on the shap values; (iii) stability, meaning that if a model changes so that the marginal contribution of a feature value increases (or stays the same), the shap value also increases (or stays the same). The construction of the shap values allow to employ them both locally, in which each observation gets its own set of shap values, and globally, by exploiting collective shap values. We highlight that shap can be realized through different explanation models that differ in how they approximate the computation of the shap values. In particular, there are five strategies: KernelExplainer is the model-agnostic one, while LinearExplainer, TreeExplainer, GradientExplainer, and DeepExplainer are model-specific. Besides, similarly to lime, shap can be used on other data types. We applied LinearExplainer to LG, TreeExplainer to XGB, and KernelExplainer to CAT. In Fig. 3 (lower part), we report the application of shap on adult through the force plot showing each feature contributes to push the output value away from the base value, which is an average among the training dataset’s output values. The red features are pushing the output value higher, while the ones in blue are pushing it lower. For each feature, it is reported the actual value for the record under analysis. Only the features with the highest shap values are shown in this plot. In the first force plot, the features that are pushing the value higher are contributing more to the output value: from a base value of 0.18, it is reached an actual output value of 0.79. In the force plot on the right, the output value is 0.0, and Age, Relationship, and Hours Per Week are contributing to pushing it lower. Figure 4 (left and center) depicts the shap values through a decision plots: the contribution of all the features are reported in decreasing order of importance. The line represents the feature importance for the record under analysis, and it starts at its actual output value. In the first plot, predicted as \(>50k\), Occupation is the most important feature, followed by Age and Relationship. For the second plot, Age, Relationship, and Hours Per Week are the most important ones. shap also offers a global interpretation of the model-driven by the local interpretations. Figure 4 (right) reports a global decision plot that represents the feature importance of 30 records of adult.

DALEX (Lipovetsky 2022) is a local and global, post-hoc, model-agnostic explanation method. dalex reveals the features importance through an implementation of a variable attribution approach (Robnik-Šikonja and Kononenko 2008) that consists of a decomposition of the model’s predictions in which each decomposition can be seen as a local gradient, and it is used to identify the contribution of each attribute. dalex also allows the calculus of shap values. In Fig. 5 are reported local explanations returned by dalex for XGB onadult. On the left are reported two explanations for a record classified as class \(> 50k\), while on the right, for one classified as \(< 50k\). On the top, there is a visualization based on Shapley values, which highlights as most important the feature Age (35 years old), followed by occupation. At the bottom, there is a Breakdown plot, in which the green bars represent positive changes in the mean predictions, while the red ones are negative changes. The plot also shows the intercept, which is the overall mean value for the predictions. It is interesting to see that Age and occupation are the most important features that positively contributed to the prediction for both plots. In contrast, Sex is positively important for Shapley values but negatively important for the Breakdown plot. In this case, there are important differences in the feature considered most important by the two methods: for the Shapley values, Age and Relationship are the two most important features, while in the Breakdown plot Hours Per Week is the most important one.

MAPLE (Plumb et al. 2018) is a local post-hoc model-agnostic explainer that can also be used as a transparent model due to its internal structure. It combines random forests with feature selection methods to return feature importance based explanations. In particular, maple is based on SILO, employed for obtaining a local training distribution based on the random forest leaves, and on DStump used to rank the features by importance. maple considers the best k features from DStump to solve a weighted linear regression problem. Similarly to lime, it returns these coefficients as features explanation.

CIU, Contextual Importance and Utility (Anjomshoae et al. 2020) is a local, post-hoc, model-agnostic explainer based on the idea that a feature that might be important in a context may be irrelevant in another one. ciu explains the model’s outcome based on the contextual importance (CI) approximating the overall importance of a feature in the current context, and on the contextual utility (CU) estimating how good the current feature values are for a certain output class. CI and CU are calculated through Monte Carlo simulations.

EBM, Explainable Boosting Machine (Nori et al. 2019) is an intrinsic local and global model specific method. ebm is a variant of a Generalized Additive Model (gam) (Hastie and Tibshirani 1990), i.e., a generalized linear model that incorporates nonlinear forms of the predictors. For each feature, ebm uses a boosting procedure to train the generalized linear model: it cycles over the features in a round-robin fashion to train one feature function at a time and mitigate the effects of co-linearity. In this way, the model learns the best set of feature functions, which can be exploited to understand how each feature contributes to the final prediction.

NAM, Neural Additive Models (Agarwal et al. 2021) is a local model-specific explainer defined as a variant of gam but tailored for neural networks. nam aims at combining the performance of deep neural networks with the inherent intelligibility of gam. nam is able to learn graphs that describe how the prediction is computed by training multiple deep neural networks in an additive fashion such that each network attends to a single feature.

CoFrNets Continued Fractions Nets (Puri et al. 2021) is similar to nam, but instead of approximating activations using neural networks, it uses continued functions. The output of a neuron is calculated as a continuous fraction of the previous one until it gets to the input layer. The propriety of continued fractions to represent every possible real number allows CoFrNets to express any possible function as in Neural Networks. On the other hand, since it is a simple fraction calculation, it is possible to compute the contribution of each input to the final output and produce feature importance explanations.

Features Importance-based Explainers Comparison. Feature importance-based explanation methods provide an importance value for each feature of the record in the input. The importance of the features is computed in different ways, depending on the kind of explanation methods exploited. The majority of the explainers are post-hoc and local, even if there are examples of methods that also provide global explanations, allowing an in-depth analysis of the overall behavior of the machine learning model (shap, dalex and ciu). Some explainers, such as lime and all its variants, create a synthetic neighborhood set used to train a surrogate model and extract the features’ importance from it. These methods are suitable for context in which the explanation is online, since they are very efficient, as they use randomization techniques and surrogate models that are very simple and quick to train. As a weak point, the randomicity and the simplicity of the surrogate models may not best represent the data space under analysis. On the other hand, explainers that do not require the creation of a surrogate model but are based on some mathematical procedure, such as game theory for shap, the decomposition of predictions exploiting local gradients for dalex, or Monte Carlo simulations for ciu might require a longer computational time w.r.t. lime-like approaches. In addition, their internal workings may be difficult to understand, shifting the difficulty from understanding the machine learning model to understanding how the explanation method works. Among feature importance-based explainers, model-specific explainers, such as nam and CoFrNets, are tailored for explaining neural networks and aim at approximating the activation functions. Overall, these explanation methods are fast, except for the model-agnostic variants of shap and dalex, because they might require a greater computing time due to their different approximations. Unfortunately, the output provided by these explainers is usually quite difficult to understand for non-experts since there are several variables, and the plots provided are usually non-self-explanatory. As an example, we can think of shap: in the plots in output, each feature importance is given by the output value, the base value, and, depending on the kind of explanator exploited, one or more arrays of feature importance.

Feature importance explainers, due to their complexity in the understating of the explanation, may be better suited for domain experts who know the meaning of the features employed, while they may be too difficult for ordinary end-users, especially when obtaining such importance values is complex.

Decision rules give the end-user an explanation about the reasons that lead to the final prediction. A decision rule r, also called factual or logic rule (Guidotti et al. 2019a), has the form \(p \rightarrow y\), in which p is a premise, composed of a Boolean condition on feature values, while y is the consequence of the rule. In particular, p is a conjunction of split conditions of the form \(x_i {\in } [v_i^{(l)}, v_i^{(u)}]\), where \(x_i\) is a feature and \(v_i^{(l)}, v_i^{(u)}\) are lower and upper bound values in the domain of \(x_i\) extended with \(\pm \infty \). An instance x satisfies r, or r covers x, if every Boolean conditions of p evaluate to true for x. If the instance x to explain satisfies p, the rule \(p \rightarrow y\) represents then a candidate explanation of the decision \(g(x) = y\). Moreover, if the interpretable predictor mimics the behavior of the black-box in the neighborhood of x, we further conclude that the rule is a candidate local explanation of \(b(x) = g(x) = y\). We highlight that, in the context of rules, we can also find the so-called counterfactual rules (Guidotti et al. 2019a). Counterfactual rules have the same structure as decision rules, with the only difference being that the consequence of the rule \(\overline{y}\) is different w.r.t. \(b(x) = y\). They are important to explain to the end-user what should be changed to obtain a different output. An example of a rule explanation is \(r = \{ age< 40, income < 30k, education \le Bachelor \}, y = deny \). In this case, the record \(\{ age = 18, income = 15k, education = High school \}\) satisfies the rule above. A possible counterfactual rule, instead can be: \(r = \{ income > 40k, education \ge Bachelor \}, y = allow \).

ANCHOR (Ribeiro et al. 2018) is a global and local model-agnostic method that outputs rules, called anchors, as explanations. The idea is that, for decisions on which the anchor holds, changes in the rest of the instance’s feature values do not change the outcome. Formally, given a record x, r is an anchor if \(r(x) = b(x)\). To obtain the anchors, anchor perturbs the instance x, obtaining a set of synthetic records employed to extract anchors with precision above a user-defined threshold. anchor exploits a multi-armed bandit algorithm (Katehakis and Jr. 1987) for the synthetic generation of the dataset and relies on a bottom-up approach and a beam search to find the anchors. Figure 6 reports some rules obtained by applying anchor to explain XGB trained on adult. The first rule has a high precision (\(0.96\%\)) but a very low coverage (\(0.01\%\)). It is interesting to note that the first rule contains Relationship and Education Num, which are the features highlighted by most of the explainers analyzed so far. In particular, in this case, for having a classification \(>50k\), the Relationship should be husband and the Education Num at least a bachelor’s degree. Education Num can also be found in the second rule, in which case has to be less or equal to College, followed by the Maritial Status, which can be anything other than married with a civilian. This rule has an even better precision (\(0.97\%\)) and suitable coverage (\(0.37\%\)).

LORE, LOcal Rule-based Explainer (Guidotti et al. 2019a), is a local model-agnostic explainer that provides explanations in the form of rules and counterfactual rules. lore is tailored explicitly for tabular data. It exploits a genetic algorithm to create the neighborhood of the record to explain. Such a neighborhood produces a more faithful and dense representation of the vicinity of x w.r.t. lime. Given a black-box model b and an instance x, with \(b(x) = y\), lore first generates a synthetic set Z of neighbors through a genetic algorithm. Then, it trains a decision tree classifier g on this set labeled with the black-box outcome b(Z). From g, it retrieves an explanation that consists of (i) a factual decision rule, that corresponds to the path on the decision tree followed by the instance x to reach the decision y, and (ii) a set of counterfactual rules, which have a different classification w.r.t. y. These counterfactual rules show the conditions that can be varied on x in order to change the output decision. In Fig. 6, we report the factual and counterfactual rules of lore for the explanation of the same records showed for anchor. It is interesting to note that, differently from anchor and the other models proposed above, lore explanations focus more on the Education Num, Occupation, Capital Gain and Capital Loss, while the features about the relationship are not present.

RuleMatrix (Ming et al. 2019) is a model-agnostic explainer that provides both local and global explanations specifically tailored for the visualization of the rules extracted. Given a training dataset and a black-box, rulematrix executes a rule induction step, in which a rule list is extracted by sampling the input data and their predicted label by the black-box. Then, the rules extracted are filtered based on thresholds of confidence and support. Finally, it outputs a visual representation of the rules.

Local rule-based explainers comparison The rule-based methods presented are all based on creating a surrogate model from which to extract the rules. In this category, we find anchor and rulematrix, which provide both local and global explanations by relying on simple rule extraction methods. The simplicity of these methods makes them efficient even if, as in the case of lime-like approaches, they may suffer in terms of the goodness of explanations provided. lore is the only explainer that provides only local explanations. Differently from the other methods, does not require to have access to the original training data, and, due to its synthetic generation process, provides more faithful explanations. Therefore it may be good in settings where the black-box training dataset is unavailable, while rulematrix and anchor need to access the training data. Rule-based explanations are considered closer to human reasoning w.r.t. the feature importance-based explanations. In addition, they exploit easy to understand mechanisms, allowing users of different backgrounds to understand how the explanation method works, increasing trust. However, these explainers usually require a greater running time w.r.t. the feature importance ones.

Local rule-based explainers produce logical rules which are close to human reasoning and make them suitable for non-experts.

Global tree-based explainers One of the most popular ways to generate explanation rules is by extracting them from a decision tree. In particular, due to the method’s simplicity and interpretability, decision trees explain black-box models’ overall behavior. Some explanation methods acting in this setting are model-specific explainers exploiting structural information of the black-box model under analysis. TREPAN (Craven and Shavlik 1995) is a model-specific global explainer tailored for neural networks. Given a neural network b, trepan generates a decision tree g that approximates the network by maximizing the gain ratio and the model fidelity. In particular, to leverage abstraction, trepan adopts n-of-m decision rules on which only n out of m conditions must be satisfied in order to fire the rule. DecText is a global model-specific explainer tailored for neural networks (Boz 2002). dectext resembles trepan with the difference that it considers four different splitting methods. Moreover, it also considers a pruning strategy based on fidelity to reduce the final explanation tree’s size. In this way, dectext can maximize the fidelity while keeping the model simple. Both trepan and dectext are presented as model-specific explainers, but they can be practically employed to explain any black-box as they do not use any internal information of neural networks. MSFT (Chipman et al. 1998) is a global, post-hoc, model-specific explainer for random forests that returns a decision tree. msft is based on the observation that, even if random forests contain hundreds of different trees, they are quite similar, differing only for a few nodes. Hence, it adopts dissimilarity metrics to summarize the random forest trees using a clustering method. Then, for each cluster, an archetype is retrieved as an explanation. CMM, Combined Multiple Model procedure (Domingos 1998), is another global, post-hoc, model-specific explainer for tree ensembles. The key point of cmm is the data enrichment. Given an input dataset X, cmm first modifies it n times. On the n variants of the dataset, it learns a black-box. Random records are generated and labeled using a bagging strategy on the black-boxes. The authors were able to increase the size of the dataset to build the final decision tree. STA, Single Tree Approximation (Zhou and Hooker 2016), is another global, post-hoc, model-specific explainer tailored for random forests. In sta, the decision tree is constructed by exploiting test hypothesis on the trees in the forest to find the best splits.

The explainers proposed are tailored for a specific machine learning model: trepan and DecText explain neural networks, while cmm and sta are tailored for random forests and msft is for any ensemble method. Among them, some explainers exploit an enrichment of the data to improve the extraction of the tree (cmm, trepan, DecText), while the others exploit the training dataset by applying some strategies based on dissimilarity metrics (msft) or test hypothesis (sta). Among the different methods, only DecText and trepan apply some strategies with the goal of maximizing the model fidelity, even if they are tailored for small feed-forward neural networks. The exploitation of trees to explain the global behavior of a more complex machine learning model have several benefits, such as fast computation and a simple process to extract explanations based on transparent strategies. However, the trees extracted may be very deep, making the explanation model difficult to understand even in cases of simple datasets. Furthermore, the effectiveness of such explanations for very deep feed forward networks has not been judged yet.

Global tree-based explainers produce a transparent model allowing the understanding of the general behavior of the black-box. Depending on the complexity of the tree, the actual ease of understanding of the explanation could be affected by this.

Global rule-based explainers In this section, we present global explainers that do not extract decision trees as a global interpretable model but as lists or sets of rules. The majority of the methods described in the following extract rules by exploiting ensemble methods or rule-based classifiers. The explainers considered are all agnostic. SkopeRules is a global, post-hoc, model-agnostic¹⁵ explainer on the rulefit (Friedman and Popescu 2008) idea to define an ensemble method and then extract the rules from it. skope-rules employs fast algorithms such as bagging or gradient boosting decision trees. After extracting all the possible rules, skope-rules removes rules redundant or too similar by a similarity threshold. Differently from rulefit, the scoring method does not solve the L1 regularization. Instead, the weights are given depending on the precision score of the rule. Scalable-BRL (Yang et al. 2017) is an interpretable rule-based model that optimizes the posterior probability of a Bayesian hierarchical model over the rule lists. The theoretical part of this approach is based on (Letham et al. 2015). GLocalX (Setzu et al. 2021) is a global model-agnostic post-hoc explainer that adopts the local to global paradigm, i.e., to derive a global explanation by subsuming local logical rules. GLocalX starts from an array of local explanation rules and follows a hierarchical bottom-up approach merging similar records expressing the same conditions. This small section comprises global explanation methods that extract rules in entirely different ways: either they exploit an ensemble method (skope-rules), a rule-based model (scalable-BRL) or several local explanations (GLocalX). In terms of goodness of explanations, skope-rules and scalable-BRL are tailored for an overall explanation of the machine learning model, focusing mostly on the data in input. GLocalX, instead, exploits local explanations and hence is able to tackle the problem from a different point of view, merging several local explanations. The output of these methods is a list of rules, and even if there are techniques to filter out meaningless rules, the complexity of the explanation produced may be huge.

Global rule-based explainers produce sets of rules describing the overall behavior of the model for each target class. Depending on the filters applied, the list of rules extracted may be long and difficult to understand.

Rules-based explainers comparison In this section, we presented a great variety of methods that provide logical rules as explanations exploiting different strategies. Independently from the strategy, due to the simplicity of the rules, they are often the preferred explanation for non-expert people. The majority of the explainers presented in this section are based on the extraction of decision trees as surrogate models (lore, trepan, cmm, sta, dectext, msft), or ensemble methods based on decision trees, such as skope-rules. The remaining methods that do not rely on decision trees extract the rules in other ways, such as rule-based classifiers (again a surrogate model), as in the case of anchor, scalable-brl and of rulematrix. To further increase the comprehensibility of the explanation, some explainers correlate the explanations by graphical visualizations, such as rulematrix, anchor, and skope-rules. Overall, the majority of the explainers require a long computing time due to the different enrichment of the data or the use of rule-based classifiers, which are among the longest interpretable models to train. Hence, they may be better fitted for offline explanations. Depending on the complexity of the machine learning model in input, the explanations may be complex, such as deep trees or long lists of rules.

Rules-based explanation methods extract rules exploiting different approaches, which may require a longer time w.r.t. feature importance methods, making them more suitable to offline settings. However, rules-based methods are tailored for common end-users due to their logical structure and simplicity.

A prototype, also called archetype or artifact, is a record highlighting the characteristics which identify a group of objects belonging to the same class. Prototypes serve as examples, i.e., the user can understand the black-box reasoning by looking at records similar to the prototype. Thus, a prototype is a local explanation. A prototype can be (i) a record of the training set close to the input data x, (ii) a centroid of a cluster to which the input x belongs to, or (iii) even a synthetic record generated following an ad-hoc process. Depending on the explanation method considered, different definitions and requirements to find a prototype can be considered. MMD-CRITIC (Kim et al. 2016) is a before the model explanation method which produces prototypes and criticisms as explanations using Maximum Mean Discrepancy (MMD) measure. While prototypes explain the dataset’s general behavior, criticisms represent records that are not well explained by the prototypes. mmd-critic selects prototypes by measuring the difference between the distribution of the instances and the instances in the whole dataset. The set of instances nearer to the data distribution are called prototypes, and the farthest are called criticisms. ProtoDash (Gurumoorthy et al. 2019) is a variant of mmd-critic. However, differently from mmd-critic, protodash also returns non-negative weights which indicate the importance of each prototype. PS, Prototype Selection (ps) (Bien and Tibshirani 2011) seeks a set of prototypes that better represent the data under analysis by solving a set cover optimization problem with constraints on the properties of the prototypes. After that, the prototypes are employed to learn a nearest neighbor rule classifier to be used as a model. TSP, Tree Space Prototype (Tan et al. 2020), is a local post-hoc explainer tailored for explaining tree ensemble methods. The goal of tsp is to find prototypes for each class in the tree space of the tree ensemble b w.r.t. a given notion of proximity between trees. Privacy-Preserving Explanations(ppe) (Blanco-Justicia et al. 2020) is a local post-hoc model-agnostic explainer that outputs prototypes and shallow trees as explanations while considering the concept of privacy in explainability while producing privacy protected explanations. The trade-off between privacy and comprehensibility is obtained through micro aggregation of the data, i.e., clustering. The clusters’ centroids are used as prototypes for the final explanation/prediction.

The strength of prototypes is the possibility of analyzing black-box behavior by comparison between the record under analysis and its analogs, which is a type of reasoning widely used by humans. Moreover, they allow data analysis before and after the black-box is applied. For this reason, in this section, we may find explanation methods that are before the model, i.e., they explain the dataset without considering the black-box model, such as mmd-critic, ps and protodash. On the other hand, local post-hoc explainers, such as tsp and ppe, provide prototypes based on the decisions of the black-boxes. Among the different methods proposed, one of the most promising ones is mmd-critic because it outputs both prototypes and criticisms. In this setting, we can also find a novel application, namely ppe, which produces privacy-protected prototypes, creating a link between two crucial ethical concepts: transparency and privacy. Indeed, using prototypes as explanations, although it may be useful for end-user understanding, may release sensitive information about the users in the training set when the explanation method exploits the training dataset.

Prototype-based explanations allow the users to reason by similarity and differences. Most of the methods in this setting are tailored to explain the data in input and not the black-box decisions.

Counterfactual explanations suggest what should be different in the input instance to change the outcome of the black-box model (Wachter et al. 2017; Lucic et al. 2020), i.e., they describe a dependency on the attributes that led to a particular decision. Counterfactual explanations can be considered as prototypes’ opposite. Thus, also counterfactuals are local explanations. In (Guidotti 2022), counterfactual explanations are formalized as follows. Given a black-box model b that outputs the decision \(y = b(x)\) for an instance x, a counterfactual explanation consists of an instance \(x'\) such that the decision for b on \(x'\) is different from y, i.e., \(b(x') \ne y\), and such that the difference between x and \(x'\) is minimal. The different values between x and a counterfactual \(x'\) reveal what should have been different in x for having a different outcome. An ideal counterfactual is minimal because it should alter the values of the variables as little as possible to find the closest setting under which y is returned instead of \(\lnot y\). Concerning counterfactual explanations, there are many properties that are desired for this kind of explanation and for the explanation methods returning them. Examples are validity, minimality, similarity, plausibility, discriminative power, actionability, causality, diversity, efficiency, robustness, etc. (Wachter et al. 2017; Karimi et al. 2020a; Kanamori et al. 2020). To better understand the complex context and the many available possibilities, we refer the interested reader to (Guidotti 2022; Artelt and Hammer 2019; Verma et al. 2020; Byrne and Johnson-Laird 2020) while we briefly present only the most representative methods in this category. WACH (Wachter et al. 2017) is among the first paper to propose a counterfactual explainer and probably the most famous one. The loss function minimized by (Wachter et al. 2017) is defined as \(\lambda (b(x') - y')^2 + d(x, x')\) where the first term is the quadratic distance between the desired outcome \(y'\) and the classifier prediction on \(x'\), and the second term is the distance d between x and \(x'\). \(\lambda \) balances the contribution of the first term against the second term. The distance function d adopted is a crucial characteristic in any counterfactual explainer. In (Wachter et al. 2017) is adopted the Manhattan distance weighted with the inverse median absolute deviation of each feature. CEM, Contrastive Explanations Method (Dhurandhar et al. 2018), is a post-hoc and model-specific explainer tailored for neural networks. In particular, cem can return Pertinent Positives (PP), which can be seen as prototypes and are the minimal and sufficient factors that have to be present to obtain the output y; and Pertinent Negatives (PN), which are counterfactuals factors, that should be minimally and necessarily absent. Given x, cem considers \(x_1 = x + \delta \), where \(\delta \) is a perturbation applied to x. cem is formulated as an optimization problem over the perturbation variable \(\delta \). C-CHVAE (Pawelczyk et al. 2020) is a local model-agnostic post-hoc explainer that accounts for plausibility when generating counterfactuals. Indeed, the loss function optimized controls that counterfactuals are not local outliers and that are close to correctly classified observations. Moreover, this method can generate counterfactuals without requiring a distance function for the input space at the cost of using a Variational AutoEncoder. DICE, Diverse Counterfactual Explanations (Mothilal et al. 2020) is a model-agnostic post-hoc explainer that solves an optimization problem with constraints to account for plausibility and diversity evaluated through distance functions. Plausibility avoids the generation of unfeasible counterfactuals, while diversity provides different ways of changing the outcome class. FACE, Feasible and Actionable Counterfactual Explanations (Poyiadzi et al. 2020) is a model-agnostic post-hoc explainer that focuses on returning actionable counterfactuals, i.e., records coherent with the input data distribution. In particular, face uncovers “feasible paths” for generating counterfactuals, i.e., the shortest path defined via density-weighted metrics starting from the input instance. Finally, it uses a shortest path algorithm to find all the records that satisfy the requirements. CFX (Albini et al. 2020) is a model-specific post-hoc explainer for Bayesian Network Classifiers. The explanations are built from relations of influence between variables, indicating the reasons for the classification. In particular, this method’s main achievement is that it can find pivotal factors for the classification task that, if removed, would lead to a different classification.

Counterfactual-based explanations are gaining attention during the past few years due to their ability to suggest what to do to achieve a different outcome w.r.t. the one predicted by the black-box. There are several characteristics to consider in a counterfactual, such as plausibility, which requires the explanations to be feasible, and actionability, so that the counterfactual can not suggest changing the values of unfeasible variables, such as age or sex. Satisfying these characteristics is of utmost importance because otherwise, the counterfactuals obtained may not be applicable or understandable by the end user. For example, a counterfactual might require changing age or height, factors that cannot be changed, thus making the counterfactual unfeasible. Among the methods proposed, c-chvae deals with the plausibility of the counterfactuals proposed, and face tackles both the plausibility and the actionability. The majority of the algorithms proposed solves an optimization problem based on a distance function and some perturbation of the original data (cem, wach, c-chvae) and only a few methods exploit different approaches, such as VAE, as c-chvae. Most of the proposed methods are local and post-hoc, with cfx and cem specifically designed for certain models, while the others are agnostic. Among the methods proposed, cem is a promising solution since it provides both prototypes and counterfactuals, allowing for an in-depth analysis, such as mmd-critic and lore.

Counterfactuals-based explanations allow the users to understand what to do to achieve a different outcome. This kind of reasoning is close to how human reason, hence it is becoming quite popular. To make counterfactuals as realistic as possible, they must meet criteria such as plausibility and actionability.

We validate explanation methods working on tabular data by considering the two metrics most widespread in the literature, i.e., fidelity and stability. We remark that we focus our benchmarking on the subset of explanation methods that are most widely adopted/extended in the literature and work according to different ideas to retrieve explanations. In particular, in this section, we focus on lime, shap,¹⁶anchor and lore. Another limitation in the benchmarking is that even though these methods are acting on the same data type, the explanations returned are of a different type. Thus, it is not only possible to compare explainers providing different types of explanations. The results of the fidelity are reported in Table 2. The fidelity values are above 90% for all the methods highlighting that the local surrogates are good at mimicking the black-box models. Regarding the feature importance-based models, lime shows higher values of fidelity w.r.t. shap, especially for adult. In particular, shap has lower values for CAT (both german and adult), suggesting that it may not be good in explaining this kind of ensemble model. Concerning rule-based models, the fidelity is high for both of them. However, we notice that anchor shows lower values of fidelity for CAT-german, a behavior which is similar to the one of shap. Besides fidelity, we compare lime and shap also on faithfulness and monotonicity. Overall, we did not find any model to be monotonic, and hence we do not report any results. On the other hand, the results for the faithfulness are reported in Table 2. For adult, the faithfulness is quite low, especially for lime. The explainer with the highest faithfulness is shap explaining CAT. Regarding german, instead, the values are higher, highlighting a better faithfulness overall. However, also for this dataset, shap has a better faithfulness w.r.t. lime. In Table 3 are reported the results obtained from the analysis on the stability: a high value means that the explainer presents high instability, meaning that we can have quite different explanations for similar inputs. None of the methods is remarkably stable according to this metric. This weakness is widely shared by many explainers independently from the data type and explanation type. Therefore, an important insight from these experiments is to work toward the stabilization of these procedures. Table 4 shows the explanation runtime approximated as order of magnitude. Overall, feature importance explanation algorithms are faster w.r.t. the rule-based ones. In particular, shap is the most efficient, followed by lime. We remark that the computation time of lore depends on the number of neighbors to generate exploiting a genetic algorithm (in this case, we considered 1000 samples). anchor, instead, requires a minimum precision as well as skoperule (we selected min precision of 0.40).

Overall, there are many explanation methods available for tabular data. As already stated, we focused our analysis and benchmark on the most widely used. We redirect the interested reader to our XAI Living Survey for more methods and details. Depending on the final user, different kinds of explanations are better suited than others. During the past few years, counterfactuals have witnessed great interest due to their logical structure and the possibility of explaining what to do instead of obtaining another prediction. In our opinion, even if rules, prototypes, and counterfactuals seem to be the best solutions w.r.t. human readability, features importance-based explainers are still the most widely used mainly due to the fact that they were the first proposed, with well-maintained libraries, and that are typically faster than the others. Moreover, regarding rules, prototypes, and counterfactuals, there are still several open questions and challenges, such as improving the efficiency and effectiveness as well as considering the constraints of the domain in which the model is being employed.

A Saliency Map (SM) is an image in which a pixel’s brightness represents how salient the pixel is. Formally, a SM is modeled as a matrix S, which dimensions are the sizes of the image for which we want to explain a decision, and the values \(s_{ij}\) are the saliency values of the pixels ij. The greater the value of \(s_{ij}\), the bigger the saliency of that pixel. To visualize a SM, divergent color maps are typically used with colors ranging, for instance, from red to blue. A positive value (red) means that the pixel ij has contributed positively to the classification, while a negative one (blue) means that it has contributed negatively. Examples of SM are reported in Fig. 7. There are two big families of methods for creating SMs. The first one assigns to every pixel a saliency value. The second one segments the image into different pixel groups using a segmentation method, and then it assigns a saliency value for each segmented group. We underline that, from a pure interpretability perspective, SMs are the correspondent of FI for images. Indeed, as shown in the following, data-agnostic approaches that can be used for extracting FI explanations on tabular data can also be used for extracting SM on images.

LIME is a local post-hoc model-agnostic explainer (already presented in Sect. 5) that can also be used to retrieve SM for image classifiers. In particular, lime returns segmentation-based SM where each segment is called superpixel. After the input image segmentation, lime adopts a one-hot vector representation of the image where the input image is a vector composed of m ones if m is the number of segments identified. Then, it creates the synthetic neighborhood by randomly substituting the superpixels with a uniform, possibly neutral, color and by also storing the one-hot representations. The neighborhood of synthetic images is then fed into the black-box, and the one-hot representation of the neighborhood, together with the black-box prediction, is used to train a sparse linear model. Finally, the coefficients of the linear model are used as the importance of the superpixels. Examples of the explanations returned by lime are shown in the second row of Fig. 7. A critical aspect for obtaining a good explanation with this approach is the choice of the segmentation algorithm and its hyper-parameters. Indeed, for small-resolution images, the segmentation in lime does not work out of the box, resulting in the algorithm selecting all the images as a superpixel. Recently, many research improved and extended lime for image classifiers¹⁸  (Shi et al. 2020; Peltola 2018; Zafar and Khan 2019; Bramhall et al. 2020).

LRP, Layer-wise Relevance Propagation (Bach et al. 2015), commonly known as \(\epsilon \) -lrp, is a local post-hoc model-specific explainer that is designed for images but can be practically applied for any data type. \(\epsilon \) -lrp was introduced for feed-forward neural networks (Arras et al. 2017) and then adapted to different type of models. \(\epsilon \) -lrp decomposes the prediction y backward using local redistribution rules until it assigns a relevance score \(R_i\) to each pixel value. The simple \(\epsilon \) -lrp rule redistributes relevance from layer \(l + 1\) to layer l is: \(R_i = \sum _j{\frac{a_iw_{ij}}{\sum _i{a_iw_{ij}}+\epsilon }}R_j\) where \(a_i\) is the neuron activations at layer l, \(R_j\) the relevance scores associated to the neurons at layer \(l + 1\), \(w_{ij}\) the weight connecting neuron i to neuron j, and \(\epsilon \) is added to prevent division by zero. Intuitively, this rule redistributes relevance proportionally from layer \(l + 1\) to each neuron in l based on the connection weights. The final explanation is the relevance of the input layer. Figure 7 shows examples of \(\epsilon \) -lrp SM in the third row. As with all the pixel-wise explanation methods, \(\epsilon \) -lrp works very well on datasets with low-resolution images like mnist, while the explanations returned for more complex images such as those of cifar and imagenet are quite unclear and only limitedly interpretable. In the literature, are introduced other variations of the lrp algorithm. \(\gamma \)-lrp favors the effect of positive contributions over negative contributions by separating the weights \(w_{ij}\) into \(w^{-}_{ij}+w^{+}_{ij}\) and adding a multiplier to the positive ones: \(w^{-}_{ij}+\gamma w^{+}_{ij}\) Another variant of \(\epsilon \) -lrp is spray (Lapuschkin et al. 2019), which builds a spectral clustering on top of the local instance-based \(\epsilon \) -lrp explanations. Similarly to (Li et al. 2019), it starts with the \(\epsilon \) -lrp of the input instance and finds the LRP attribution relevance for a single input of interest x.

INTGRAD, Integrated Gradient (Sundararajan et al. 2017), similarly to \(\epsilon \) -lrp, is a local post-hoc model-specific data-agnostic explainer designed for images. intgrad utilizes the gradients of a black-box providing access to the gradient, such as a neural network, along with sensitivity techniques. For this reason, it can be applied only to differentiable models. Formally, given b and x, and let \(x'\) be the baseline input,¹⁹intgrad constructs a path from \(x'\) to x and computes the gradients of points along the path. For images, the points are taken by overlapping x on \(x'\) and gradually modifying the opacity of x. Integrated gradients are obtained by cumulating the gradients of these points. Formally, the integrated gradient along the \(i^{th}\) dimension for an input x and baseline \(x'\) is defined as follows. Here, \(\partial b(x)/\partial x_i\) is the gradient of b(x) along the \(i^{th}\) dimension. The equation for computing the scores is: \(e_i(x) = (x_i-x'_i) \int _{\alpha =0}^{1} \tfrac{\partial b(x' + \alpha (x-x'))}{\partial x_i }~d\alpha \). Examples of SM produced by intgrad are in Fig. 7. They tend to have more uniform pixels with a similar saliency than \(\epsilon \) -lrp. As in \(\epsilon \) -lrp, intgrad highlighted the background when explaining the “deer” image. However, an arbitrary choice of baselines could cause issues. For example, a black baseline image could cause intgrad to lower the importance of black pixels in the source image. This problem is due to the difference between the image’s pixel and the baseline (\(x_i-x_i'\)) present in the integral equation. Expected Gradients (Erion et al. 2019) tries to overcome this problem by averaging intgrad to different baselines. A different variation of intgrad is called Adversarial Gradient Integration (AGI). An adversarial example of an image is the image with a different class, most similar to the original one. AGI (Pan et al. 2021) integrates the gradients from adversarial examples of different classes to the target example along the curve of the steepest ascent. The contributions from the different examples are then summed up to obtain the final SM. By not relying on the choice of references, it is possible to reduce the sparsity of intgrad.

DEEPLIFT (Shrikumar et al. 2017), is local post-hoc model-specific explainer for differentiable models. It computes SMs in a backward fashion, similarly to \(\epsilon \) -lrp, but it uses a baseline reference like intgrad. deeplift exploits the slope, instead of the gradients, which describes how the output \(y=b(x)\) changes as the input x differs from a baseline \(x'\). Like \(\epsilon \) -lrp, an attribution value r is assigned to each unit i of the neural network going backward from the output y. This attribution represents the relative effect of the unit activated at the original network input x compared to the activation at the baseline reference \(x'\). deeplift computes the starting values of the last layer L by the difference between the output of the input and baseline. Then, it uses the following recursive equation to compute the attribution values of layer l using the attributions of layer \(l+1\) to obtain the values of the starting layer: \( r_i^{(l)} = \sum _j \frac{a_{ji} - a'_{ji}}{\sum _{i} a_{ji} - \sum _{i} a'_{ji}} r_j^{(l+1)}, a_{ji} = w^{(l+1,l)}_{ji} x_i^{(l)}, a'_{ji} = w^{(l+1,l)}_{ji} x_i^{'(l)} \) where \(w_{ij}^{l+1,l}\) are the weights of the network between the layer l and the layer \(l+1\), and a are the activation values. As for intgrad, picking a baseline is not trivial and might require domain experts. From Fig. 7, we notice that the SMs obtained with deeplift are very similar to those obtained with \(\epsilon \) -lrp.

SMOOTHGRAD (Smilkov et al. 2017) is a local post-hoc model-specific explanation method. Since SMs tend to be noisy, especially for pixel-wise SMs, smoothgrad tries to overcome produce less noisy SM through smoothing. Usually, a SM is created directly on the gradient of the black-box output w.r.t. the input \(\partial y/\partial x\). smoothgrad augments this process by smoothing the gradients with a Gaussian noise kernel. More in detail, it takes the image x, applies Gaussian noise to it, and retrieves the SM for every perturbed image, using the gradient. The final SM is an average of these perturbed SMs. Formally, given a saliency method f(x) which produces a SM s, its smoothed version \(\hat{f}\) can be expressed as: \( \hat{f} = \frac{1}{n} \sum _1^n f(x + \mathcal {N}(0, \sigma ^2))\) where n is the number of samples, and \(\mathcal {N}(0, \sigma ^2)\) is the Gaussian noise. In (Adebayo et al. 2018, 2020), some weaknesses of smoothgrad are shown: people tend to evaluate SMs on what they are expected to see. For example, in a bird image, we want to see the shape of a bird. However, this does not mean that this is what the network is looking at. Figure 8 highlights this problem. We obtained the SMs taking the gradient of the output w.r.t. the input, and then we used smoothgrad. We observe that the SMs completely changed their behavior, moving in the direction of the subject.

SHAP provides two versions that can be employed for deep networks tailored for image classification: deep-shap and grad-shap. Therefore, even though shap is a local post-hoc model-agnostic explainer, the deep-shap and grad-shap versions are model-specific implementations. deep-shap is a high-speed approximation algorithm for shap values in deep learning models that builds on a connection with deeplift. The implementation is different from the original deeplift by using as baseline a distribution of background samples instead of a single value and using Shapley equations to linearise non-linear components of the black-box such as max, softmax, products, divisions, etc. grad-shap, instead, is based on intgrad and smoothgrad (Sundararajan et al. 2017; Smilkov et al. 2017). intgrad values are a bit different from shap values and require a single reference value to integrate. As an adaptation to make them approximate shap values, grad-shap reformulates the integral as an expectation and combines that expectation with sampling reference values from the background dataset as in smoothgrad. Examples of explanations of deep-shap and grad-shap are shown in Fig. 9. grad-shap produces a single SM, while deep-shap produces a SM for each class in the input image. Even if more expensive to compute, having a SM for each class could be useful to compare the importance of the predicted class with different classes.

XRAI (Kapishnikov et al. 2019) is based on intgrad and inherits its properties. Differently from intgrad, xrai first over-segments the image. It iteratively tests each region’s importance, fusing smaller regions into larger segments based on attribution scores. xrai follows three steps: segmentation, get attribution, and selecting regions. The segmentation is repeated several times with different segments to reduce the dependency on image segmentation. For attribution, xrai uses intgrad with black and white baselines averaged. Finally, to select regions, xrai leverages the fact that, given two regions, the one that sums to the more positive value should be more important to the black-box. Thus, xrai starts with an empty mask and then selectively adds the regions that yield the maximum gain in the total attributions per area. From Fig. 7, we notice how the SMs obtained by xrai are very different from the others already presented. Like all segmentation methods, xrai performs best when high-resolution images are available. However, because the segmentation is performed on the values obtained from intgrad and not on the raw image, good results are obtained even with low-resolution images.

GRAD-CAM (Selvaraju et al. 2020) is a local post-hoc model-specific explainer for image data. It uses the gradient information flowing into the last convolutional layer of a CNN to assign saliency values to each neuron for a particular decision. Convolutional layers naturally retain spatial information in fully-connected layers, so it assumes that the last convolutional layers have the best compromise between high-level semantics and detailed spatial information. To create the SM, grad-cam takes the feature maps created at the last layer of the convolutional network a. Then, it computes the gradient of an output of a particular class \(y^c\) for every feature map activations k, i.e.,  \(\partial y^c/\partial a^k\). This equation returns a tensor of dimensions [k, v, u] where k is the number of features maps and u and v are the height and the width of the image. grad-cam compute the saliency value for every feature map by pooling the dimensions of the image. The final heatmap is calculated as a weighted sum of these values. This results in a coarse heatmap of the same size as the convolutional feature maps. An up-sampling technique is applied to the final result to produce a map of the initial image dimension. From Fig. 7 is clear that this coarse grain heatmap style is very characteristic of grad-cam. These heat maps highlight very different parts of the image compared to other methods.

GRAD-CAM++ (Chattopadhay et al. 2018) extends grad-cam solving the following issue. The spatial footprint in an image is essential for grad-cam’s visualizations to be robust. Hence, if multiple objects have slightly different orientations or views, different feature maps may be activated with differing spatial footprints. The one with lesser footprints fades away in the final sum. grad-cam++ fix this problem by taking a weighted average of the pixel-wise gradients. In particular, grad-cam++ reformulates grad-cam by explicitly coding the structure of the weights \(\alpha _k^c\) as: \(\alpha _k^c = \sum _{i}\sum _{j}w_{ij}^{kc}\cdot \text {ReLU}\left( \partial y^c/\partial a_{ij}^k\right) \) where ReLU is the Rectified Linear Unit activation function, and \(w_{ij}^{kc}\) are the weighting co-efficients for the pixel-wise gradients for class c and convolutional feature map \(a^k\). The idea is that \(w_k^c\) captures the importance of a particular activation map \(a^k\), and positive gradients are preferred to indicate visual features that increase the output neuron’s activation rather than those that suppress it. Besides grad-cam++, there are many variations of pixel masking methods based on grad-cam. Score-CAM (Wang et al. 2020) gets rid of the dependence on gradients by obtaining the weight of each activation map through its forward passing score on the target class, the final result is obtained by a linear combination of weights and activation maps. Eigen-CAM (Muhammad and Yeasin 2020) extracts the importance scores by projecting the last convolutional layer into the first eigenvector extracted from the same layer. The scores produced appear more uniform and less sparse. Ablation-CAM (Desai and Ramaswamy 2020) uses ablation analysis, i.e., setting activation value to zero, to determine the importance of individual feature map units with respect to class. Extreme Perturbation (Fong et al. 2019) are regions of an image that, for a given area, maximally affect the activation of a class in the last layer of a neural network. As the perturbation area increases, the explainer reveals more of the image in order of decreasing importance. CXPlain (Schwab and Karlen 2019) frames the task of providing explanations as a causal learning task. It trains a causal explainer that learns to estimate to what degree certain inputs cause outputs in another model.

RISE (Petsiuk et al. 2018) is a local post-hoc model-agnostic explainer for image data. To produce a SM for an image x, rise generate N random mask \(M_i \in [0,1]\) from Gaussian noise. The input x is element-wise multiplied with these masks \(M_i\), and the result is fed to the base model. The SM is obtained as a linear combination of the masks \(M_i\) with the predictions from the black-box corresponding to the respective masked inputs. The intuition behind this is that \(b(x \odot M_i)\) is high when pixels preserved by mask \(M_i\) are essential.

Saliency maps-based explainers comparison Saliency Maps (SM) are the most diffused explanations for images. The literature presents a multitude of explainers that are capable of producing such a type of explanation. As seen in Fig. 7, lime can lead to useless SM for mnist and cifar as it segments the images in superpixels big as the whole image in some cases. On the other hand, those produced by xrai are much more clear. lime computes the segmentation at the very beginning of the algorithm on the raw images. Thus, for low-resolution images, segmentation algorithms are more difficult to calibrate. xrai instead firstly compute intgrad values and then agglomerate them using segmentation. This result seems to be much clearer, even with very small images. In general, we observe that segmentation methods work best for high-resolution images where the concept of the image can be easily separated. For instance, in the SM of the “seashore” image produced by xrai is very clear how the method selected three parts of the image: the horizon, the sea, and the promontory. Since pixel wise-methods produce SMs in terms of single pixels, which are low-level features, they are useful only for an expert user who wants to check the robustness of the black-box. Overall, we deduce that SMs returned by the segmentation methods are more human-friendly than the ones returned by pixel-wise methods.

Pixel based explainers like deep-shap, grad-shap, deeplift, intgrad, and \(\epsilon \) -lrp, typically return very similar results. deeplift and \(\epsilon \) -lrp return similar results, probably due to their similarity in the computation of the SM values. However, deeplift is model-agnostic, while \(\epsilon \) -lrp is model-specific, and it needs adjustments for nonstandard neural networks. In conclusion, deeplift result to be the best competitor. grad-cam, grad-cam++, and rise return SMs that could resemble those returned by segmentation-based explainers. This is due to the fact that scores of these explainers are computed on smaller layers with lower resolution and then interpolated on the original data resolution. grad-cam++ produces better images than grad-cam due to the usage of the second order derivative. Another problem with SMs is the confirmation bias, i.e., a user can hardly realize if a SM is a good explanation or only shows what the user wants to see (Adebayo et al. 2018). smoothgrad and most guided methods that use the target label to alter the SM suffer from this bias, moving the salient values from the background to the subject.

Analyzing the images by dataset: in cifar we notice that all the methods highlight the background of the images in particular in the “deer” class. This result is a problem in the learning phase of the black-box and should not be referred to as problems of the explainers. On the other hand, for imagenet, we observe very different SMs. For instance, for the ice hockey image in Fig. 7, the class in the dataset is “puck”, i.e., the hockey disk. lime highlights the ice as important, while xrai and grad-cam++ highlight the stick of the player, grad-cam highlights the fans while rise the hockey player. Thus, for the same image, we can obtain very different explanations, further highlighting the fragility of the SMs. Regarding the second image of imagenet (the second from the right), we can observe that all the methods capture the same pattern. A straw hat in the background triggered the class “shower cap” while the correct one was “mask”. Finally, in the “seashore” of imagenet, we have an island in the sea. The top three predicted classes are seashore (0.91), promontory (0.04), and cliff (0.01). Half of the tested methods like lime, smoothgrad, rise, and grad-cam were fooled that the promontory is important to the class “seashore”. We can conclude that SMs are very fragile when we have multiple classes in the image, even if these classes have a very low predicted probability.

Segmentation methods are more human-understandable than pixel-wise methods. Guided propagation methods can hardly be trusted due to confirmation bias, and therefore it is better not to adopt them.

Most ML models are designed to operate on low-level features like edges and lines in a picture that does not correspond to high-level concepts that humans can easily understand. In (Adebayo et al. 2018; Yang and Kim 2019), the authors pointed out that feature-based explanations applied to state-of-the-art black-box models can yield non-sensible explanations. For example, we can consider SM as low-level explanations for images, as they assign to every pixel a saliency value. Although it is possible to look at every pixel and infer their numerical values, these make little to no sense to humans: we do not say that the 5th pixel of an image has a value of 28. On the other hand, Concept Attribution (CA) methods quantify, for instance, how much the concepts “stripes” has contributed to the class prediction of “zebra”. Indeed, CA-based explanation methods construct the explanation based on human-defined concepts rather than representing the inputs based on features and internal model (activation) states. Hence, this idea of high-level features might be more familiar to humans, that can be more likely to accept it (Hartmann et al. 2022; Renard et al. 2019). Formally, given a set of images belonging to a concept \({[x^{(1)}, x^{(2)}, \dots , x^{(i)}] \text { with } x^{(i)} \in C}\), CA methods can be thought as a function \(f: (b,[x^{(i)}]) \rightarrow e\) which assign a score e to the concept C basing on the predictions and the values of the black-box b on the set \([x^{(i)}]\).

TCAV, Testing with Concept Activation Vectors (Kim et al. 2018) is a global post-hoc model-agnostic explainer for image classifiers that provides a quantitative explanation of how important a concept is for the prediction. In tcav, every concept is represented by a particular vector called Concept Activation Vectors (CAVs) created by interpreting an internal state of a neural network in terms of human-friendly concepts. tcav uses directional derivatives to quantify the degree to which a user-defined concept is vital to a classification result. For example, how sensitive a prediction of “zebra” is to the presence of “stripes”. tcav requires two main ingredients: (i) concept-containing inputs and negative samples, i.e., random inputs, and (ii) pre-trained ML models on which the concepts are tested. Inputs containing concepts and random inputs are fed into the black-box to obtain predictions that will be used by tcav to test the ability of the machine learning model to capture a particular concept. Then, a linear classifier is trained to distinguish the activation of the network due to concept-containing vs. random inputs. The result of this training is concept activation vectors (CAVs). Once CAVs are defined, the directional derivative of the class probability along CAVs can be computed for each instance that belongs to a class. The “concept importance” for a class is computed as a fraction of the class instances that get positively activated by the concept containing inputs vs. random inputs. In Fig. 10, we report an example of tcav explanation. We remark that to obtain this explanation, the user must collect concepts like “ice”, “hockey player” and “fans”. Then tcav computes the scores unveiling which one has more impact on the prediction.

Several extensions of tcav are present in the literature. ACE, Automated Concept-based Explanation (Ghorbani et al. 2019), is the evolution of tcav, and it does not require concepts in input as it can automatically discover them. A set of images from the same class is segmented with multiple resolutions resulting in a pool of segments all coming from the same class. Then the activation space of one bottleneck layer of a CNN classifier is used as a similarity space. After that, similar segments are clustered in the activation space, and, for each concept, its tcav importance score is computed given its examples segments. ConceptSHAP (Yeh et al. 2020) is an evolution of SHAP which tries to define an importance score for each concept discovered. conceptshap finds the importance of each individual concept from a set of m concept vectors \(C_s = \{c_1, c_2, \dots , c_m\}\) by utilizing Shapley values. Similar to ace, conceptshap aims at having concepts clustered to certain coherent spatial regions. CaCE, Causal Concept Effect (Goyal et al. 2019) is another variation of tcav that looks at the causal effect of the presence or absence of high-level concepts on the black-box prediction. Indeed, tcav can suffer from confounding concepts that could happen if the training data instances have multiple classes, even with a low correlation. PACE (Kamakshi et al. 2021) is a variation of ace that introduce the use of the black-box for identifying concepts by automatically extracting small sub-regions of the image, called concepts, relevant to the black-box prediction. In (Shen et al. 2021) is presented ICNN, an interpretable method to modify traditional CNNs into interpretable CNN. In an interpretable CNN, each filter in a high convolutional layer represents a specific object part. An interpretable CNNs use the same training data as ordinary CNNs without additional annotations of object parts or textures for supervision. The interpretable CNN automatically assigns each filter in a high convolutional layer with an object part during the learning process. The explicit knowledge of interpretable CNN can help people understand their logic, i.e., what patterns are memorized by the CNN for prediction.

Concept attribution-based explainers comparison SMs are the best known and most widely used type of explanation. However, they are very fragile and suffer from interpretation problems as their usage from a cognitive perspective is unclear. On the other hand, the usage of concepts is a new approach that seeks to overcome SMs problems by producing high-level explanations that are more understandable by the end user. Concept-based explanations are a very recent type of explanation for images, and they have potential improvements. It is the first step in the direction of human-like explanations. Human-friendly concepts make it possible to build straightforward and valuable explanations. Humans still need to map images to concepts, but it is a small price to pay to augment human–machine interaction. Generally, concept attribution explainers compute a score that evaluates the probability that a selected concept has influenced the prediction. The main problem, for now, is that this concept has to be manually selected. tcav is the most advanced algorithm. However, the concepts need to be provided as a set of images reflecting the selected concept, e.g., a set of ice images, a set of people, and a set of stripes. Several variations of tcav have been introduced to automate the concept selection phase. PACE is the most advanced search when looking for concepts concerning black-box behavior. Researchers are focusing on finding concepts in an automated way, but for the time being, there is no explainer to ensure finding a concept that is humanly understandable.

There is a need to build an explanation in terms of higher features called concepts for a general audience.

Another valid explanation type for images is a set of prototypical images that represent a particular class. Human reasoning is often prototype-based, using representative examples as a basis for categorization and decision-making. Similarly, prototype explanation models use representative examples to explain and cluster data. MMD-CRITIC (Kim et al. 2016), already presented in Sect. 5, is an interpretable approach that can be applied to retrieve image prototypes and criticisms. In Fig. 11 is presented an application of mmd-critic on cifar. We can extract some interesting knowledge from these methods. For example, from the prototype set, we can deduce that birds usually stand on a tree or fly in the sky, while, in the criticism images, we see that planes are all on a white background or have a different shape from the usual one used for passengers. Influence Functions (Koh and Liang 2017) is a global post-hoc model-agnostic explainer that tries to find the images most responsible for a given prediction through influence functions. The usage of influence functions is a technique from robust statistics to trace a model’s prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Visualizing the training points most responsible for a prediction can be useful for more in-depth insights into the black-box behavior. PROTOPNET (Chen et al. 2019) is a global interpretable model for image data that aims at identifying prototypical parts of images (named prototypes) and using them to implement an interpretable classification process. A special deep learning architecture is designed to retrieve these prototypes. The network learns a limited number of prototypical parts from the training set and then identifies parts on the test image that look like the prototypical parts. Then, it predicts based on a weighted combination of the similarity scores between parts of the image and the learned prototypes.

Prototype-based explainers comparison Prototype explanations are not widespread as explanations for image classifiers. The explainers presented have very different approaches: mmd-critic use the Maximum Mean Discrepancy measure to select the prototype set, protopnet instead used part of the image as prototypes. mmd-critic is faster than protopnet but the explanations provided are more obscure.

Explanation prototypes are not very common for images because the usefulness of such explanations is not clear.

In parallel with prototypes, also counterfactuals are another widely adopted form of explanation for images. In this setting, counterfactual explainers for images produce images similar to the input image but with a different prediction from the black-box under analysis. Therefore, also in this case, counterfactual methods are post-hoc explainers. Similarly to SM, we can distinguish between counterfactual explainers altering single pixels against those altering the whole image or part of it. CEM, Contrastive Explanation Method (cem) (Dhurandhar et al. 2018), already presented in Sect. 5, is a local post-hoc model-specific counterfactual explainer that can also be applied on image data. In this setting, Pertinent Positives (PP) or Pertinent Negatives (PN) are the pixels that lead to the same or a different class w.r.t. the original instance. To create PP’s and PN’s, feature-wise perturbation is done by keeping the perturbations sparse and close to the original instance through an objective function that contains an elastic net \(\beta L_1 + L_2\) regularizer and also consider an auto-encoder trained to reconstruct images of the training set. As a result, the perturbed instance lies close to the training data manifold. In Fig. 12, we can see that very few pixels are obtained as the counterfactual explanation on mnist. However, from a human perspective, this approach might seem much more adversarial than useful for an explanation (Guidotti 2022). An extension of CEM to resolve this problem is presented in (Luss et al. 2021), where the authors leverage the usage of latent features created by a generative model to produce more trustful perturbations. L2X (Chen et al. 2018) is a local post-hoc model-agnostic explanation method that searches for the minimal number of pixels that change the classification. It is based on learning a function for extracting a subset of the most informative features for each given sample using Mutual Information. l2x adopts a variational approximation to efficiently compute the Mutual Information and gives a value for a group of pixels called patches. If the value is positive, a group contributed positively to the prediction. Otherwise, it contributed negatively. Guided Prototypes, Interpretable Counterfactual Explanations Guided by Prototypes (guidedproto) (Looveren and Klaise 2021) proposes a local post-hoc model-specific explainer that perturbs the input image by using a loss function \(\mathcal {L} = cL_{pred}+\beta L_1 + L_2\) optimized using gradient descent. The first term, \(cL_{pred}\), encourages the perturbed instance to predict another class then x, while the others are regularisation terms. In Fig. 12, we show the application of guidedproto on mnist. It is interesting to notice that the counterfactuals unveil how easy it is to change the class with very few pixels. However, this kind of explanation is not easily human understandable because the few pixels modified can barely be noticed by human eyes. ABELE, Adversarial black-box Explainer generating Latent Exemplars) (Guidotti et al. 2019b), is a local post-hoc model-agnostic explainer that produces explanations composed of: (i) a set of exemplars and counter-exemplar images, i.e., prototypes and counterfactuals, and (ii) a SM. abele exploits an adversarial autoencoder (AAE) to generate the synthetic images used in the neighborhood to train the surrogate model used to explain x. Indeed, it builds a latent local decision tree that mimics the behavior of b and selects prototypes and counterfactuals from the synthetic neighborhood exploiting the tree. Finally, the SM is obtained by a pixel-by-pixel difference between x and the exemplars. In Fig. 12, we report an example of the application of abele on mnist.

Counterfactual-based explainers comparison The goal of counterfactual explainers is to produce examples similar to the input but with a different predicted class. They are more intuitive than the other types of explanations because counterfactual thinking is typical from the human cognitive point of view. There is a wide offer of counterfactual explainers (Guidotti 2022), and it is difficult to define which counterfactual explainer is better than the others for the image domain. If the priority is the execution time, then cem is the fastest method. Methods such as abele and guided prototypes create more realistic counterfactuals but require training additional models such as autoencoders which are notoriously difficult to train (Doersch 2016).

Counterfactual explanations are more user-friendly than prototypes and other forms of explanations because they highlight the changes to make to obtain the desired prediction.

To quantitatively investigate the performance of the SM explainers analyzed, we computed the deletion and the insertion metric, presented in Sect. 4. For the computation of the deletion metric, we substitute with black pixels the original pixels in increasing order of importance w.r.t. the scores given by the SM. On the other hand, for the insertion metric, we blur the whole image with a Gaussian Kernel, and then we slowly insert highly salient pixels w.r.t the SM. For every substitution, we query the black-box with the image, and we measure the performance. The final score is obtained by taking the area under the curve (AUC) (Hand and Till 2001) as a function of the percentage of pixels removed/inserted. In Fig. 13, we show an example of these metrics computed on an image of imagenet. Table 6 reports the average results of the calculation of these metrics for a set of 100 randomly selected images for every dataset. Independently from the explainer adopted, we notice that insertion scores decrease while augmenting the dataset dimensions because we have higher information and more pixels have to be inserted to increase the performance. On the other hand, the deletion scores decrease. This might be tied to the fact that since we have more information, it is easier to decrease the performance. The best explainers are highlighted in bold. We notice that rise is the best approach overall, followed by intgrad, deeplift, and \(\epsilon \) -lrp. We notice that all these approaches are pixel-wise-based methods. Thus it seems that this kind of evaluation can be advantageous for these explainers. On the contrary, the segmentation-based explainers lime and xrai struggle in general and even more when handling low-resolution images.

Table 7 shows the runtime for the image explainers experimented with. For imagenet, we only tested SMs methods due to the increasing computational costs of the other explainers. On the contrary, we tested tcav only on imagenet as tcav needs to obtain different images representing different concepts, and this is difficult to do for very simple images like the one in mnist and cifar. From the results, we notice that grad-cam and grad-cam++ are the fastest methods, especially for complex models like the VGG network. In general, SM pixel-wise explanations are faster to achieve because segmentation slows down a lot, especially for high-resolution images. CA, CF, and PR methods are very slow compared to SM. This problem happens because these methods require additional training or use some search algorithm to return their explanations. CA, CF, and PR methods produce more useful explanations, but since SMs are easier and faster to obtain, they are seen more applied in the literature than the other methods making them more widespread.

As seen in Sect. 6.1, saliency-based explanations are prevalent because they present visually perceptive explanations. Saliency Highlighting (SH) can be seen as a version of SMs applied to text. Indeed, it practically consists of assigning to every word (or set of words) a score based on the importance that that word (or set of words) had in the final prediction. Formally, Sentence Highlighting (SH) is modeled as a vector s that explains a classification \(y = b(x)\) of a black-box b on x. The dimensions of s are the words present in the sentence x we want to explain, and the value \(s_{i}\) is the saliency value of the word i. The greater the value of \(s_{i}\), the greater the importance of that word. A positive value indicates a positive contribution towards y, while a negative one means that the word has contributed negatively. Examples of SH explanations are reported in Fig. 14.

To obtain such an explanation, it is possible to adapt some of the SMs methods presented in Sect. 6.1. LIME (Ribeiro et al. 2016), presented in Sect. 5, can be applied to text with a modification to the perturbation of the original input. Given an input sentence x, lime creates a neighborhood of sentences by replacing one or multiple words with empty spaces. Another possible variation is to insert a stop word instead of removing it to maintain the meaning of the sentence. INTGRAD (Sundararajan et al. 2017), presented in Sect. 6, can also be exploited to explain text classifiers. Indeed, gradient-based methods are challenging to apply to NLP models because the vector representing every word is usually averaged into a single sentence vector. Since a mean operation gradient does not exist, the explainer cannot redistribute the signal back to the original vectors. On the other hand, intgrad is immune to this problem because the saliency values are computed as a difference with a baseline value. intgrad computes the saliency value of a single word as a difference from the sentence without it. For a fair comparison, we substituted the words with spaces as done for lime and also for intgrad. DEEPLIFT (Shrikumar et al. 2017), presented in Sect. 6, can also be applied to text following the same principle of intgrad. For the experiments, we adopt the same preprocessing used for lime and intgrad. L2X (Chen et al. 2018) can produce a SH explanation for text. In particular, for text, the patches adopted are a group of words. Finally, a baseline for text explainers is formed by gradient \(\times \) input that uses the black-box gradient of the input w.r.t to the output and multiply these value by the input values.

Sentence highlighting explainers comparison The SM-based explainers for images can be applied to text with minor modifications, which is why it turns out to be the most popular type of explanation also for textual data. As said for explanations on images in Sect. 6, these low feature explanations are helpful to check the model’s robustness, not to give a valuable understanding for the final inexpert user. In addition, removing a single word from a sentence is typically not a good measure of the goodness of an explanation since the sentence might lose its meaning. Among the methods tried, intgrad is the fastest and most reliable due to its use of the gradients of the black-box. deeplift also uses the gradients, but its calculation method fails and returns an explanation very similar to the baseline, as reported in Fig. 14. lime is the only model-agnostic method; however, NLP models tend to be very deep and are generally gradient-based. Thus, lime results in being very slow, and its performance is clearly inferior to gradient-based methods.

Methods that use gradients such as intgrad perform better on the textual data since very deep ML models are usually implemented in the NLP field.

Attention was first proposed for images in (Xu et al. 2015) to improve the model performance. The authors managed to show through an attention layer which parts of the image most contributed to realize the caption. Formally, the attention layer is indeed a layer to put on top of the model that, for each word, ij of the sentence x generates a positive weight \(\alpha _{ij}\), i.e., the attention weight. This value can be interpreted as the probability that a word ij is in the right place to focus on producing the next word in the caption. Attention mechanisms allow models to look over all the information the original sentence holds and learn the context (Wu and Ong 2021; Bahdanau et al. 2015). Therefore, it has caught the interest of XAI researchers, who started using these weights as an explanation. The explanation e of the instance x is composed of the attention values (\(\alpha \)), one for each feature \(x_i\). Attention is nowadays a delicate argument, and while it is clear that it augments the performance of models, it is less clear if it helps gain interpretability and the relationship with model outputs (Jain and Wallace 2019). Attention Based Heatmap (Li et al. 2016) is a local, intrinsic, model-specific explainer based on the attention mechanism. It produces a heatmap explanation similar to the one used for SMs by using the weights of the black-box. It can only be applied to attention-based methods, such as BERT, in which the weights \(\alpha _{ij}\) of the attention layers are used as a score for every word in the sentence. The higher the score, the more important the word; therefore, the redder the heatmap. Attention Matrix (Cheng et al. 2016) looks at the dependencies between words for producing explanations. It is a self-attention method, sometimes called intra-attention. attention matrix relates different positions of a single sequence to compute its internal representation. The attention of a sentence x composed of N words can be understood as an \(N \times N\) matrix, where each row and columns represent a word in the input sentence. The values of the matrix are the attention values of every possible combination of the tokens. This matrix is a representation of values pointing from each word to every other word (Vaswani et al. 2017) (see Fig. 15). We can also visualize this matrix with a focus on the connection between words (Hoover et al. 2019) as in Fig. 16, where the thickness of the lines is the self-attention value between two tokens.

Attention based explainers comparison Attention is a mechanism good for improving the performance of the model but is not usable as an explanation. As described in (Jain and Wallace 2019), it is unclear what relationship exists between attention weights and model outputs. Learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and it is possible to identify very different attention distributions that nonetheless yield equivalent predictions. Therefore we might suggest focusing on other types of explanations for text data.

It is unclear if attention can be considered a valid explanation. We suggest focusing on other types of explanations.

Counterfactual and prototype explanations are not very common explanations for text data. The richness of meaning in the textual data makes it complex to generate explanations since changing even a single word can deeply alter the meaning of the sentence. In this section, we report the main existing methods that attempt counterfactual or prototype explanations for text data.

PROTOTEX (Das et al. 2022) is a local, intrinsic, model specific NLP classification architecture based on prototype network (Li et al. 2018). prototex faithfully explains model decisions based on prototype tensors. At inference time, classification decisions are based on the distances between the input text and the prototype tensors, explained via the training examples most similar to the most influential prototypes. SEDC (Martens and Provost 2014) is a local, post-hoc, model-agnostic explainer able to provide counterfactual instances for documents. The approach proposed is based on a best-first search with pruning. The idea is that, given the words in the document, the algorithm tries to predict the label by removing a single word at a time: if the prediction changes, then it is added as an explanation. Then, also the combinations of words are considered. CAT, Contrastive Attributed explanations for Text (Chemmengath et al. 2022) is a local, post-hoc, model-agnostic explainer for NLP which returns contrastive explanations through a data twist that guarantees semantically meaningful explanations. In particular, cat rely on a minimal perturbation approach regularized using a BERT language model (Hoover et al. 2019) and an attribute classifier trained on available attributes on the text. POLYJUICE (Wu et al. 2021a) is a local, post-hoc, model-agnostic counterfactual generator that returns a set of realistic textual counterfactuals that can be employed for explanation purposes. In particular, it accounts for returning counterfactuals that are grammatically correct besides being minimal and realistic. The generation makes use of a fill-in-the-blank structure to specify where the perturbation occurs and control codes like negation, delete, insert, shuffle, etc., to specify how it occurs. GYC, Generate Your Counterfactuals (Madaan et al. 2021), is a local, post-hoc, model-agnostic framework to generate diverse counterfactual texts for testing automated decision-making systems. gyc returns plausible, diverse, goal-oriented, and effective explanations through an approach based on constraints that can be specified to guide the generation w.r.t. custom defined class labels, name-entities, topics, etc. Like polyjuice, also this approach is not specifically designed for XAI, but it can be easily used for interpretability purposes. XSPELLS (Lampridis et al. 2020) is a local, post-hoc, model-agnostic explainer returning exemplars and counterexamples sentences. In practice, it re-implements abele for text data by using LSTM layers in the autoencoder. Exemplars and counter-exemplars are selected using rules extracted from the decision tree learned in the latent space.

Prototype and counterfactual explainers comparison Counterfactuals and prototypes explanations for text are very difficult to generate. cat and sedc alter single words to change the prediction of the model; the former is more advanced, substituting words with an NLP model to make correct sentences. However, as seen previously, changing a single word hardly alters the meaning of the sentence and, therefore, the black-box prediction. gyc and polyjuice are interactive tools in which human intervention is needed, so they can be used for explanation purposes. xspell is the most automatic approach, which seems to be also the best one among the others, but it suffers from the same limitations of abele. The only prototype explanation method we identified for text is prototex.

Prototype and Counterfactual explanations are very difficult to generate for text data due to the meaning of the sentences. Interactive methods such as polyjuice and gyc are promising approaches that introduce the human cognitive process that helps in the process of counterfactual generation.

In this section, we report on other types of explainers that may not fit into one of the previous sections but are nonetheless relevant to the field. ANCHOR (Ribeiro et al. 2018), presented in Sect. 5, can be adapted to text by using as perturbation the word UNK. It consists of perturbing a sentence by substituting words with UNK (unknown). For example, It shows how “sucks” contributed to the negative prediction of the sentence, but when coupled with “love”, the sentence prediction switches to positive. Natural Language Explanation verbalizes explanations in natural human language. Natural language can be generated with complex deep learning models, e.g., by training a model with natural language explanations and coupling it with a generative model (Rajani et al. 2019). Besides, it can also be generated using a simple template-based approach (Abujabal et al. 2017). NLP is a very complex field, and finding a human-friendly explanation is challenging. Researchers are working in various directions of creating explanations with high concept (Srivastava et al. 2017) and using humans to augment these types of concept (Rajani et al. 2019), as done for Concept Attribution. Another possible development is the use of generative text methods to build a friendly narrative for the final user, which is not always an expert in the field.

Explanations for classifiers acting on text data are at the very early stages compared to tabular data and images.

Similarly to what we did for images, to quantitatively investigate the performance of the SH explainers analyzed, we computed the deletion and the insertion metric, presented in Sect. 4. For the computation of the deletion metric, we removed words in order of importance. In contrast, for the insertion metric, we started with an empty text, and we added words in order of importance. For every substitution we made, we queried the text to the black-box, obtaining accuracy. The final score is obtained by taking the area under the curve (AUC) (Hand and Till 2001) of accuracy as a function of the percentage of removed words. The results are shown in Fig. 14. We notice that the highlighted words are very different among the various methods. intgrad and lime are the ones who return more meaningful explanations, while deeplift struggles a lot to diversify from the baseline. We also measured the deletion/insertion and reported the results in Table 9. For both metrics, we have very poor performance among all the methods. However, this metric is not ideal with text since removing a single word barely changes the meaning of the sentence. On the other hand, concerning runtime, NLP models are usually very deep and large, resulting in poor performance in terms of runtime. In particular lime is the slowest since it queries the black-box for prediction multiple times. deeplift and intgrad are faster since they operate on the gradients of the black-box. Finally, AB methods are instant since they only need to extract the weights and do not need to perform any operation.

Due to the tremendous amount of data generated by sensors over time, there is a widespread diffusion of ML models working on time series (Theissler 2017). A time series \(x = \{t_1, t_2, \dots , t_m\} \in \mathbb {R}^{m\times d}\) is an ordered set of m real-valued observations (or time steps), with dimensionality d. We say that a time series is univariate when \(d=1\), while when \(d>1\), we name x a multivariate time series. There are areas such as the medical or financial field where temporal data is of particular importance and where black-box ML models are applied to provide support on decision-making for various tasks. For this reason, recently, we have been assisting with the emerging proposal for explainability related to time series  (Theissler et al. 2022).

Many explainers for time series can be categorized with the taxonomy introduced in Sect. 3. The most important difference with the other types of data relies on the type of explanation produced. Shapelet is the most characteristic explanation for time series data. They are time series subsequences that are maximally representative of a class. Shapelets are more interpretable, faster, and more accurate than k-Nearest Neighbors (kNN) (Cover and Hart 1967), which is a traditional approach to perform time series classification (Lee et al. 2012). As usual, SMs can be used to highlight which part of the series has contributed the most to the classification. Finally, also the attention methods illustrated for text in Sect. 7.2 can also be applied to time series data. In the following, we briefly illustrate some peculiar explainers for time series classification.

In (Geler et al. 2020), introduced a transparent by design method named Weighted-kNN that extends the classic majority-voting kNN by proposing weighting schemes. By emphasizing the nearer neighbors using a weighting scheme, it is possible to improve the kNN classifier’s quality and stability. The nearest neighbors are considered part of the prototypical explanation. LASTS, Local Agnostic Shapelet-based Time Series explainer (lasts) (Guidotti et al. 2020b), is a variation of abele for time series. As explanation lasts returns exemplars and counterexamples composed of subseries with a shapelet-based rule. An example of a rule is: “if these shapelets are present and these others not, then x is classified as y”. DOCTORXAI (Panigutti et al. 2020) is a local post-hoc model-specific explainer acting on sequential data in the medical setting. In particular, it exploits a medical ontology to perturb the data and generate neighbors. doctorxai is designed on healthcare data, but it can theoretically be applied to every type of sequential data with an ontology. For more information on explaining black-box models for time series, we remand the reader to the following survey (Rojat et al. 2021; Theissler et al. 2022).

Graph Neural Networks (GNNs) have become increasingly popular since many real-world data are represented as graphs, such as social networks, chemical molecules, and financial data. A graph can be used for specific tasks like link prediction (Cai and Ji 2020) or node labeling (Calamoneri 2006). However, many advanced GNN operations are also proposed to improve the performance of classification models, for example, in graph convolution models (Kipf and Welling 2017). Compared with image and text domains, the explainability of black-box models working on graphs is less explored, but it is critical for understanding to behavior of GNN.

The types of explanations available for graphs are various. The most common one is node relevance which derives from feature relevance methods. It consists of assigning a value to every node of the graph that represents how much that node has contributed to the prediction. Since graphs are composed of nodes and edges, the same operation can be accomplished with edges. The focus of the explanation is shifted toward the interactions between the nodes instead of the nodes themselves. There are different ways to do this. The most common one is to backpropagate the signal from the prediction to the input graph in lrp style. Other approaches learn a surrogate model to assign values as done by lime. LRP (for Graphs) (Schwarzenberg et al. 2019) extends the original LRP method for GNN. It decomposes the output prediction score into different node importance scores. The score decomposition rule is developed based on the hidden features and weights. GraphLIME (Huang et al. 2020) extends the lime for GNN and studies the importance of different node features for node classification tasks. Given a target node in the input graph, graphlime considers its N-hop neighboring nodes and their predictions as its local neighborhood, where a reasonable choice of N is the number of layers in the GNNs. Then a nonlinear surrogate model is employed to fit the local neighborhood. Finally, based on the weights of different features in the surrogate model, it selects important features to explain the predictions. XGNN (Yuan et al. 2020a) (eXplainable GNN) is a global, post-hoc, model-specific explainer that proposes graph patterns as explanations to investigate which structure in the graph maximizes a specific prediction. Specifically, they trained a graph generator to generate small graphs, which pattern can be used to explain GNN behavior. The graph generator uses reinforcement learning that starts from an empty graph and, at each step, determines how to add an edge or node and form a new graph. This graph generator is trained based on feedback from trained graph models using a policy gradient algorithm. CF-GNNExplainer (Lucic et al. 2022) is a local, post-hoc, model-specific explainer for GNNs. It aims at finding the minimal perturbation to the graph such that the prediction changes. In particular, cf-gnnexplainer iteratively removes edges from the original adjacency matrix based on a matrix sparsification technique, keeping track of the perturbations that lead to a change in prediction and returning the perturbation with the smallest change w.r.t. the number of edges. For more information on explaining black-box models for graphs, we remand the reader to the following survey (Prado-Romero et al. 2022; Yuan et al. 2020c).

GNN explainers comparison GNN models are becoming more and more common in literature (Wu et al. 2021b), and due to their similarity to neural networks, researchers are trying to adapt existing methods to this type of black-box. graphlime and graphlrp) are two perfect examples of that, the former adapting lime while the latter \(\epsilon \) -lrp. Both methods return scores for nodes, but graphlime does not output a score for the edges features as it totally ignores the graph structure, which is more critical for graph data. xgnn and cf-gnnexplainer are more aimed at counterfactual explanations. xgnn returns more robust counterfactuals due to the reinforcement learning policy; however, it is much slower than cf-gnnexplainer for the same reason.