What is Missing in XAI So Far?

With the growing number of real-world applications of AI-technology, the question about how to control such systems has become a growing concern [73]. Especially the rise of models with large degrees of freedom and relatively little pre-defined structure in the last decade has increased need for understanding learned models through explanation. The requirement that humans need to be able to understand on what information an AI-system relies to derive a specific output, has been identified as crucial early in the discussion and had been labeled explainable AI (XAI) [2].

Originally, the focus has been mostly on visual explanations for image classification [67, 74]. At first, such explanations by visual highlighting have been – rather naively – seen as a suitable means to make decisions of a black box system transparent to end-users and thereby inspire trust. However, this view has been pointed out as being simplistic from early on from an interdisciplinary perspective [56]. Consequently, XAI methods have been developed with different stakeholders in mind. Visual highlighting based on relevance of information in the input has been shown to be especially helpful from a developer’s perspective [61] with the goal to identify overfitting of models [52] (see Fig. 1). As a second stakeholder group, end users are considered. Here, the European Union’s General Data Protection Regulation (GDPR) boosted research initiatives that target XAI as a means to support the right for explanations – especially in Europe [89].

Furthermore, it has been noted that especially critical application areas of AI such as medicine, industrial production, or justice also require transparency and tractability which can serve as justification and to provide trustworthy support by enabling domain experts to comprehend how the AI system has come to a specific proposition and which part of the input data has contributed most to a decision [40, 60]. For domain experts, explanations have mostly the function to support decision making. In this context, there are high demands on the faithfulness of explanations and on their expressiveness [77]. Besides system developers, end-users, and domain experts, recently additional stakeholder groups have come into focus such as affected parties, deployers, and regulators [50].

While the term “explainable AI” has found its way into the wider public discourse, the problem itself has generated a range of different terms that have been used in addition to explainability [40] – namely comprehensibility [59] and interpretability [23, 58]. These latter two terms have been mainly used with respect to the transparency of machine learned models. Nowadays, the term interpretable machine learning mostly refers to symbolic, white-box machine learning approaches which can be used either as surrogate models [20, 21, 65] or as alternative for black-box models derived from training deep neural networks or other statistics approaches [49, 72]. The choice of the term “explainability” gave rise to some misunderstandings outside the XAI research community, being for example interpreted such that some expert explains what is meant by artificial intelligence or how a specific system works. Consequently, sometimes now “explanatory” is used instead [3, 47, 83]. Furthermore, the characteristics of an AI-system to provide an explanation for a decision has been proposed a crucial feature of so-called third wave of AI systems¹, superseding the second wave of AI focussing on data-intensive blackbox machine learning approaches, which have followed to first wave of AI knowledge-based symbolic approaches [41, 79].

The topic of explanations has been addressed in AI research already long before the current interest. In the context of expert-system research, approaches to generate verbal explanations from reasoning traces have been proposed [18]. In the German Journal of Artificial Intelligence (KI – Künstliche Intelligenz) the topic of explanations has been addressed in the context of expert systems [45] and has been dedicated a special issue in 2008² presenting research on explanations in different areas of AI such as case-based reasoning and theorem proving [70]. More recently, explanations were researched extensively in the context of recommender systems [86] and in interactive systems [14, 31, 47]. In addition to visual explanations and other methods to highlight feature relevance, verbal explanations as proposed in symbolic AI research receive new interest, especially for explanations intended for domain experts in complex relational domains [60, 80, 91, 92] (see Fig. 1). Furthermore, explanations by example are investigated [37, 43].

Over the last years, on the one hand, a large variety of computational approaches to XAI have been developed, and, on the other hand, the topic of explanations has received more attention in different disciplines such as human-computer interaction, educational and cognitive psychology, cognitive science as well as sociology and ethics. Furthermore, the topic has been treated in numerous surveys [2, 23, 34, 87]. Nevertheless, it is our impression that rigorous integration of XAI methods and insights from different disciplines is still missing. In the following, we will first introduce building blocks of an interdisciplinary perspective on XAI. Afterwards, we highlight selected aspects which, in our opinion, should receive more attention in future XAI research: faithfulness and consistency of explanations, adaptability to specific information needs and explanatory dialog for informed decision making, the possibility to correct models and explanations by interaction, as well as the need for an integrated interdisciplinary perspective and rigorous approaches to empirical evaluation based on psychological theories.

The AI community has developed a plethora of approaches for making AI decisions or models better understandable mostly to developers and AI experts. Only very recently the question what kind of explanations lay users actually need in their everyday interactions with AI and how their cognitive system affects their processing and ultimately their understanding of explanations has come into focus and points towards more human-centered approaches of explaining. We therefore argue to strengthen research in those XAI areas that are supported by interdisciplinary research with the aim to take the human explainee – the addressee in an explanation setting – and the situational context of the application into account.

In this paper we especially focus on the following perspectives:

Philosophy of science. Explanations have been studied for a long time in the field of philosophy of science addressing the question what are good scientific explanations. According to this explanations should follow logical rules in a deductive way [36] which is very similar to the road XAI has followed in its beginnings and is still pursuing. However, more recently it has been noted that lay persons do not follow such strict rules in their everyday explanations [42] and the understanding thereof. Keil argues that, for example, people are often not aware of why they prefer one explanation over another, thus apparently using implicit and not necessarily logical strategies to evaluate explanations. Also, they tend to construe causal explanations based on very sparse data, often simply on single-trial learning. For humans, the quality of an explanation is not tied to its logical correctness and completeness. Rather, they judge explanations along the dimensions circularity, relevance and coherence. Yet, it has been shown that circularity can be difficult for humans to detect [68]. Although understanding explanations is important for humans in order to carry out (complex) tasks and to achieve their goals, humans are often not aware that they have not understood an explanation, a phenomenon which has been termed “the illusion of explanatory depth” [71]. Thus, everyday explanations as they are used by most people in their daily lives seem to follow very different mechanisms than scientific ones.

Cognitive Psychology. In general, humans seldom make sense of their environment by explicit computation and evaluation of different hypotheses. Rather, they rely on heuristics that have proven helpful and mostly true in many situations and have often found their way into the human cortex. These heuristics can actually be interpreted as biases that shape how humans make sense of their complex environment and interactions such as explanations. These biases are well known in the literature. For example, [91] refer to cognitive biases in medical decision support systems. One frequent bias that unexperienced physicians experience is the positivity bias, i.e. once the physician has formed a hypothesis s/he searches for confirmation of this hypothesis instead of searching for falsification which would provide a much higher information gain. The anchoring bias is the tendency to rely too heavily on the first piece of information when making decisions. The availability bias is the tendency to overestimate the likelihood of events with greater “availability” in memory. This greater availability in memory can be achieved through more recent memories, e.g. a very rare disease which has been recently encountered, or through unusual or emotionally loaded events. Another bias in this context is a cognitive dissonance reduction bias, such as effort justification which is the tendency to attribute greater value to an outcome if one had to put effort into achieving it. In everyday situations we continuously encounter biases through prejudices and stereotypes which we also apply to explaining situations involving AI systems. Especially in the context of decision support systems Berkson’s paradox can be difficult to overcome. It is the tendency to misinterpret statistics involving conditional probabilities which are fundamentally important in order to understand decision support suggestions from an AI system in a given context.

For XAI applications this means that the explaining system should actively search for cues that indicate that the human explainee is following heuristics and cognitive biases rather than logically correct reasoning and provide active support against these [90].

Educational Psychology. In the context of teaching explaining plays an important role. However, in this context it has been observed that the behavior or engagement of the learner seems to be at least equally important. It should be noted, that this kind of research focuses on scientific explanations rather than everyday explanations. However, we believe that its focus on engagement is a valuable perspective also for XAI.

[16] have proposed a model that targets the involvement of the explainee in order to enable her to ask questions and actively generate new insights through reasoning and discussing. In detail, the ICAP model (for Interactive, Creative, Active, and Passive engagement behavior) [17] targets teaching situations and postulates that different learner engagements yield different learning outcomes. More specifically, according to this account the least learning success is achieved by passive engagement alone, i.e. attentive listening or reading. Active engagement which requires manipulative behaviors such as repeating and rehearsing as achieved e.g. by watching explanations videos yields better learning outcomes. Constructive engagement which is achieved by generative behaviors such as “reflecting out loud and self-explaining” which require to actively draw inferences based on one’s mental model achieves better results. Best results are postulated by co-generative collaborative behaviors which are characterised by dialogical interactions involving “defending and arguing a position, and discussing similarities and differences” which challenge the learner to co-infer together with a peer or the teacher new propositions. Indeed, many studies support the basic assumptions of the ICAP model in lab and teaching contexts (e.g. [19, 30]). Insights from education, thus, strongly suggest that explanations should be embedded in a dialogical interaction allowing the explainee to be actively engaged and challenge those aspects of an explanation that have not been understood.

Developmental Linguistics. Explanations and teaching play an important role in parent-child interactions and important concepts have been developed in this context.

Scaffolding is a frequent and highly useful strategy that parents use for teaching their children [11, 93]. The specific benefit of scaffolding has been identified as enabling the child - or explainee - to solve a problem which would be beyond his/her current capabilities. It means that the teacher is carrying out those elements that the learner can not yet do. In a joint task setting with children this often pertains to motor actions, but it can also mean to explicate a reasoning step in a more abstract explanation situation or to ask a (suggestive) question, enabling the explainee to complete those elements of the problem that are within his/her range of competence and to achieve a successful completion of the task. A feature that so far has received less attention pertains to the emotional quality of scaffolding. In general, scaffolding interactions have a highly positive emotional component that serves as a strong motivator to the learner. However, how these emotional processes work in detail together with the understanding process remains largely unknown.

Taking features of scaffolding into account, as provided by [93], scaffolding requires from the explainer to provide contingent feedback on the learner’s actions, to give hints or tips, to provide instructions and explanations, and to ask questions. Importantly, the teacher should introduce and divide the task into manageable pieces to allow for continuous progress in dealing with the challenge. This entails to provide meaningful associations between objects, concepts and actions and to make use of non-verbal means.

Scaffolding, thus, requires the ability to monitor the learner’s attention and understanding process in order to identify knowledge gaps that need do be addressed by the teacher.

New perspectives on explaining in XAI. Humans’ explanations are characterised by an interactive account which follows specific sequences [44, 63]. Such interactive explanations enable adaptation to the explainee’s level of understanding and should be the basis of any explanation approach [69].

More specifically [69] challenge a range of often implicitly made assumptions that dominate much of current XAI research. According to this, XAI targets scientific explanations for which accuracy and completeness are mandatory. Whereas in their view “everyday explanations can be understood as partial descriptions of causes that enhance understanding in the user in terms of why something happened”. This places the user’s understanding in the center of the explaining process rather than the explanandum, i.e. the elements that need to be explained. Furthermore, [69] argue that personalisation which is a standard approach in Human-Computer Interaction and also being explored in XAI is not an adequate strategy to adapt to individual differences as the understanding process is dynamic, that is the level of understanding is continuously changing and adaptation, thus, has to take place at each step. This also entails to model the explainee’s prior knowledge which can also vary widely. To address this, a conceptual framework is proposed that allows to “co-construct understanding” through a “bidirectional interaction involving both partners in constructing the task and its relevant (recipient-oriented) aspects”. This idea of co-construction seems to carry interesting similarities to the above mentioned concept of “co-generation” from the ICAP model which has been shown to yield better learning results. This co-construction requires that the explainer closely monitors the explainee for signs of (non-)understanding and to provide scaffolding where understanding becomes difficult (cf. Fig. 2). This also requires the explainer to continuously update her partner model \(M_{ER}\). Through this contingent process of monitoring and scaffolding the explanans is constructed, i.e. the co-constructed explanation which only encompasses a part of the explanandum and that (optimally) addresses those aspects that are relevant to the explainee.

Mental Models, Partner Models and Theory of Mind. Thus, an important prerequisite for construing as well as understanding explanations are mental models and cognitive processes that allow to represent, retrieve and apply relevant aspects of a phenomenon [42] as well as a model of the partner and the partner’s model.

Human explainers utilize a mental model about the explainee’s understanding, i.e. they form hypotheses about what the explainee has understood and what aspects need to be corrected or explained again, or should be attended to more strongly. For example, [16] investigated how mental models are formed by students and how they are addressed by the teacher when explaining learning content. According to her research, explainers form general models about their tutee’s understanding that represent canonical misunderstandings. However, they found surprisingly little evidence that explainers also were able to build models of individualized misconceptions that go beyond canonical misunderstandings [16]. Nevertheless, having a model of the explainee’s understanding seems to be important for successful explanation processes. Indeed, it could be shown that an Intelligent Tutoring System with contingent and fine-grained tutoring sub-steps achieved a similar learning effect as a human tutor [88].

In HCI and HRI partner models are used that represent relevant aspects of the ongoing interaction and the partner’s knowledge in terms of perception, understanding, acceptance and agreement [13]. In order to incorporate the interaction partner’s perspective into the own planning and reasoning Theory-of-Mind models [51] are being developed and investigated in various contexts. Such a mentalistic approach might provide a good basis for explaining in XAI. However, so far research on partner models or TOM models in the context of XAI seems to be very limited.

While many authors from the social sciences demand for user-driven approaches that are grounded in real world situations, there actually exists almost no empirical data on how AI is used in real life and what explanatory demands actually arise from these situations.

Based on the observations above, in the following, we will highlight some aspects on XAI which are, in our opinion, crucial for successful real-world applications of AI in human-in-the loop settings.

Over the last years, research on XAI has broadened from focusing on developer oriented explanations of the AI model to consider other stakeholders such as end users, domain experts, affected parties, deployers or regulators. This has lead to the general insight that explanations need to be adaptive to individual user needs and take human cognitive processes into account that require certain strategies and may be subject to biases. This requires a strong interdisciplinary approach in order to equip mental and partner models with those features that are relevant for the contingent explaining process. Also, linguistic expertise is needed in order to develop dialog modeling approaches that can model contingent multi-modal interaction.

Accordingly, evaluations need to address multiple dimensions ranging from the correctness and performance of the classification and explanation model to the effect that the explanations generated by this system have on users in their specific contexts.

In our view, problematic issues of XAI that should receive more attention in the future arise from the pragmatics, that is, the specific use and context of an AI system in a concrete setting. This situatedness demands for a new perspective on evaluation and has led to a debate on the question under what conditions specific explanatory approaches are helpful and how this might be measured. For example, is an explanation helpful when the explainee is being supported (or “enabled”) to take the “optimal” decision according to the decision support system? This entails further questions regarding among others effects of non-rational factors such as emotions or social biases.

A second strand of discourse pertains to the question of faithfulness. There is currently only little research on how faithful explainers actually are, that is, whether the proposed features are indeed the most relevant features that have lead to a certain decision, and how this can be measured reliably. This also impacts fairness as unfaithfulness may lead to a situation where biases and unfair decisions are not being detected.

Finally, while most explanation approaches focus on explaining at the domain level, that is how much a certain feature has contributed to a certain decision, it requires more research to investigate whether explaining important elements of the underlying AI model is also important to enable explainees to take better decisions, that is how the model has derived the decision and what role the data may play in better understanding how a decision support system has come to a conclusion.