Explainable AI

During the last years, explainable AI (XAI) has been established as a new area of research focussing on approaches which allow humans to comprehend and possibly control machine learned (ML) models and other AI-systems whose complexity makes the process which leads to a specific decision intransparent. In the beginning, most approaches were concerned with post-hoc explanations for classification decisions of deep learning architectures, especially for image classification. Furthermore, a growing number of empirical studies addressed effects of explanations on trust in and acceptability of AI/ML systems. Recent work has broadened the perspective of XAI, covering topics such as verbal explanations, explanations by prototypes and contrastive explanations, combining explanations and interactive machine learning, multi-step explanations, explanations in the context of machine teaching, relations between interpretable approaches of machine learning and post-hoc explanations, neuro-symbolic approaches and other hybrid approaches combining reasoning and learning for XAI. Addressing criticism regarding missing adaptivity more interactive accounts have been developed to take individual differences into account. Also, the question of evaluation beyond mere batch testing has come into focus.

In this special issue, we address such recent developments in XAI and also take a closer look at interdisciplinary perspectives. We thereby address methods and applications ranging from model change, robotics and image processing over event sequences from a smart home to fact checking. An interdisciplinary insight is given by an analysis of real life explanations of a medical expert in a health care situation.

In our survey “What is missing in XAI so far? An interdisciplinary overview” we bring together interdisciplinary perspectives on explaining and understanding in everyday situations and explaining of AI. We identify challenges and point to open questions regarding faithfulness and consistency of explanations, adaptability to specific information needs and explanatory dialog for informed decision making, the possibility to correct models and explanations by interaction, as well as the need for an integrated interdisciplinary perspective and rigorous approaches to empirical evaluation based on psychological theories.

Johannes Rabold presents in his contribution on a “Neural-symbolic Approach for Explanation Generation based on Sub-concept Detection: An Application of Metric Learning for Low-time-budget Labeling” an approach that enriches visual explanations with verbal local explanations of relational information through combining metric learning and inductive logic programming. It allows a human to provide labels for a small subset of important image parts which are then used to learn a first-order theory that locally explains the black-box with respect to the given image.

Kerzel and colleagues target human–robot interaction scenarios in their contribution on “What’s on Your Mind, NICO? XHRI: An eXplainable HRI Framework for Humanoid Robot Social Interaction and Collaboration”. Settings that require users to interact with a robot confront users with complex, embodied AI systems that react in unanticipated ways. Explanations about the robot’s perception and internal reasoning are, thus, an important prerequisite for smooth and controllable interaction. The authors, therefore, propose an eXplainable HRI (XHRI) approach for efficient communication. With NICO, the Neuro-Inspired COmpanion, an XHRI framework is presented that provides different types of explanations using verbal and non-verbal modalities based on novel approaches to extract such explanations from neural network modules.

In a more abstract scenario Finzel and colleagues in their contribution on “Generating Explanations for Conceptual Validation of Graph Neural Networks: An Investigation of Symbolic Predicates Learned on Relevance-Ranked Sub-Graphs” propose to combine Inductive Logic Programming with Graph Neural Networks (GNN) to generate explanations regarding spatial relations within Kandinsky Patterns with ILP based on the relevance output of GNN explainers and human-expected relevance for concepts learned by GNNs.

Muschalik et al. in their contribution “Agnostic Explanation of Model Change based on Feature Importance” turn our attention to the difficult problem of XAI in the context of online learning in dynamic environments, where models need to be adapted continuously due to the changing data. The authors motivate the problem of explaining model change, i.e. the difference between models before and after adaptation, instead of explaining the models themselves. To this end a first efficient model-agnostic approach to dynamically detecting, quantifying, and explaining significant model changes based on an adaptation of the Permutation Feature Importance measure is provided that allows the explainee to adapt several hyperparameters affecting the explanation.

Hartmann et al. present the project XAINES in their report “XAINES: Explaining AI with narratives”. The XAINES project investigates the explanation of AI systems through narratives targeted to the needs of a specific audience, focusing on two important aspects that are crucial for enabling successful explanation: generating and selecting appropriate explanation content, i.e. the information to be contained in the explanation, and delivering this information to the user in an appropriate way. In this article, the project’s roadmap towards enabling the explanation of AI with narratives is presented.

With anomalie detection in a smart home the next project report from Baudisch et al. addresses an everyday AI application. Their Project Report “A Framework for Learning Event Sequences and Explaining Detected Anomalies in a Smart Home Environment” presents an approach to explaining detected anomalies in event sequences in a smart home environment based on simple rule analysis. Anomalies are detected with a model learned from the behavior of the residents by finding deviations from the “normal” behavior.

The following report on “Exploring monological and dialogical phases in naturally occurring explanations” by Fisher and colleagues provides an interdisciplinary perspective on naturally occurring explanations. The project investigates the structure of naturally explanations in a medical context and the roles and engagement of the participants with respect to their conversational jobs. By identifying monological and dialogical phases it reveals that despite a high variability in the sequential structure of different explanations there is a pattern indicating that explainees take an active part by initiating dialogical phases. This has consequences for XAI systems as they need to be able to provide similar phases and activities also in interactions with AI systems.

The following article “Towards Explainable Fact Checking” is a summary of a habilitation (doctor scientiarum) thesis submitted by Isabelle Augenstein to the University of Copenhagen. The dissertation addresses several fundamental research gaps within automatic fact checking. The contributions are organised along three verticles: (1) the fact-checking subtask they address; (2) methods which only require small amounts of manually labelled data; (3) methods for explainable fact checking, addressing the problem of opaqueness in the decision-making of black-box fact checking models.

The article “Identification of explainable structures in data with a human-in-the-loop” presents the thesis of Michael C. Thrun which describes the central steps of an approach for a parameter-free methodology for the estimation and visualization of probability density functions which serve as a basis for selecting a hypothesis and an appropriate distance metric independent of the data context. Projection-based Clustering allows for subsequent interactive identification of separable structures in the data. The complete XAI approach is based on a decision tree guided by distance-based structures in data. The DSD-XAI shows initial success in application to multivariate time series and non-sequential high-dimensional data and generates meaningful and relevant explanations that are evaluated by Grice’s maxims.

Finally, in the “Interview with the Speakers of the TRR 318 Constructing Explainability Katharina Rohlfing and Philipp Cimiano” Ute Schmid elicits insights into the newly established TransRegio/Collaborative Research Center. In the interview, the two speakers explain the core ideas of their approach to explaining AI and its social meaning in different contexts.