Explanatory Interactive Machine Learning

Increasingly, it is becoming apparent that Interactive Machine Learning (IML), i.e., the integration of user feedback into a Machine Learning (ML) process to modify an ML model (e.g., Amershi et al. 2015), may play a leading role in shaping Artificial Intelligence (AI) and in particular ML-based systems for effective use in organizations. To realize their full potential, AI systems must become capable of communicating and collaborating with, learning from, and teaching their users. Finding the right ways to induce learning in human-machine interaction is required to unlock many scientific and commercial opportunities in AI (Teso and Hinz 2020).

Challenges to thorough understanding, potential learning from AI-based Systems, and effective organizational use result from the lack of system transparency (e.g., Rai 2020), which is often accompanied by the uncertainty of whether a system is biased. In the past five years, various scholars have shown the potential adverse effects of algorithmic biases on human decision-making (e.g., Lambrecht and Tucker 2019). Outcries demanding transparency and accountability have become louder and, as such, have found their way into legislation (e.g., Casey et al. 2019). Hence, there is an urgent need in many industries to fulfill such regulatory requirements for AI-based Systems (e.g., Sorantin et al. 2022; Casey et al. 2019). By now, many organizations are aware of this problem yet face a challenge in finding appropriate ways to deal with biases and erroneous systems (e.g., Holstein et al. 2019).

Researchers and practitioners have proposed various software engineering approaches that offer best practices for data science projects (e.g., Amershi et al. 2019; Wang et al. 2019). Many of these methods propose not a linear development process but a process that includes recursions to previous stages when necessary. Especially during model evaluation, data scientists need to reflect on the intermediate results of their work by, e.g., inspecting metrics and predictions and possibly rearranging their analytical workflow and data (e.g., Amershi et al. 2019). In this regard, research on data science collaboration (e.g., Zhang et al. 2020) suggests that – rather than having only data scientists working on their own – the inclusion of various kinds of users in the development process is beneficial to reducing algorithmic biases and increasing domain correctness.

Human-in-the-loop concepts (e.g., Grønsund and Aanestad 2020) and especially IML (e.g., Amershi et al. 2015; Abdel-Karim et al. 2020) constitute one possibility to include non-technical users in a data science process. Although human-in-the-loop methods have proven successful in improving various systems (e.g., Amershi et al. 2015) by including different kinds of users (e.g., Kulesza et al. 2015), users often are lost without appropriate explanations and thus do not know how to interpret and alter a system’s result appropriately (e.g., Dudley and Kristensson 2018, p. 24).

Schramowski et al. (2020) address this problem by including domain experts in a generic explanatory interactive machine learning (XIL) (Teso and Kersting 2019) process. While data scientists are usually aware of apparently wrong system attributions and can fix them on their own, XIL and the input of the domain expert can help identify and even correct predictors which only appear to be highly performant but are based on incorrect inferences.

There is an ongoing discussion on how interpretability, explainability, and transparency are related to each other and whether explanations (see Gregor and Benbasat 1999) help to better understand AI-based systems (e.g., Rudin 2019; Linardatos et al. 2020). Although both explainable and interpretable AI aim to bring more transparency to the application of ML, usually the former aims to do so by employing additional models (e.g., Ribeiro et al. 2016) that explain the ML model in focus (e.g., Deep Learning models, LeCun et al. 2015), while interpretable AI aims to use more lightweight, inherently interpretable ML models (e.g., GANs, Caruana et al. 2015). Rudin (2019) presents arguments that explanations of black-box models cannot provide the necessary insights to fully understand how such models arrive at their predictions (but have other benefits). This separates inherently interpretable models from ‘explained’ black-box models, which, on the one hand, offer benefits by usually providing greater modeling capacities, while the use of interpretable models, on the other hand, often requires less expertise (Rudin 2019, p. 206, pp. 208–210). Nevertheless, although the two streams appear to be in conflict, both share the common goal of providing insight into ML models. Hence, we use a more universal term of explainability for this work, as explanations provide insights that increase transparency and user acceptance of a system and help with the transfer of knowledge (Gregor and Benbasat 1999). Explanations support an analytical interpretation of black-box model results (despite them not being inherently interpretable).

Because XIL is a powerful yet flexible approach to interpreting and rectifying even systems with strong black-box characteristics, it may be a promising basis to leverage the full potential of AI-based systems for organizations. However, to do so, XIL needs to be embedded in a more generalizable approach to ease its applicability for machine learning projects in organizations and research alike.

Senior scholars have already recognized the need to extend existing methodologies such as action research (e.g., Maass et al. 2018) and see the potential of such extensions and improvements as valuable “contributions to the IS community” (Baskerville et al. 2018, p. 365). Furthermore, scholars theorizing on future problems caused by biased ML systems demand that IS should tackle the challenge of mitigating biases in ML systems (e.g., Kane et al. 2021, p. 375). In their investigation of the pitfalls of the strong reliance on ground truth values in AI, Lebovitz et al. (2021) emphasize the need to pay respect to ‘know-how’ measures (i.e., expert knowledge and knowledge that explains how an AI makes its decisions). It is, therefore, not sufficient to only rely on seemingly objective ‘know-what’ measures (i.e., ground-truth, performance metrics such as accuracy) (Lebovitz et al. 2021, pp. 1516–1517). These works imply the need for dedicated methods that enable a participatory, interactive, and explanatory workflow, honing problematic areas with great attention, diligence, and human expertise. The current paucity of applicable Information Systems Development Methods (ISDM) with foci on participatory feedback of users and the elimination of potential bias within machine learning projects thus suggests that we need to engineer suitable methods for this case.

We engage in situational method engineering (e.g., Goldkuhl and Karlsson 2020) and propose an ISDM that embeds cutting-edge technology, namely Explainable Artificial Intelligence (XAI) and Interactive Machine Learning (IML), referred to as Explainable Interactive Machine Learning (XIL) in the established framework of Action Design Research (ADR) (e.g., Mullarkey and Hevner 2019). The resulting XIL-ADR methodology positions itself as an implementation-centered ADR methodology along the elaborated ADR process that puts humans and machines into a loop which aims to remedy potential biases in an AI-based system and to generate novel insights on the side of the human user.

The proposed cyclic XIL-ADR process is generic and transferrable to other domains. In particular, it provides a development-focused organizational frame that is not limited to certain data types, specific algorithms for the explanations, ML methods, or the IML part. XIL-ADR certainly touches a truly interdisciplinary research field with connections to Computer Science, Learning and Knowledge Discovery on the human side, and domain knowledge from the application area.

To highlight the benefits of XIL-ADR, this paper uses a healthcare setting and presents an illustrative study conducted with medical experts from a leading university hospital. We exemplify our proposed approach in this healthcare scenario and show how XIL-ADR can help build an innovative computer vision system to efficiently (in terms of model performance) predict viral pneumonia. In particular, we will outline how XIL-ADR can help arrive at more meaningful models and, at the same time, help identify interesting patterns in the data that radiologists and pneumologists should pay attention to. The illustrative case study also reveals problems that, using standard approaches without participatory human-machine feedback, would be recognized much later, i.e., when the developed AI-based system fails in practice. Thus, the approach has two advantages: (1) it helps to create more dependable models, as it enables users to recognize intricate data and architecture problems early in the project, and (2) it simultaneously allows experts to reflect on the existing knowledge base by opening the otherwise black-box process of many Deep Learning (DL) algorithms and by this means may arrive at new insights and learning.

This paper proceeds as follows: First, we will introduce and conceptualize an ideal XIL-ADR process. We will do so by first outlining relevant research on IML, referencing XIL, embedding XIL in ADR, and then presenting how to structure ML projects for XIL-ADR. Next, we will briefly introduce our illustrative case of an AI-based diagnosis system for viral pneumonia, through which we aim to highlight how to apply XIL-ADR. We first introduce the reader to the problem to solve, previous research in computer vision, Covid-19, and the technical methods. This introduction also informs the first stage (Diagnosis) in our XIL-ADR project. For stage two (Design), we will give a brief overview of the technological methods we used, i.e., DL, XAI, the loss function for the penalization of the model, and the method for XAI visualization. What follows is the evaluation of the Implementation stage using the XIL-ADR methodology: A detailed description of the case study conducted together with pneumologists, radiologists, computer scientists, and information systems researchers is presented, which describes every step and intermediate artifact of the cyclic XIL-ADR implementation process in three subsequent cycles. Lastly, we discuss XIL-ADR considering the presented case and outline directions for future research and practice.

In this section, we engage in method engineering (e.g., Goldkuhl and Karlsson 2020) and embed XIL into the ADR framework. For this, we follow the “Method Engineering as Design Science” (ME-DS) process by Goldkuhl and Karlsson (2020) and develop a “Category 4” Information Systems Development Method (a “Scholar generated ISDM targeted for business practice & scholars”) (Goldkuhl and Karlsson 2020, pp. 1245–1246). The first activity is identifying the problem and motivating the development of XIL-ADR, which has already been dealt with in the previous section. The next step, theorizing the ISDM and engineering the method, follows now by briefly introducing the reader to relevant work on combining XAI and IML as well as to XIL itself. We describe how we can integrate XIL into the elaborated ADR process (Mullarkey and Hevner 2019) to engineer a novel methodology called XIL-ADR and point out the differences from classical ADR. We complete this method engineering activity by describing the different activities of the XIL-ADR cycle and how machine-learning projects may benefit from conducting XIL-ADR. For the last two activities of the ME-DS process, which are demonstrating and evaluating the proposed ISDM, we use our healthcare case presented in the subsequent sections. Hence, this paper aims at the demonstration and ‘communication’ (in terms of Design Science) of XIL-ADR (Goldkuhl and Karlsson 2020, p. 1250; Hevner et al. 2004).

Guided by the “Method Engineering as Design Science” (Goldkuhl and Karlsson 2020) process, the following two sections serve as the demonstration and evaluation of the proposed ISDM method. To illustrate how we envision the application of XIL-ADR in research and practice (Goldkuhl and Karlsson 2020), we conducted an exemplary ADR case study. In the following, we provide an overview of our case study situated in healthcare. This case description also guides the reader along the first two of four stages in ADR as put forward by Mullarkey and Hevner (2019), i.e., the Diagnosis and Design stages (refer to the section “Embedding XIL in the Elaborated ADR Process” for an overview of the purposes of these stages). While the developments of current research on Computer Vision in Health IT informed the domain experts and the domain-specific research on Covid-19 informed the researchers in the Diagnosis stage, discussions with clinical personnel (i.e., pneumologists) on the application of ML-based systems in healthcare largely affected the Design stage. For the Design stage, we also introduce the data basis and the technical methods to prepare for the XIL-ADR-based Implementation stage in the subsequent section. The subsection on technical methods also includes (technical) insights from the Implementation cycles, e.g., about the XAI visualizations used. We chose this presentation approach to separate the technical results from the evaluation of the XIL-ADR method, which takes place in Sect. 4, “Evaluation Case Study”.

Figure 3 shows that XIL-ADR can result in several loops/cycles, and each loop can generate new insights. This case study will describe the different loops we conducted in the Implementation stage of our medical case using XIL-ADR with experienced pneumologists and radiologists and how the result of one loop shaped the next loop. In effect, while this section describes the implementation phase of the medical case (conducted with XIL-ADR), it primarily serves as an exemplary evaluation case of our XIL-ADR method. As such, this description of the implementation does not focus on the results of each loop but on the developmental journey using XIL-ADR. The following subsections describe the XIL-ADR loops we conducted in the course of this implementation following the activities of the XIL-ADR cycles (see Fig. 3). In this context, we will also describe the intermediate artifacts (e.g., lessons learned, design requirements) that emerged from each activity. At the beginning of the XIL-ADR process, we describe the Instantiation of the model prototype (subsequent cycles start with Refine Data and Retrain Model). Next, we describe the activity of Explaining, Visualizing and Inspecting the model and its performance. This step forms the basis of the subsequent Reflection on the model explanations and performance, which leads to the generation of Insights. Finally, each loop results in the Definition of XIL Strategies for the subsequent loops. For an overview of the four conducted XIL-ADR cycles, the data basis, and their intermediate artifacts, refer to Table 2 at the end of this section.

This paper presented a novel situational ISDM called XIL-ADR. In engineering the method, we followed the approach of Goldkuhl and Karlsson (2020) by identifying the problem, theorizing, and engineering the method, before demonstrating and evaluating it in a healthcare development case. XIL-ADR is generalizable to iterative ML development-evaluation activities.

From an ADR perspective, XIL-ADR contributes to extending ADR (e.g., Baskerville et al. 2018, p. 365) such that research and organizational teams can use the methodology to conduct ADR more tailored to data science projects. In doing so, it proposes to shift the focus from an organizational BIE perspective (e.g., Sein et al. 2011) toward a more technical BEI perspective with five XIL-tailored activities and a focus on interventions toward the ML artifact, which helps to account for the technical intricacies of data science projects and especially ML-based system development.

Especially the combination of XAI and IML reinforces important parts of ADR (e.g., Mullarkey and Hevner 2019; Sein et al. 2011) in the development cycles and allows the participants to engage in Reflection and Insight activities, resulting in well-informed strategies for reengineering. For research, this may amount to better theoretical insights and to artifacts that are more refined. For practice, this presents the opportunity to achieve more efficient development cycles, with a higher potential to eliminate biases in ML-based IS, compared to other ISDM. Conclusively, XIL-ADR not only presents a valuable guideline for practitioners and researchers alike to uncover problematic issues with underlying ML strategies and data but also serves the purpose of more intriguingly supporting insight generation.

This article contributes in multiple ways: First, we show how XIL-ADR can lead to more meaningful models and help experts to validate existing recommendations and/or generate new insights in their application domain. Interactivity allows experts to force the algorithm to focus on specific areas in the data, and the algorithm can help the expert to better understand the data at hand by highlighting the patterns that determine its classification result. This information can either be used to assess the recommendations made by human experts¹¹ or – in the best case – allow to identify new patterns that have not yet been recognized by human experts before (e.g., Teso and Hinz 2020). In that sense, the human-machine-loop can serve as a method to arrive at novel scientific insights, just like classical statistics helped us to better analyze and understand data in the past decades. We especially believe that the Information Systems discipline will play a crucial role in its development, as this is where human factors and expertise meet with technological developments. We used the example of imaging in medicine because it is very illustrative, but the methodology of XIL-ADR is, in general, also applicable to other types of data and other domains.

Second, we show that a simple training approach for ML models, which is currently widely used, can very often suffer from confounding factors that can heavily influence the results. In our illustrative case, confounders like tubes, marks, or texts (e.g., timestamps) on the X-ray can serve as predictors for classification, while this would not help us to predict new, out-of-sample cases. This problem can partly explain why so many AI classifiers with high accuracy on holdout test sets later fail to deliver in the real world. In tackling this problem and engineering XIL-ADR, we empower IS to fulfill an important mission and work against bias in ML systems (e.g., Kane et al. 2021, p. 375) in an effective manner. Thus, XIL-ADR also contributes to the stream of research on Human-AI Augmentation and directs the AI toward favorable outcomes (e.g., Teodorescu et al. 2021). As Lebovitz et al. (2021) state, many AI models are built quickly and yet achieve high performance metrics, but do so with lots of uncertainty about how these models using the underlying data perform so well, and with suspicions of flaws (e.g., regarding the data or model), which “a diligent evaluation of such tools could have surfaced” (Lebovitz et al. 2021, p. 1515). By combining XAI and IML, we can not only iteratively uncover but also eliminate such flaws and confounding effects, which otherwise would require excessive manual labor or cost-intensive processes. To combat confounders as part of one XIL Strategy in our evaluation case, we resorted to crowdsourcing guided by an instructional video, which helped code nearly 3000 X-rays manually with a customized annotation tool. For other use cases, depending on the type and amount of data, this annotation strategy may not warrant the expected results or may simply be too costly. As an alternative strategy, automated tools offer the chance to (semi-)automatically remove or mark confounders on images or other types of data. Furthermore, research and practice should experiment with the inclusion of diverse kinds of users in different roles, as well as different collaborative constellations to find effective and efficient strategies for specific ML tasks. From the viewpoint of data science, this is undoubtedly an important research field that could help XIL-ADR to become productive.

Third, we propose that task-specific visualization approaches to XAI may improve the activities of Reflection and Insight in XIL-ADR. Our approach, for example, extends current state-of-the-art visualization by incorporating the special characteristics of X-rays, the application domain medicine, and, in particular, radiology, which helped to analyze the domain data better. An iterative comparison to the constantly evolving stream of methods in XAI (e.g., Evans et al. 2022) could generate further extensions of such customized XAI solutions for the XIL-ADR process (e.g., XRAI by Kapishnikov et al. (2019), or Guided Integrated Gradients by Kapishnikov et al. (2021)). Recent research also suggests that even ostensibly insightful XAI methods should be subjected to vicious inspection of their fidelity (e.g., Adebayo et al. 2020; Tomsett et al. 2020). It would thus be recommended to try and assess different XAI methods (e.g., Teso and Kersting 2019) for the purpose of inspection, especially in the earlier cycles of XIL-ADR, and to compare their results regarding uncovering potential bias or confounding factors. For doing so, research on and methods of the assessment of causability of XAI (i.e., effectiveness in explaining) can prove of high value (Holzinger and Müller 2021).

Unlike CRISP-DM, XIL-ADR is embedded in a larger development context and benefits from prior and posterior defined activities in the ADR process, e.g., considering the business problem and its domain in the Diagnosis stage. Unlike other CRISP-DM adaptations, such as CRISP-ML (e.g., Studer et al. 2021), XIL-ADR explicitly incorporates XAI and IML methods and accounts for the data-driven intricacies of ML projects. Lastly, XIL-ADR – in contrast to traditional implementation phases in ADR – puts heavy emphasis on technical and data aspects while simultaneously focusing development team efforts on defining appropriate and effective strategies for subsequent ADR cycles.

In this paper, we proposed the methodology of XIL-ADR to enable research and practice to construct AI-based systems that are more reliable. We outlined in which way XIL-ADR presents an opportunity for improving machine-learning projects. We showed that ML projects can largely profit from the iterative XIL-ADR process by refining the model and the underlying data, training, and testing strategies, and also how such a process benefits from including multiple kinds of users. Lastly, through this iterative refinement of promising models, the full potential of AI for organizational learning may be unleashed.

Although XIL-ADR comprises some destructive constructionism which can be painful at times, we should not underestimate the potential of AI-based systems. Instead, we believe XIL-ADR helps to carefully grind the rough diamond of ML projects to arrive at actionable solutions that efficiently support organizational goals eventually.

All in all, by creating a virtuous circle of human and machine collaboration, we could not only make our systems smarter but also allow different kinds of users to enhance their knowledge in the focal application domain. This vision, however, requires potent AI systems that allow for interactivity, explainability, accountability, and a better understanding of this human-machine hybridity loop.