nach oben

AI & SOCIETY

Erschienen in:

26.07.2022 | Open Forum

An explanation space to align user studies with the technical development of Explainable AI

verfasst von: Garrick Cabour, Andrés Morales-Forero, Élise Ledoux, Samuel Bassetto

Erschienen in: AI & SOCIETY | Ausgabe 2/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Providing meaningful and actionable explanations for end-users is a situated problem requiring the intersection of multiple disciplines to address social, operational, and technical challenges. However, the explainable artificial intelligence community has not commonly adopted or created tangible design tools that allow interdisciplinary work to develop reliable AI-powered solutions. This paper proposes a formative architecture that defines the explanation space from a user-inspired perspective. The architecture comprises five intertwined components to outline explanation requirements for a task: (1) the end-users’ mental models, (2) the end-users’ cognitive process, (3) the user interface, (4) the Human-Explainer Agent, and (5) the agent process. We first define each component of the architecture. Then, we present the Abstracted Explanation Space, a modeling tool that aggregates the architecture’s components to support designers in systematically aligning explanations with end-users’ work practices, needs, and goals. It guides the specifications of what needs to be explained (content: end-users’ mental model), why this explanation is necessary (context: end-users’ cognitive process), to delimit how to explain it (format: Human-Explainer Agent and user interface), and when the explanations should be given. We then exemplify the tool’s use in an ongoing case study in the aircraft maintenance domain. Finally, we discuss possible contributions of the tool, known limitations or areas for improvement, and future work to be done.

Vorheriger Artikel Training philosopher engineers for better AI

Nächster Artikel Evidence-based AI, ethics and the circular economy of knowledge

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

https://aiir.nl/

https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai.html

In this article, the word end-user refers to the actors who interact directly with the technical device.

A user’s understanding and representation of a process, a phenomenon, or a system (Hoffman et al., 2018).

See Hoffman et al. (2018), Wilson & Sharples (2015), and Milton (2007) for a complete description of existing data collection methods and techniques.

https://www.mitacs.ca/en/projects/automated-visual-inspection-sentencing-dressing.

This simplified process does not list the upstream tasks related to the preparation of the workstation, or the downstream operations after the response.

Akatsuka J, Yamamoto Y, Sekine T, Numata Y, Morikawa H, Tsutsumi K, Yanagi M, Endo Y, Takeda H, Hayashi T (2019) Illuminating clues of cancer buried in prostate mr image: deep learning and expert approaches. Biomolecules 9(11):673CrossRef

Bisantz A, Roth EM, Watts-Englert J (2015) Study and analysis of complex cognitive work. In: Evaluation of Human Work, edited by John R. Wilson and Sarah Sharples, 61–82. CRC Press.

Cabitza F, Zeitoun J-D (2019) The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Ann Trans Med 7 (8). https://doi.org/10.21037/atm.2019.04.07.

Cabour G, Ledoux É, Bassetto S (2021a) Extending system performance past the boundaries of technical maturity: human-agent teamwork perspective for industrial inspection. In: Proceedings of the 21st Congress of the International Ergonomics Association (IEA 2021a), edited by Nancy L. Black, W. Patrick Neumann, and Ian Noy, 75–83. Cham: Springer International Publishing.

Cabour G, Ledoux É, Bassetto S (2021b) A work-centered approach for cyber-physical-social system design: applications in aerospace industrial inspection. ArXiv:2101.05385 [Cs], January.

Cabour G, Ledoux É, Bassetto S (2022) Aligning work analysis and modeling with the engineering goals of a cyber-physical-social system for industrial inspection. Appl Ergonomics.

Chen JYC, Barnes MJ (2014) Human–agent teaming for multirobot control: a review of human factors issues. IEEE Trans Hum-Mach Syst 44(1):13–29. https://doi.org/10.1109/THMS.2013.2293535CrossRef

Chen JY, Procci K, Boyce M, Wright J, Garcia A, Barnes M (2014) Situation Awareness-Based Agent Transparency. Army research lab aberdeen proving ground md human research and engineering

Clancey WJ (1983) The epistemology of a rule-based expert system—a framework for explanation. Artif Intell 20(3):215–251. https://doi.org/10.1016/0004-3702(83)90008-5CrossRef

Crabtree A, Rouncefield M, Tolmie P (2012) Doing design ethnography. SpringerCrossRef

Cutillo CM, Sharma KR, Foschini L, Kundu S, Mackintosh M, Mandl KD (2020) Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency. Npjdigital Medicine 3(1):1–5. https://doi.org/10.1038/s41746-020-0254-2CrossRef

Darius A, Damaševičius R (2014) Gamification of a Project Management System. In: Proc. of Int. conference on advances in computer-human interactions ACHI2014, 200–207. Citeseer.

Demir M, McNeese NJ, Cooke NJ (2020) Understanding human-robot teams in light of all-human teams: aspects of team interaction and shared cognition. Int J Hum Comput Stud 140:102436CrossRef

Dhanorkar S, Wolf CT, Qian K, Xu A, Popa L, Li Y (2021) Who needs to know what, when?: Broadening the explainable AI (XAI) design space by looking at explanations across the ai lifecycle. In: Designing Interactive Systems Conference 2021, 1591–1602. Virtual Event USA: ACM. https://doi.org/10.1145/3461778.3462131.

Dong H, Kechen Song Yu, He JX, Yan Y, Meng Q (2019) PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans Industr Inf 16(12):7448–7458CrossRef

Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. ArXiv:1702.08608 [Cs, Stat], February. http://arxiv.org/abs/1702.08608.

Elsawah S, Guillaume JHA, Filatova T, Rook J, Jakeman AJ (2015) A methodology for eliciting, representing, and analysing stakeholder knowledge for decision making on complex socio-ecological systems: from cognitive maps to agent-based models. J Environ Manage 151(March):500–516. https://doi.org/10.1016/j.jenvman.2014.11.028CrossRef

Endsley MR, Hoffman R, Kaber D, Roth E (2007) Cognitive engineering and decision making: an overview and future course. J Cognit Eng Decision Making 1(1):1–21CrossRef

Fidel G, Bitton R, Shabtai A (2020) When Explainability Meets Adversarial Learning: Detecting Adversarial Examples Using SHAP Signatures. In: 2020 International Joint Conference on Neural Networks (IJCNN), 1–8. IEEE.

Friedman S, Forbus K, Sherin B (2018) Representing, running, and revising mental models: a computational model. Cogn Sci 42(4):1110–1145CrossRef

Goh YM, Micheler S, Sanchez-Salas A, Case K, Bumblauskas D, Monfared R (2020) A variability taxonomy to support automation decision-making for manufacturing processes. Prod Planning Control 31(5):383–399CrossRef

Government of Canada (2021). Responsible use of artificial intelligence (ai). Retrieved from https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai.html

Haberfellner R, de Weck O, Fricke E, Vössner S (2019) Process models: systems engineering and others. In: Systems Engineering, 27–98. Springer.

Habli I, Lawton T, Porter Z (2020) Artificial intelligence in health care: accountability and safety. Bull World Health Organ 98(4):251–256. https://doi.org/10.2471/BLT.19.237487CrossRef

He Yu, Song K, Meng Q, Yan Y (2019) An end-to-end steel surface defect detection approach via fusing multiple hierarchical features. IEEE Trans Instrum Meas 69(4):1493–1504CrossRef

Henelius A, Puolamäki K, Ukkonen A (2017) Interpreting classifiers through attribute interactions in datasets. ArXiv Preprint ArXiv:1707.07576.

Hoffman RR, Mueller ST, Klein G, Litman J (2018) Metrics for explainable AI: challenges and prospects. ArXiv Preprint ArXiv:1812.04608.

Imran A, Jeon G, Piccialli F (2022) From artificial intelligence to explainable artificial intelligence in industry 4.0: a survey on what, how, and where. IEEE Trans Ind Inform 1–1. https://doi.org/10.1109/TII.2022.3146552

Johnson M, Bradshaw JM, Feltovich PJ (2017) Tomorrow’s human-machine design tools: from levels of automation to Interdependencies. J Cognit Eng Decision Making, October. https://doi.org/10.1177/1555343417736462CrossRef

Johnson M, Bradshaw JM (2021) The role of interdependence in trust. In: Trust in Human-Robot Interaction, 379–403. Elsevier.

Klein G, Wright C (2016) Macrocognition: from theory to toolbox. Front Psychol 7. https://doi.org/10.3389/fpsyg.2016.00054

Kobrin JL, Sinharay S, Haberman SJ, Chajewski M (2011) An investigation of the fit of linear regression models to data from an SAT® validity study. ETS Research Report Series 2011(1):i–21CrossRef

Konig R, Johansson U, Niklasson L (2008) G-REX: A versatile framework for evolutionary data mining. In: 2008 IEEE International Conference on Data Mining Workshops, 971–74. IEEE.

Lakkaraju H, Bach SH, Leskovec J (2016) Interpretable decision sets: a joint framework for description and prediction. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1675–84.

Langer M, Landers RN (2021) The future of artificial intelligence at work: a review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers. Comput Hum Behav 123(October):106878. https://doi.org/10.1016/j.chb.2021.106878CrossRef

Lapuschkin S, Wäldchen S, Binder A, Montavon G, Samek W, Müller K-R (2019) Unmasking clever hans predictors and assessing what machines really learn. Nat Commun 10(1):1–8CrossRef

Lewis M, Li H, Sycara K (2021) Deep learning, transparency, and trust in human robot teamwork. In: Trust in Human-Robot Interaction, 321–52. Elsevier.

Lockton D, Brawley L, Aguirre Ulloa M, Prindible M, Forlano L, Rygh K, Fass J, Herzog K, Nissen B (2019) Tangible thinking: materializing how we imagine and understand systems, experiences, and relationships.

Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Adv Neural Inform Process Syst 30.

Marcus G, Davis E (2019) Rebooting AI: Building Artificial Intelligence We Can Trust. Vintage.

Matthews G, Panganiban AR, Lin J, Long M, Schwing M (2021) Super-Machines or Sub-Humans: Mental Models and Trust in Intelligent Autonomous Systems. In: Trust in Human-Robot Interaction, 59–82. Elsevier.

McMeekin N, Olivia Wu, Germeni E, Briggs A (2020) How methodological frameworks are being developed: evidence from a scoping review. BMC Med Res Methodol 20(1):1–9CrossRef

Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1–38MathSciNetCrossRefMATH

Milton NR (2007) Knowledge Acquisition in Practice: A Step-by-Step Guide. Springer Science & Business Media.

Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Deborah Raji I, Gebru T (2019) Model Cards for Model Reporting. In: Proceedings of the conference on fairness, accountability, and transparency, 220–29. FAT* ‘19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287596.

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236CrossRef

Mohseni S, Zarei N, Ragan ED (2021) A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Trans Interactive Intell Syst (TiiS) 11(3–4):1–45

Moore J (1988) Explanation in expert systems: a survey. In.

Morales-Forero A, Bassetto S (2019) Case study: a semi-supervised methodology for anomaly detection and diagnosis. In: 2019 IEEE international conference on industrial engineering and engineering management (IEEM), 1031–37. IEEE.

Morales-Forero A, Bassetto S, Coatanea E (in press). Toward safe AI. AI & Society.

Mor-Yosef S, Samueloff A, Modan B, Navot D, Schenker JG (1990) Ranking the risk factors for cesarean: logistic regression analysis of a nationwide study. Obstet Gynecol 75(6):944–947

Mueller ST, Hoffman RR, Clancey W, Emrey A, Klein G (2019) Explanation in human-ai systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. ArXiv:1902.01876 [Cs], February. http://arxiv.org/abs/1902.01876.

Mueller ST, Veinott ES, Hoffman RR, Klein G, Alam L, Mamun T, Clancey WJ (2021) Principles of explanation in human-AI systems ArXiv:2102.04972 [Cs], February. http://arxiv.org/abs/2102.04972.

Muller M, Wolf CT, Andres J, Desmond M, Joshi NN, Ashktorab Z, Sharma A et al. (2021) Designing ground truth and the social life of labels. In: Proceedings of the 2021 CHI conference on human factors in computing systems, 1–16. Yokohama Japan: ACM. https://doi.org/10.1145/3411764.3445402.

Naiseh M, Jiang N, Ma J, Ali R (2020) Personalising explainable recommendations: literature and conceptualisation. In: World conference on information systems and technologies, 518–33. Springer.

National academies of sciences, engineering, and medicine. 2021. Human-AI Teaming: State of the Art and Research Needs. Washington, DC: The National Academies Press. https://doi.org/10.17226/26355.

Nickerson RC, Varshney U, Muntermann J (2013) A method for taxonomy development and its application in information systems. Eur J Inf Syst 22(3):336–359CrossRef

Nunes I, Jannach D (2017) A systematic review and taxonomy of explanations in decision support and recommender systems. User Model User-Adap Inter 27(3):393–444CrossRef

Pekka AP, Bauer W, Bergmann U, Bieliková M, Bonefeld-Dahl C, Bonnet Y, Bouarfa L (2018) The European commission’s high-level expert group on artificial intelligence: ethics guidelines for trustworthy Ai. Working Document for Stakeholders’ Consultation. Brussels, 1–37.

Poursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H (2021) Manipulating and measuring model interpretability. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–52. CHI ‘21. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3411764.3445315.

Preece A, Harborne D, Braines D, Tomsett R, Chakraborty S (2018) Stakeholders in explainable AI. ArXiv Preprint ArXiv:1810.00184.

Rajabiyazdi F, Jamieson GA (2020) A review of transparency (seeing-into) models. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 302–8. https://doi.org/10.1109/SMC42975.2020.9282970.

Rathi S (2019) Generating counterfactual and contrastive explanations using SHAP.” ArXiv Preprint ArXiv:1906.09293.

Ribeiro MT, Singh S, Guestrin C (2016) Why Should I Trust You?’: Explaining the Predictions of Any Classifier. ArXiv:1602.04938 [Cs, Stat], August. http://arxiv.org/abs/1602.04938.

Ribera M, Lapedriza A (2019) Can we do better explanations? A proposal of user-centered explainable AI. Los Angeles, 7.

Rolls-Royce (2021) The aletheia framework. Retrieved 2021–07–01, from https://www.rolls-royce.com/sustainability/ethics-and-compliance/the-aletheia-framework.aspx

Roth EM, Bennett KB, Woods DD (1987) Human interaction with an ‘Intelligent’ machine. Int J Man Mach Stud 27(5):479–525. https://doi.org/10.1016/S0020-7373(87)80012-3CrossRef

Roth EM, Bisantz AM, Wang X, Kim T, Hettinger AZ (2021) A work-centered approach to system user-evaluation. J Cognit Eng Decision Making 15(4):155–174CrossRef

Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206–215CrossRef

Salembier P, Wagner I (2021) Studies of work ‘in the wild.’ Computer Supported Cooperative Work (CSCW) 30(2):169–188CrossRef

Sanneman L, Shah JA (2020) A situation awareness-based framework for design and evaluation of explainable AI. In: Calvaresi D, Najjar A, Winikoff M, Främling K (eds) Explainable, transparent autonomous agents and multi-agent systems. Springer International Publishing, Cham, pp 94–110CrossRef

Seeber I, Bittner E, Briggs RO, de Vreede T, de Vreede G-J, Elkins A, Maier R et al (2020) Machines as teammates: a research agenda on AI in team collaboration. Information & Management 57(2):103174. https://doi.org/10.1016/j.im.2019.103174CrossRef

Selbst AD, Boyd D, Friedler SA, Venkatasubramanian S, Vertesi J (2019) Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. FAT* ‘19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287598.

Shadbolt N, Smart PR (2015) Knowledge elicitation: methods, tools and techniques. In: Evaluation of human work, edited by John R. Wilson and Sarah Sharples, 163–200. CRC Press. https://eprints.soton.ac.uk/359638/.

Shahri A, Hosseini M, Phalp K, Taylor J, Ali R (2014) Towards a code of ethics for gamification at enterprise. In: IFIP working conference on the practice of enterprise modeling, 235–45. Springer.

Shepherd A (2015) Task analysis. In: Evaluation of Human Work, 4th ed. CRC Press.

Shmelova T, Sterenharz A, Dolgikh S (2020) Artificial intelligence in aviation industries: methodologies, education, applications, and opportunities. In: Handbook of research on artificial intelligence applications in the aviation and aerospace industries, 1–35. IGI Global.

Shneiderman B (2020) Human-centered artificial intelligence: reliable, safe & trustworthy. Int J Hum-Comput Interact 36(6):495–504. https://doi.org/10.1080/10447318.2020.1741118CrossRef

Song K, Yan Y (2013) A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Appl Surf Sci 285:858–864CrossRef

St-Vincent M, Vézina N, Bellemare M, Denis D, Ledoux É, Imbeau D (2014) Ergonomic intervention. Institut de recherche Robert-Sauvé en santé et en sécurité du travail.

Tomsett R, Widdicombe A, Xing T, Chakraborty S, Julier S, Gurram P, Rao R, Srivastava M (2018) Why the failure? How adversarial examples can provide insights for interpretable machine learning. In: 2018 21st international conference on information fusion (FUSION), 838–45. IEEE.

Tramer F, Boneh D (2019) Adversarial training and robustness for multiple perturbations. Advances in Neural Information Processing Systems 32.

Vasey B, Clifton DA, Collins GS, Denniston AK, Faes L, Geerts BF, Liu X et al (2021) DECIDE-AI: New reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med 27(2):186–187. https://doi.org/10.1038/s41591-021-01229-5CrossRef

Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. CRC PressCrossRef

Wilson JR, Sharples S (2015) Evaluation of human work. CRC PressCrossRef

Xiao C, Li B, Zhu J-Y, He W, Liu M, Song D (2018) Generating adversarial examples with adversarial networks. ArXiv Preprint ArXiv:1801.02610.

Yeung K (2020) Recommendation of the council on artificial intelligence (OECD). Int Leg Mater 59(1):27–34CrossRef

Zouinar M (2020) Évolutions de l’Intelligence Artificielle : quels enjeux pour l’activité humaine et la relation Humain-Machine au travail? Activités, no. 17–1 (April). https://doi.org/10.4000/activites.4941.

Zsambok CE, Klein G (2014) Naturalistic decision making. Psychology PressCrossRef

Titel: An explanation space to align user studies with the technical development of Explainable AI
verfasst von: Garrick Cabour
Andrés Morales-Forero
Élise Ledoux
Samuel Bassetto
Publikationsdatum: 26.07.2022
Verlag: Springer London
Erschienen in: AI & SOCIETY / Ausgabe 2/2023
Print ISSN: 0951-5666
Elektronische ISSN: 1435-5655
DOI: https://doi.org/10.1007/s00146-022-01536-6

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 2/2023

Against the new space race: global AI competition and cooperation for people

Intelligent service robots for elderly or disabled people and human dignity: legal point of view

Responsibility of AI Systems

Posthuman perception of artificial intelligence in science fiction: an exploration of Kazuo Ishiguro’s Klara and the Sun

Algorithmic fairness through group parities? The case of COMPAS-SAPMOC

A machine learning approach to detecting fraudulent job types

Premium Partner