Upol Ehsan
Mark O. Riedl
ABSTRACT
We introduce \em AI rationalization, an approach for generating explanations of autonomous system behavior as if a human had performed the behavior. We describe a rationalization technique that uses neural machine translation to translate internal state-action representations of an autonomous agent into natural language. We evaluate our technique in the Frogger game environment, training an autonomous game playing agent to rationalize its action choices using natural language. A natural language training corpus is collected from human players thinking out loud as they play the game. We motivate the use of rationalization as an approach to explanation generation and show the results of two experiments evaluating the effectiveness of rationalization. Results of these evaluations show that neural machine translation is able to accurately generate rationalizations that describe agent behavior, and that rationalizations are more satisfying to humans than other alternative methods of explanation.
- Andreas, J.; Dragan, A. D.; and Klein, D. 2017. Translating neuralese. CoRR abs/1704.06960.Google Scholar
- Aronson, J. 1995. A pragmatic view of thematic analysis. The qualitative report 2(1):1--3.Google Scholar
- Core, M.; Lane, H. C.; van Lent, M.; Gomboc, D.; Solomon, S.; and Rosenberg, M. 2006. Building Explainable Artificial Intelligence Systems. In Proceedings of the 18th Innovative Applications of Artificial Intelligence Conference. Google ScholarDigital Library
- Dorst, K., and Cross, N. 2001. Creativity in the design process: co-evolution of problem--solution. Design studies 22(5):425--437.Google Scholar
- Fonteyn, M. E.; Kuipers, B.; and Grobe, S. J. 1993. A description of think aloud method and protocol analysis. Qualitative Health Research 3(4):430--441.Google ScholarCross Ref
- Goldman, A. I., et al. 2012. Theory of mind. The Oxford handbook of philosophy of cognitive science 402--424.Google Scholar
- Hollander, M.; Wolfe, D. A.; and Chicken, E. 2013. Nonparametric statistical methods. John Wiley & Sons.Google Scholar
- Krause, J.; Perer, A.; and Ng, K. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5686--5697. ACM. Google ScholarDigital Library
- Laird, J., and van Lent, M. 2001. Human-level aiâ's killer application: Interactive computer games. AI Magazine 22(2):15--25.Google Scholar
- Litman, L.; Robinson, J.; and Abberbock, T. 2017. Turkprime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods 49(2):433--442.Google Scholar
- Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1412--1421. Lisbon, Portugal: Association for Computational Linguistics.Google Scholar
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.; Veness, J.; Bellemare, M.; Graves, A.; Riedmiller, M.; Fidjeland, A.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529--533.Google ScholarCross Ref
- Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, 311--318. Association for Computational Linguistics. Google ScholarDigital Library
- Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135--1144. ACM. Google ScholarDigital Library
- Strauss, A., and Corbin, J. 1994. Grounded theory methodology. Handbook of qualitative research 17:273--85.Google Scholar
- van Lent, M.; ; Carpenter, P.; McAlinden, R.; and Brobst, P. 2005. Increasing replayability with deliberative and reactive planning. In 1st Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 135--140. Google ScholarDigital Library
- van Lent, M.; Fisher, W.; and Mancuso, M. 2004. An explainable artificial intelligence system for small-unit tactical behavior. In Proceedings of the 16th conference on Innovative Applications of Artifical Intelligence. Google ScholarDigital Library
- Watkins, C., and Dayan, P. 1992. Q-learning. Machine Learning 8(3--4):279-292. Google ScholarDigital Library
- Weston, J.; Bordes, A.; Chopra, S.; Rush, A. M.; van Merriënboer, B.; Joulin, A.; and Mikolov, T. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.Google Scholar
- Yosinski, J.; Clune, J.; Fuchs, T.; and Lipson, H. 2015. Understanding neural networks through deep visualization. In In ICMLWorkshop on Deep Learning. Citeseer.Google Scholar
- Zeiler, M. D., and Fergus, R. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision, 818--833. Springer.Google Scholar
Recommendations
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
IUI '19: Proceedings of the 24th International Conference on Intelligent User InterfacesAutomated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human ...
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningDespite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when ...
Expanding Explainability: Towards Social Transparency in AI systems
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsAs AI-powered systems increasingly mediate consequential decision-making, their explainability is critical for end-users to take informed and accountable actions. Explanations in human-human interactions are socially-situated. AI systems are often socio-...
Comments