Reliability of eXplainable Artificial Intelligence in Adversarial Perturbation Scenarios

Nowadays, Deep Neural Networks (DNNs) are widely adopted in several fields, including critical systems, medicine, self-guided vehicles etc. Among the reasons sustaining this spread there are the higher generalisation ability and performance levels that DNNs usually obtain when compared to classical machine learning models. Nonetheless, their black-box nature raises ethical and judicial concerns that lead to a lack of confidence in their use in sensitive applications. For this reason, recently there has been a growing interest in eXplainable Artificial Intelligence (XAI), a field providing tools, techniques and algorithms designed to generate interpretable explanations, comprehensible to humans, for the decisions made by a machine learning model. However, it has been demonstrated that DNNs are susceptible to Adversarial Perturbations (APs), namely procedures intended to mislead a target model by means of an almost imperceptible noise. The relation existing between XAI and AP is extremely of interest since it can help improve trustworthiness in AI-based systems. To this aim, it is important to increase awareness of the risks associated with the use of XAI in critical contexts, in a world where APs are present and easy to perform. On this line, we quantitatively analyse the impact that APs have on XAI in terms of differences in the explainability maps. Since this work wants to be just an intuitive proof-of-concept, the aforementioned experiments are run in a fashion easy to understand and to quantify, by using publicly available dataset and algorithms. Results show that AP can strongly affect the XAI outcomes, even in the case of a failed attack, highlighting the need for further research in this field.