Handling uncertainty in self-adaptive systems: an ontology-based reinforcement learning model

Ubiquitous and pervasive systems interact with each other and perform actions favoring the emergence of a global desired behavior. To function well, these systems need to be self-adaptive to handle noisy data and partially observable dynamic environments. However, facing unpredictable and rare events while only accessing incomplete information about the environment causes uncertainty in the adaptation process. Such uncertainty results in inconsistent decisions and unexpected system behavior. Currently, SAS handles such unpredictable conditions using adaptive modeling mechanisms to select default actions or exploiting reinforcement learning (RL) algorithms to learn new actions. However, the current mechanisms do not address rare events in an environment. This paper improves the system’s decision-making when facing rare events by providing extra information about alternative adaptation actions using domain ontologies, which provide a thorough understanding of a domain. In this paper, we propose an Ontology-based unCertainty handling model (OnCertain), which enables the RL-based system to augment its observation and reason about the rare event using prior ontological knowledge. The overall aim of this model is to improve the system’s decision-making process under conditions of uncertainty. Our model is evaluated in a traffic signal control system and an edge computing environment. The results show that the OnCertain model can improve the RL-based systems’ observation and, consequently, their performance in such environments.