Cognitively Inspired Three-Way Decision Making and Bi-Level Evolutionary Optimization for Mobile Cybersecurity Threats Detection: A Case Study on Android Malware

Malicious apps use a variety of methods to spread infections, take over computers and/or IoT devices, and steal sensitive data. Several detection techniques have been proposed to counter these attacks. Despite the promising results of recent malware detection strategies, particularly those addressing evolving threats, inefficiencies persist due to potential inconsistency in both the generated malicious malware and the pre-specified detection rules, as well as their crisp decision-making process. In this paper, we propose to address these issues by (i) considering the detection rules generation process as a Bi-Level Optimization Problem, where a competition between two levels (an upper level and a lower one) produces a set of effective detection rules capable of detecting new variants of existing and even unseen malware patterns. This bi-level strategy is subtly inspired by natural evolutionary processes, where organisms adapt and evolve through continuous interaction and competition within their environments. Furthermore, (ii) we leverage the fundamentals of Rough Set Theory, which reflects cognitive decision-making processes, to assess the true nature of artificially generated malicious patterns. This involves retaining only the consistent malicious patterns and detection rules and categorizing these rules into a three-way decision framework comprising accept, abstain, and reject options. Our novel malware detection technique outperforms several state-of-the-art methods on various Android malware datasets, accurately predicting new apps with a 96.76% accuracy rate. Moreover, our approach is versatile and effective in detecting patterns applicable to a variety of cybersecurity threats.