Self-organized femtocells: a Fuzzy Q-Learning approach

We introduce in this paper the innovative concept of self-organized femtocells for future generation broadband cellular networks. Since the home is the basic unit at which femtocells will be located, their deployment will be massive and their number and position unknown to the operator. This requires femtocells to be autonomous and self-organized, and able to work without human intervention. We propose self-organization to be implemented through Reinforcement Learning (RL) and femtocells to make transmission decisions as a multiagent system, with the objective of maximizing the system capacity and not generating additional interference to the traditional macrocell network. In particular, we manage the femto-to-macro aggregated interference, in realistic wireless settings, by means of Q-Learning (QL) techniques, which allow the femtocells to learn online and distributively the most appropriate resource allocation policy by continuous interactions with the environment. However, QL is based on discrete representation of state and action spaces, which makes the proposed approach not independent of the environment and designer criterion, since it requires a significant human intervention in the definition of the state and action spaces. As a result, we propose to optimize the self-organization capabilities of the proposed scheme by combining QL with the Fuzzy Inference System theory. We then propose a Fuzzy Q-Learning approach which allows avoiding the subjectivity of the QL design with continuous state and action representation, besides improving performance and convergence capabilities. We evaluate simulation results in a 3rd Generation Partnership Project (3GPP) compliant scenario and we compare them to heuristic approaches. Results will show the unique ability of these RL approaches to self-adapt to the dynamics of realistic wireless scenarios. Finally, we discuss the implementability of the proposed schemes in 3GPP systems, and in terms of memory and computational requirements.