A multi-state Q-learning based CSMA MAC protocol for wireless networks

Due to the shared nature of wireless channels, competition among the nodes to access media is inevitable. P-persistent carrier sense multiple access (CSMA) is a medium access control scheme widely used for resource allocation in wireless environments. The probability of transmission highly affects the throughput. We model the wireless nodes as agents in a network game. The strategy of an agent is defined as the probability of transmission. Agents don’t have a priori information about the network (e.g., number of nodes, other agents strategies) and learn their optimal strategy using the historical sensory information including the number of collisions or successful transmissions. In this paper, a multi-state reinforcement learning (RL) method is designed for this purpose. The main challenge in designing an RL agent is to define the states of the environment from agent’s perspective. For this purpose, in this paper, various state representations are proposed in a multi-state Q-learning model. This leads to different agents personalities ranging from cautious agents with risk aversion to aggressive risky agents. We also propose agents with combined personalities of cautiousness and aggressiveness. The performance of the proposed Q-learning agents with different state definitions in comparison with each other and also other benchmarking agents is examined via comprehensive simulation results.