Intelligent deep reinforcement learning-based scheduling in relay-based HetNets

With the increasing demand for high-speed data transmission, it is increasingly important to improve network capacity. One promising solution that has garnered significant attention recently is the heterogeneous networks (HetNets). By deploying different types of elements, such as picocells, femtocells, and relay stations, within the macrocells coverage, HetNets can increase the potential for spatial resource reuse and enhance the quality of service (QoS) of wireless users [1]. Meanwhile, due to the explosive growth of multimedia traffic, broadcasting/multicasting has recently received much interest as an efficient means for dissemination of multimedia information to wireless users, e.g., 5G New Radio Multimedia Broadcast/Multicast Service (NR-MBMS) [2, 3].

In this paper, we consider a fundamental problem of minimum-delay file dissemination in two-hop relay-based HetNets. As shown in Fig. 1, the system is composed of a macro base station (BS), a half-duplex relay station (RS), and multiple wireless users associated with the RS. Such a two-hop relaying structure is a key building block from which to construct versatile networks in 5G/6G [4, 5]. The BS intends to disseminate a block of packets to all users over time-varying channels. To minimize the dissemination delay, rateless code [6] is employed at the BS, such that coded packets are disseminated to the users, and the original packets can be decoded by a user once a sufficient number of coded packets are received.

At each time slot, the half-duplex RS can either broadcast a coded packet to the users or fetch a coded packet from the BS. Due to the dynamic nature of the wireless environment, at each time slot, only a subset of users are able to receive a transmitted packet successfully. The goal is to minimize the file dissemination delay by making optimal channel-aware scheduling decisions at the RS, i.e., either fetching or broadcasting a packet at each time slot.

Due to the time-varying nature of wireless channels, it is desirable for the RS to broadcast a packet if the overall channel states between the RS and the users are favorable, e.g., when most of the users are able to receive the broadcast packet at the current time slot successfully. Note that, the users’ channel conditions are typically asymmetric in practice, such that the dissemination delay is usually bottlenecked by the users with poor channel conditions. Hence, the RS should also pay more attention to the users with poor channel conditions. Moreover, making a decent balance between fetching and broadcasting packets at the RS is important, such that the RS buffer can be replenished with new packets in time during the transmission, and also the users can receive packets with high rates. However, it is challenging to make optimal scheduling decisions at the RS, even for small-sized systems (i.e., the number of users is small).

Our contributions are summarized below. We study a channel-aware scheduling problem for efficient file dissemination in two-hop relay-based HetNets. We model the scheduling problem at the half-duplex RS as a Markov decision process (MDP) and propose a deep reinforcement learning (DRL)-based intelligent scheduling algorithm. The proposed scheduling algorithm does not rely on the knowledge of statistic channel parameters, and can be used in general scenarios where the BS-to-RS channel is not ideal. We also extend the proposed algorithm to adapt to the network dynamics. Through simulations, we show that the proposed algorithm performs close to a lower bound on dissemination delay and significantly outperforms baseline schemes such as the uncoded Automatic Repeat-reQuest (ARQ).

The organization of this paper is summarized below. The related work is presented in Sect. 2. Section 3 describes the system model. The MDP formulation of our problem and the proposed scheduling algorithm are presented in Sect. 4. Simulation results are provided in Sect. 5. Finally, Sect. 6 concludes this paper.

There have been numerous studies on multicast/broadcast in single-hop networks. In [7], the authors provide precise bounds on the maximum throughput after analyzing the throughput performance of rateless code in single-hop broadcast networks. The study in [7] is focused on symmetric channel distributions; however, in our study, we consider a generic asymmetric channel distribution. Sim et al. [8] and Low et al. [9] study opportunistic coded multicast scheduling policies over a single-hop network, where the key trade-off is between multi-user diversity and multicast gain. Khamfroush et al. [10] proposes a scheduling policy for the transmission of coded packets in cooperative networks. However, the authors in [10] employ a dynamic programming-based approach, that is computationally intractable when the network size increases. Coded multicast scheduling over relay-based HetNets is investigated in [11], where the authors employ fluid relaxation to formulate the scheduling problem at the half-duplex RS. A threshold-based policy termed WBP-TRACK is proposed, and a lower bound on dissemination delay is derived. However, they only consider the special case where the BS-to-RS channel is perfect, which is different from ours.

Recently, reinforcement learning (RL) has seen wide spread applications for wireless communication networks, particularly in decision-making problems [12, 13]. Iqbal et al. [14] studies the adaptive resource allocation scheme based on Q-learning in HetNets. The cell handover problem in HetNets is studied in [15], where a context-aware scheduling algorithm based on RL is proposed. The authors in [16] propose a sleep scheduling algorithm based on RL for compressive data gathering, which aims to enhance the energy efficiency and prolong the lifespan of wireless sensor networks. The authors in [17] investigate a downlink power allocation problem in HetNets and propose power allocation algorithms based on Q-learning. The spectrum sharing issue in cognitive radio networks is studied in [18] using RL, where the authors propose an RL-based channel sensing and selection algorithm, which does not rely on the knowledge of system parameters. The above works are all based on the framework of RL. However, when the state and/or action space grows large, the effectiveness of RL decreases. In that case, DRL becomes a more appropriate choice for decision-making problems. Tang et al. [19] proposes a DRL-based algorithm for real-time wireless resource allocation in high mobility HetNets. The authors in [20] study efficient power allocation in HetNets and propose a DRL-based power allocation algorithm. The work [21] considers end-to-end network slicing enabling HetNets, and proposes a switching algorithm using DRL. Ma and Wong [22] and Lie t al. [23] investigate dynamic content caching in HetNets and propose efficient DRL-based scheduling algorithms for updating cache content. Cong and Lang [24] studies the dynamic multichannel access problem in multi-user wireless networks, and proposes a DRL-based algorithm to enhance the spectrum utilization in dynamic spectrum access. In [25], the authors consider a joint relay selection and power allocation problem in two-hop cooperative relay networks, and propose a hierarchical reinforcement learning-based policy to minimize the outage probability. However, they simply assume a fixed operation pattern at half-duplex relays, i.e., one time slot for fetching a packet and the next time slot for broadcasting a packet, which is different from our paper.

In summary, the existing scheduling schemes used in relay-based HetNets either have high computational complexity or only apply to systems with special structures. By contrast, in this paper, we consider a general system model and propose a DRL-based scheduling scheme that surpasses the performance of existing schemes.

As shown in Fig. 2, we consider a relay-based HetNet in which a macro BS aims to disseminate a common data file to a group of N users via a half-duplex RS. The set of user indices is denoted as \(I:=\{1,\ldots ,N\}\). The data file contains K original (raw) packets, each of which can be represented as a vector of length v over a finite coding field \({\mathbb {F}}_d\) where d is the size of the coding field. Hence, each packet has a length of \(v\lceil \log _2 d\rceil\) bits. To minimize the file dissemination delay, rateless code is employed at the BS. That is, an infinite number of coded packets are generated and backlogged at the BS buffer initially. The buffers of the RS and users are initially empty. We consider a time-slotted system, where the half-duplex RS can either fetch a coded packet from the BS or broadcast a previously received coded packet to the users at each time slot. Once a packet is broadcast by the RS, it will be removed from the RS buffer. For the sake of simplicity, we assume that once a user receives any K coded packets, it can recover (decode) the original packets, and the coding overhead is negligible [26].

We assume that the RS broadcasts packets utilizing a single frequency-time resource block (RB) and a predetermined modulation and coding scheme (MCS). Let the random process \(C_i(t)\in {\mathbb {Z}}_0\) denote the number of bits that can be successfully transmitted from the RS-to-user i on the transmission RB at time slot t. We assume that users’ channels undergo fast fading, i.e., \(C_i(\cdot )\) varies independently over time slots for all \(i \in I\). If the RS chooses to broadcast a packet at time slot t, two possibilities can occur for the user i: it either successfully receives the packet, i.e., \(C_i(t)\) is greater than the packet length \(v\lceil \log _2 d\rceil\), or it fails to receive the packet. Thus, we define users’ channel states as binary random processeswhere \({\textbf {1}} (\cdot )\) is the indicator function. Thus, the channels are inherent ‘ON-OFF’ channels. In the sequel, we say that the RS-to-user i channel is in ON (resp. OFF) state if user i can successfully receive a packet at time slot t, i.e., \(X_i(t)=1\) (resp. \(X_i(t)=0\)). Similarly, we let \(C_0(\cdot )\) denote the number of bits that can be successfully transmitted from the BS to the RS at time slot t, and define the BS-to-RS channel’s state as a binary random processLet \(X(t):=(X_0(t),X_1(t),\ldots ,X_N(t))\) denote the aggregate channel state information (CSI) at time slot t. We assume that CSIT such that X(t) is known at the RS at the start of each time slot t. The probability that the RS-to-user i channel is in ON state at time slot t is denoted by \(p_i:= {\mathbb {P}}(X_i(t) = 1)\), and the probability that the BS-to-RS channel is in ON state at time slot t is denoted by \(p_0:= {\mathbb {P}}(X_0(t) = 1)\). Let \(p:=(p_0,p_1,\ldots ,p_N)\) denote the aggregate channel parameters of the system.

Let \(B_i(t)\) (resp. \(B_0(t)\)) denote the number of coded packets backlogged in the buffer of user i (resp. the RS) at time slot t. Define \(B(t):=(B_0(t),B_1(t),\ldots ,B_N(t))\) as the aggregate buffer state information (BSI) of the system at time slot t. B(t) can be easily tracked over time by the RS based on the CSI and scheduling decisions made in previous time slots.

The goal is to minimize the dissemination delay, which is defined as the number of time slots it takes for all users to receive at least K coded packets, through making optimal scheduling decisions (fetching a packet from the BS or broadcasting a packet to the users) at each time slot. Table 1 provides a list of the major notations used in this work.

In this section, we first model our problem as a Markov decision process (MDP), and then propose an intelligent DRL-based scheduling policy.

Our problem is a complex decision-making problem in stochastic networks, where the aggregate channel parameters \(p\) may not be known to the RS. Generally, when the system model is unknown, there are two main approaches to solving decision-making problems: model-based and model-free methods [27]. In the model-based method, the decision maker first estimates the system model according to the observations and then applies a dynamic programming or heuristic policy based on the estimated system model. By contrast, in the model-free method, the decision maker learns a policy directly through interactions with the system without estimating the system model. The model-based method may not be applicable due to incorrect estimations of the system model caused by the limited observation ability of the decision maker. Moreover, even if the system model can be accurately estimated, solving a decision-making problem with a large model size is typically computationally intractable. Thus, in this paper, we will adopt the model-free method to solve our problem.

In this section, we will evaluate the performance of our IDS by simulation. We will compare the performance of our algorithms to the following baseline schemes. In our simulation, we also include a lower bound of the dissemination delay proposed in [11], which is obtained by numerically solving the fluid approximation [33] of our problem.

We use the open-source framework PyTorch [34] for the implementation of the IDS algorithm and use MATLAB for performance comparison of all the algorithms. The simulation runs on a PC platform equipped with Windows 10 operating systems, 16G RAM, Intel i5 4.10GHz CPU and Nvidia GTX 1080 Ti GPU. The Dueling-DDQN is composed of an input layer with \(2N+2\) ports, corresponding to \(2N+2\) elements in s(t), two fully connected hidden layers, then connected to the state value function network and advantage function network with 50 neurons, respectively, and an output layer of two ports, which correspond to two candidate actions available to the RS. One hundred neurons with the ReLU activation function are present in each hidden layer. Thus, the number of multiplications is \((2N +2)\times 100 + 100\times 100+100\times 50\times 2\).

We use an \(\epsilon\)-greedy algorithm with \(\epsilon _0=0.1, \epsilon _{\rm min}=0.001\) and \(\alpha _{\epsilon }=2000\). The parameters of the proposed algorithms are set as \(Z=30, T= 128, M = 10\), reward discount rate \(\gamma =0.99\) and experience replay memory size is \(10^4\). The Adam optimizer with a learning rate \(10^{-4}\) is used for weights updating. The size of the block is set to \(K=100\). All results are averaged over \(10^{4}\) random runs. A summary of the major parameters used in our simulation is shown in Table 2.

Figure 4 presents the dissemination delay of the trained policy using IDS and DQN as a function of the number of iterations when \(N = 2\) and \(p = (1, 0.5, 0.9)\). The dissemination delay decreases as the number of training iterations increases for both methods, and IDS has a faster convergence speed compared with DQN.

Figure 5 shows the average dissemination delay performance for symmetric-user systems with \(N=2\), \(p_0=1\) , and \(p_1=p_2= q\), where the value of q is varied. The figure shows that, as the channel conditions improve, i.e., as q increases, the dissemination delay of each scheme decreases. IDS performs the best, incurring a dissemination delay that is only 2–5% higher than the lower bound, and it reduces the dissemination delay by up to 7%, 23%, and 36% compared with WBP-TRACK, ARQ, and GREEDY, respectively.

The dissemination delay performance for asymmetric-user systems with varying N is presented in Fig. 6. For each N, we set the aggregate channel parameters as \(p_i=0.5+0.4(i-1)/(N-i)\) for all \(i \in I\). The performance of GREEDY is almost constant in this scenario, since it mainly depends on the channel parameter of the “worst” user, i.e., user 1 with \(p_1=0.3\). The performance of ARQ decays rapidly with N, and one can expect that GREEDY would be outperformed by ARQ as N increases. IDS performs closest to the lower bound and reduces the dissemination delay by 4% compared to WBP-TRACK.

The previous simulations focus on systems with a perfect BS-to-RS channel. Next, we will compare the dissemination delay performance for systems with an imperfect BS-to-RS channel, and WBP-TRACK will not be included in the following simulations.

The average dissemination delay performance for symmetric-user systems with \(N=2\), \(p_0=0.5\) , and \(p_1=p_2= q\) with varying q is displayed in Fig. 7. IDS significantly reduces dissemination delay, where the gains are up to 22% and 40% compared with ARQ and GREEDY, respectively. Figure 8 considers the case that \(N=2, p_i = 0.7\) for all \(i\in I\) and varying \(p_0\). The performance of all schemes improves as the quality of the BS-to-RS channel increases. GREEDY incurs 23–32% higher dissemination delay than IDS, and ARQ achieves 16–22% higher delay than IDS.

With \(p_0 = 0.7\) and \(p_i=0.5+0.4(i-1)/(N-1)\) for all \(i \in I\), Fig. 9 shows how the dissemination delay changes varies with N for asymmetric-user systems. The dissemination delay incurred by IDS is within 10% of the lower bound. The dissemination delays for GREEDY and ARQ are up to 24% and 40% higher than that of IDS, respectively.

Next, we evaluate the performance of the adaptive IDS in Fig. 10, we set \(N=2, L=20\) and \(\xi = 0.1\). The aggregate channel parameters are set as \(p = (1,0.7,0.7)\) initially, such that the dissemination delay of IDS is around 230 for each data block. At the beginning of the transmission of the 200th data block, the aggregate channel parameters are reduced to \(p = (1, 0.5, 0.5)\). We observe that the dissemination delay of the proposed algorithm quickly increases, since the previous learned policy is no longer efficient in the changed network environment. However, our adaptive IDS is able to detect the network dynamics by measuring the average dissemination delay of the previous L data blocks, i.e., (15) in our paper, and comparing it with the average dissemination delay of the learned algorithm. If an abnormal change is detected, it will perform a new round of training based on the previously learned parameters to update the scheduling policy. Thus, in Fig. 10, we can observe that the dissemination delay of our algorithm reduces from 450 to 300 gradually from the 200th data block to 220th data block. That is, our algorithm automatically adjusts with changes in the environment.

In this paper, we examined the minimum-delay file dissemination problem using rateless code in two-hop relay-based HetNets. We constructed the MDP formulation of the scheduling problem at the half-duplex RS. Due to the large state of the MDP system, we proposed an intelligent DRL-based scheduling algorithm. We also extended the proposed algorithm to adapt to dynamic network conditions. Simulation results demonstrated that the proposed algorithm outperforms other transmission schemes and performs nearly as well as a lower bound of the dissemination delay.