Top

Wireless Personal Communications

Published in:

17-06-2024 | Research

Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks

Authors: Aruna Valasa, Anjaneyulu Lokam, Chayan Bhar

Published in: Wireless Personal Communications | Issue 1/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Heterogeneous networks (HetNets) are essential in contemporary wireless communication networks as they help operators address challenges related to coverage, capacity, and quality. HetNets also enable the efficient use of resources and preparedness for future wireless technologies. The efficient allocation of limited resources to user equipments (UEs) is a critical problem that can be tackled using joint resource allocation and user association (JRAUA) techniques. However, this problem is challenging due to its non-convex and combinatorial nature, which is further complicated by the hybrid action space that involves both continuous actions and discrete actions. Consequently, determining the most efficient approach for JRAUA is a challenging endeavor. The objective of this study is to address the above challenge and improve the energy efficiency of networks by examining the JRAUA problem in downlink HetNets using orthogonal frequency division multiple access (OFDMA). This work presents a new strategy, called the Variable Hybrid Action Space—Deep Q-Network (VHAS-DQN), to optimize the learning policy and improve the average cumulative reward by incorporating constraints like wireless backhaul capacity of each UE and quality-of-service (QoS) into the learning process. To demonstrate the advantages of the JRAUA model in terms of energy efficiency while guaranteeing that the backhaul capacity constraints and quality of service (QoS) criteria are met, the simulation results from the VHAS-DQN were compared with other experimental case studies.

previous article Design and Analysis of Four-Port MIMO System Optimization Methodology with Machine Learning Approaches of Validated Antenna Parameters

next article Trade-Off in Energy Efficiency and Response Performance in Green CRNs: Perspective of Two Sleep Timers and Bursty Traffic

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Saad, W., Bennis, M., & Chen, M. (2020). A vision of 6G wireless systems: applications, trends, technologies, and open research problems. IEEE Network, 34(3), 134–142. https://doi.org/10.1109/MNET.001.1900287CrossRef

Dryjanski, M., & Kliks, A. (2020). A hierarchical and modular radio resource management architecture for 5G and beyond. IEEE Communications Magazine, 58(7), 28–34. https://doi.org/10.1109/MCOM.001.1900796CrossRef

Wu, Q., Li, G. Y., Chen, W., Ng, D. W. K., & Schober, R. (2017). An overview of sustainable green 5G networks. IEEE Wireless Communications, 24(4), 72–80. https://doi.org/10.1109/MWC.2017.1600343CrossRef

Mili, M. R., Hamdi, K. A., Marvasti, F., & Bennis, M. (2016). Joint optimization for optimal power allocation in OFDMA femtocell networks. IEEE Communications Letters, 20(1), 133–136. https://doi.org/10.1109/LCOMM.2015.2497697CrossRef

Amiri, R., Almasi, M. A., Andrews, J. G., & Mehrpouyan, H. (2019). Reinforcement learning for self-organization and power control of two-tier heterogeneous networks. IEEE Transactions on Wireless Communications, 18(8), 3933–3947. https://doi.org/10.1109/TWC.2019.2919611CrossRef

Teng, Y., Liu, M., Yu, F. R., Leung, V. C. M., Song, M., & Zhang, Y. (2019). Resource allocation for ultra-dense networks: A survey, some research issues and challenges. IEEE Communications Surveys and Tutorials, 21(3), 2134–2168. https://doi.org/10.1109/COMST.2018.2867268CrossRef

Zhou, T., Liu, Z., Zhao, J., Li, C., & Yang, L. (2018). Joint user association and power control for load balancing in downlink heterogeneous cellular networks. IEEE Transactions on Vehicular Technology, 67(3), 2582–2593. https://doi.org/10.1109/TVT.2017.2768574CrossRef

Wen, Z., Zhu, G., Ni, M., & Lin, S. (2017). User association-based interference management in ultra-dense networks. In 2017 IEEE international symposium on antennas and propagation & USNC/URSI national radio science meeting (pp. 1903–1904). IEEE. https://doi.org/10.1109/APUSNCURSINRSM.2017.8072994

Deng, D., Li, X., Zhao, M., Rabie, K. M., & Kharel, R. (2020). Deep learning-based secure mimo communications with imperfect CSI for heterogeneous networks. Sensors (Switzerland), 20(6), 1–15. https://doi.org/10.3390/s20061730CrossRef

10.

Liu, S., He, J., & Wu, J. (2021). Dynamic cooperative spectrum sensing based on deep multi-user reinforcement learning. Applied Sciences (Switzerland), 11(4), 1–16. https://doi.org/10.3390/app11041884CrossRef

11.

Munaye, Y. Y., Juang, R. T., Lin, H. P., Tarekegn, G. B., & Lin, D. B. (2021). Deep reinforcement learning based resource management in UAV-assisted iot networks. Applied Sciences (Switzerland), 11(5), 1–20. https://doi.org/10.3390/app11052163CrossRef

12.

Meng, F., Chen, P., Wu, L., & Cheng, J. (2020). Power allocation in multi-user cellular networks: Deep reinforcement learning approaches. IEEE Transactions on Wireless Communications, 19(10), 6255–6267. https://doi.org/10.1109/TWC.2020.3001736CrossRef

13.

Ma, H., Zhang, H., Wang, X., & Cheng, J. (2017). Backhaul-aware user association and resource allocation for massive MIMO-enabled HetNets. IEEE Communications Letters, 21(12), 2710–2713. https://doi.org/10.1109/LCOMM.2017.2755021CrossRef

14.

Hasan, M. K., Shahjalal, M., Islam, M. M., Alam, M. M., Ahmed, M. F., & Jang, Y. M. (2020). The role of deep learning in NOMA for 5G and beyond communications. 2020 international conference on artificial intelligence in information and communication, ICAIIC 2020 (pp. 303–307). https://doi.org/10.1109/ICAIIC48513.2020.9065219

15.

Nasir, Y. S., & Guo, D. (2019). Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks. IEEE Journal on Selected Areas in Communications, 37(10), 2239–2250. https://doi.org/10.1109/JSAC.2019.2933973CrossRef

16.

He, C., Hu, Y., Chen, Y., & Zeng, B. (2019). Joint power allocation and channel assignment for NOMA with deep reinforcement learning. IEEE Journal on Selected Areas in Communications, 37(10), 2200–2210. https://doi.org/10.1109/JSAC.2019.2933762CrossRef

17.

Chen, Y., Li, J., Chen, W., Lin, Z., & Vucetic, B. (2016). Joint user association and resource allocation in the downlink of heterogeneous networks. IEEE Transactions on Vehicular Technology, 65(7), 5701–5706. https://doi.org/10.1109/TVT.2015.2452953CrossRef

18.

Han, Q., Yang, B., Miao, G., Chen, C., Wang, X., & Guan, X. (2017). Backhaul-aware user association and resource allocation for energy-constrained HetNets. IEEE Transactions on Vehicular Technology, 66(1), 580–593. https://doi.org/10.1109/TVT.2016.2533559CrossRef

19.

Amiri, R., Mehrpouyan, H., Fridman, L., Mallik, R. K., Nallanathan, A., & Matolak, D. (2018). A machine learning approach for power allocation in HetNets considering QoS. In IEEE international conference on communications (Vol. 2018-May, pp. 1–7). IEEE. https://doi.org/10.1109/ICC.2018.8422864

20.

Ahmed, K. I., & Hossain, E. (2019). A deep Q-learning method for downlink power allocation in multi-cell networks. Retrieved from http://arxiv.org/abs/1904.13032

21.

Xu, Z., Wang, Y., Tang, J., Wang, J., & Gursoy, M. C. (2017). A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs. In 2017 IEEE International Conference on Communications (ICC) (pp. 1–6). IEEE. https://doi.org/10.1109/ICC.2017.7997286

22.

Wei, Y., Yu, F. R., Song, M., & Han, Z. (2018). User scheduling and resource allocation in HetNets with hybrid energy supply: An actor-critic reinforcement learning approach. IEEE Transactions on Wireless Communications, 17(1), 680–692. https://doi.org/10.1109/TWC.2017.2769644CrossRef

23.

Li, D., Zhang, H., Long, K., Huangfu, W., Dong, J., & Nallanathan, A. (2019). User association and power allocation based on Q-learning in ultra dense heterogeneous networks. In 2019 IEEE global communications conference, GLOBECOM 2019—proceedings. https://doi.org/10.1109/GLOBECOM38437.2019.9013455

24.

Zhang, Y., Kang, C., Ma, T., Teng, Y., & Guo, D. (2018). Power allocation in multi-cell networks using deep reinforcement learning. In IEEE vehicular technology conference, 2018-Augus (pp. 1–6). https://doi.org/10.1109/VTCFall.2018.8690757

25.

Huang, L., Bi, S., & Zhang, Y. J. A. (2020). Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Transactions on Mobile Computing, 19(11), 2581–2593. https://doi.org/10.1109/TMC.2019.2928811CrossRef

26.

Yau, K. L. A., Komisarczuk, P., & Teal, P. D. (2012). Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues. Journal of Network and Computer Applications, 35(1), 253–267. https://doi.org/10.1016/j.jnca.2011.08.007CrossRef

27.

Asheralieva, A., & Miyanaga, Y. (2016). An autonomous learning-based algorithm for joint channel and power level selection by D2D pairs in heterogeneous cellular networks. IEEE Transactions on Communications, 64(9), 3996–4012. https://doi.org/10.1109/TCOMM.2016.2593468CrossRef

28.

Ghadimi, E., Davide Calabrese, F., Peters, G., & Soldati, P. (2017). A reinforcement learning approach to power control and rate adaptation in cellular networks. IEEE International Conference on Communications. https://doi.org/10.1109/ICC.2017.7997440CrossRef

29.

Zhang, T., Zhu, K., & Wang, J. (2021). Energy-efficient mode selection and resource allocation for D2D-enabled heterogeneous networks: A deep reinforcement learning approach. IEEE Transactions on Wireless Communications, 20(2), 1175–1187. https://doi.org/10.1109/TWC.2020.3031436CrossRef

30.

Li, Z., & Guo, C. (2020). Multi-agent deep reinforcement learning based spectrum allocation for D2D underlay communications. IEEE Transactions on Vehicular Technology, 69(2), 1828–1840. https://doi.org/10.1109/TVT.2019.2961405CrossRef

31.

Sun, M., Jin, Y., Wang, S., & Mei, E. (2022). Joint deep reinforcement learning and unsupervised learning for channel selection and power control in D2D networks. Entropy, 24(12), 1363–1378. https://doi.org/10.3390/e24121722MathSciNetCrossRef

32.

Park, H., & Lim, Y. (2020). Reinforcement learning for energy optimization with 5G communications in vehicular social networks. Sensors, 20(8), 2361. https://doi.org/10.3390/s20082361CrossRef

33.

Xu, Z., Tang, J., Yin, C., Wang, Y., Xue, G., Wang, J., & Gursoy, M. C. (2022). ReCARL: Resource allocation in cloud RANs with deep reinforcement learning. IEEE Transactions on Mobile Computing, 21(7), 2533–2545. https://doi.org/10.1109/TMC.2020.3044282CrossRef

34.

Lu, Y., Lu, H., Cao, L., Wu, F., & Zhu, D. (2018). Learning deterministic policy with target for power control in wireless networks. In 2018 IEEE global communications conference, GLOBECOM 2018—Proceedings, (pp. 1–7). https://doi.org/10.1109/GLOCOM.2018.8648056

35.

Khalili, A., Akhlaghi, S., Tabassum, H., & Ng, D. W. K. (2020). Joint user association and resource allocation in the uplink of heterogeneous networks. IEEE Wireless Communications Letters, 9(6), 804–808. https://doi.org/10.1109/LWC.2020.2970696CrossRef

36.

Jabeen, S., & Ho, P. H. (2019). A benchmark for joint channel allocation and user scheduling in flexible heterogeneous networks. IEEE Transactions on Vehicular Technology, 68(9), 9233–9244. https://doi.org/10.1109/TVT.2019.2930884CrossRef

37.

Zhang, H., Huang, S., Jiang, C., Long, K., Leung, V. C. M., & Poor, H. V. (2017). Energy efficient user association and power allocation in millimeter-wave-based ultra dense networks with energy harvesting base stations. IEEE Journal on Selected Areas in Communications, 35(9), 1936–1947. https://doi.org/10.1109/JSAC.2017.2720898CrossRef

38.

Ye, Q., Rong, B., Chen, Y., Al-Shalash, M., Caramanis, C., & Andrews, J. G. (2013). User association for load balancing in heterogeneous cellular networks. IEEE Transactions on Wireless Communications, 12(6), 2706–2716. https://doi.org/10.1109/TWC.2013.040413.120676CrossRef

39.

Kim, T., & Chang, J. M. (2018). QoS-aware energy-efficient association and resource scheduling for HetNets. IEEE Transactions on Vehicular Technology, 67(1), 650–664. https://doi.org/10.1109/TVT.2017.2737629CrossRef

40.

Ye, Q., Zhuang, W., Zhang, S., Jin, A. L., Shen, X., & Li, X. (2018). Dynamic radio resource slicing for a two-tier heterogeneous wireless network. IEEE Transactions on Vehicular Technology, 67(10), 9896–9910. https://doi.org/10.1109/TVT.2018.2859740CrossRef

41.

Luo, X. (2017). Delay-oriented QoS-aware user association and resource allocation in heterogeneous cellular networks. IEEE Transactions on Wireless Communications, 16(3), 1809–1822. https://doi.org/10.1109/TWC.2017.2654458CrossRef

42.

Asheralieva, A., & Miyanaga, Y. (2017). Optimal contract design for joint user association and intercell interference mitigation in heterogeneous LTE-A Networks with asymmetric information. IEEE Transactions on Vehicular Technology, 66(6), 5284–5300. https://doi.org/10.1109/TVT.2016.2615849CrossRef

43.

Liu, J., Tao, X., & Lu, J. (2019). Mobility-aware centralized reinforcement learning for dynamic resource allocation in HetNets. In 2019 IEEE Global Communications Conference, GLOBECOM 2019—Proceedings (pp. 1–6). https://doi.org/10.1109/GLOBECOM38437.2019.9013191

44.

De Domenico, A., &Ktenas, D. (2018). Reinforcement learning for interference-aware cell DTX in heterogeneous networks. In IEEE Wireless Communications and Networking Conference, WCNC (pp. 1–6). https://doi.org/10.1109/WCNC.2018.8376993

45.

Yang, H., Alphones, A., Zhong, W. D., Chen, C., & Xie, X. (2020). Learning-Based energy-efficient resource management by heterogeneous RF/VLC for ultra-reliable low-latency industrial IoT networks. IEEE Transactions on Industrial Informatics, 16(8), 5565–5576. https://doi.org/10.1109/TII.2019.2933867CrossRef

46.

Hsieh, C. K., Chan, K. L., & Chien, F. T. (2021). Energy-efficient power allocation and user association in heterogeneous networks with deep reinforcement learning. Applied Sciences (Switzerland), 11(9), 4135. https://doi.org/10.3390/app11094135CrossRef

47.

Mohajer, A., Daliri, M. S., Mirzaei, A., Ziaeddini, A., Nabipour, M., & Bavaghar, M. (2022). Heterogeneous computational resource allocation for NOMA: Toward green mobile edge-computing systems. IEEE Transactions on Services Computing, 16(2), 1225–1238.CrossRef

48.

Dong, S., Zhan, J., Hu, W., Mohajer, A., Bavaghar, M., & Mirzaei, A. (2023). Energy-efficient hierarchical resource allocation in uplink-downlink decoupled NOMA HetNets. IEEE Transactions on Network and Service Management, 20(3), 3380–3395.

49.

Mohajer, A., Sorouri, F., Mirzaei, A., Ziaeddini, A., Rad, K. J., & Bavaghar, M. (2022). Energy-aware hierarchical resource management and backhaul traffic optimization in heterogeneous cellular networks. IEEE Systems Journal, 16(4), 5188–5199.CrossRef

50.

Hausknecht, M., Stone, P., & Mc, O. P. (2016, July). On-policy vs. off-policy updates for deep reinforcement learning. In Deep reinforcement learning: Frontiers and challenges, IJCAI 2016 Workshop. New York, NY, USA: AAAI Press.

51.

Karandikar, R. L., & Vidyasagar, M. (2021). Convergence of batch asynchronous stochastic approximation with applications to reinforcement learning, 1, 1–28. Retrieved from http://arxiv.org/abs/2109.03445

52.

Zhang, S., & Sutton, R. S. (2017). A deeper look at experience replay. Retrieved from http://arxiv.org/abs/1712.01275

53.

Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized experience replay. In 4th International conference on learning representations, ICLR 2016—conference track proceedings (pp. 1–21).

Title: Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks
Authors: Aruna Valasa
Anjaneyulu Lokam
Chayan Bhar
Publication date: 17-06-2024
Publisher: Springer US
Published in: Wireless Personal Communications / Issue 1/2024
Print ISSN: 0929-6212
Electronic ISSN: 1572-834X
DOI: https://doi.org/10.1007/s11277-024-11255-4

On the Effective Throughput of Shadowed Beaulieu-Xie Fading Channel

Research

Speaker Identification Using MFCC Feature Extraction ANN Classification Technique

A New Secure Data Communication Method Using Wavelet Transform

Trade-Off in Energy Efficiency and Response Performance in Green CRNs: Perspective of Two Sleep Timers and Bursty Traffic

Research

Springer Professional

Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks

Abstract

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2024

On the Effective Throughput of Shadowed Beaulieu-Xie Fading Channel

Speaker Identification Using MFCC Feature Extraction ANN Classification Technique

A New Secure Data Communication Method Using Wavelet Transform

Trade-Off in Energy Efficiency and Response Performance in Green CRNs: Perspective of Two Sleep Timers and Bursty Traffic

A Review of Distributed Generation Optimization with Shunt Capacitors in Reconfigured Distribution Networks

A Comparative Study of Deep Leaning-Aided Relay Selection Protocols in Cooperative Communication

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

On the Effective Throughput of Shadowed Beaulieu-Xie Fading Channel

Speaker Identification Using MFCC Feature Extraction ANN Classification Technique

A New Secure Data Communication Method Using Wavelet Transform

Trade-Off in Energy Efficiency and Response Performance in Green CRNs: Perspective of Two Sleep Timers and Bursty Traffic

A Review of Distributed Generation Optimization with Shunt Capacitors in Reconfigured Distribution Networks

A Comparative Study of Deep Leaning-Aided Relay Selection Protocols in Cooperative Communication