Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Neural Computing and Applications
Published in: Neural Computing and Applications 12/2021

16-11-2020 | Original Article

Online reliability optimization for URLLC in HetNets: a DQN approach

Authors: Leyou Yang, Jie Jia, Jian Chen, Xingwei Wang

Published in: Neural Computing and Applications | Issue 12/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Heterogeneous cellular networks (HetNets) have been proven as a promising approach to deal with ever-growing data traffic. Supporting ultra-reliable and low-latency communication (URLLC) is also considered as a new feature of the upcoming wireless networks. Due to the overlapping structure and the mutual interference between cells in HetNets, existing resource allocation approaches cannot be directly applied for real-time applications, especially for URLLC services. As a novel unsupervised algorithm, Deep Q Network (DQN) has already been applied to many online complex optimization models successfully. However, it may perform badly for resource allocation optimization in HetNets, due to the tiny state change and the large-scale action space characteristics. In order to cope with them, we first propose an auto-encoder to disturb the similarity of adjacent states to enhance the features and then divide the whole decision process into two phases. DQN is applied to solve each phase, respectively, and we iterate the whole process to find the joint optimized solution. We implement our algorithm in 6 scenarios with different numbers of user equipment (UE), redundant links, and sub-carriers. Simulations results demonstrate that our algorithm has good convergence for the optimization objective. Moreover, by further optimizing the power allocation, a 1–2 nines of reliability improvement is obtained for bad conditions. Finally, the experiment result shows that our algorithm reaches the reliability of 8-nines in common scenarios. As an online method, the algorithm proposed in this paper takes only 0.32 s on average.
previous article Automated organization of interaction between modules of information systems based on neural network data channels
next article Deep learning feature selection to unhide demographic recommender systems factors

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now
Literature
1.
Bacci G, Belmega EV, Mertikopoulos P, Sanguinetti L (2014) Energy-aware competitive power allocation for heterogeneous networks under QOS constraints. IEEE Trans Wirel Commun 14(9):4728–4742CrossRef
2.
Baldi P (2011) Autoencoders, unsupervised learning and deep architectures. In: International conference on unsupervised and transfer learning workshop
3.
Bennis M, Debbah M, Poor HV (2018) Ultra-reliable and low-latency wireless communication: tail, risk and scale. Proc IEEE 106(10):1834–1853CrossRef
4.
Cao J, Peng T, Qi Z, Duan R, Yuan Y, Wang W (2018) Interference management in ultradense networks: a user-centric coalition formation game approach. IEEE Trans Veh Technol 67(6):5188–5202CrossRef
5.
6.
Chen J, Zhang L, Liang Y, Ma S (2020) Optimal resource allocation for multicarrier NOMA in short packet communications. IEEE Trans Veh Technol 69(2):2141–2156CrossRef
7.
Cui J, Liu Y, Nallanathan A (2020) Multi-agent reinforcement learning-based resource allocation for UAV networks. IEEE Trans Wirel Commun 19(2):729–743CrossRef
8.
D’Oro S, Marotta MA, Both CB, DaSilva L, Palazzo S (2019) Power-efficient resource allocation in c-RANS with SINR constraints and deadlines. IEEE Trans Veh Technol 68(6):6099–6113CrossRef
9.
Fadlullah Z, Fouda MM, Kato N, Takeuchi A, Iwasaki N, Nozaki Y (2012) Toward intelligent machine-to-machine communications in smart grid. IEEE Commun Soc 49(4):60–65CrossRef
10.
Forecast CV (2019) Cisco visual networking index: global mobile data traffic forecast update, 2017–2022 white paper
11.
Frotzscher A, Wetzker U, Bauer M, Rentschler M, Beyer M, Elspass S, Klessig H (2014) Requirements and current solutions of wireless communication in industrial automation. In: IEEE international conference on communications workshops
12.
Ghosh A, Mangalvedhe N, Ratasuk R, Mondal B, Cudak M, Visotsky E, Thomas TA, Andrews JG, Xia P, Jo HS et al (2012) Heterogeneous cellular networks: from theory to practice. IEEE Commun Magn 50(6):54–64CrossRef
13.
Guo C, Liang L, Li GY (2019) Resource allocation for high-reliability low-latency vehicular communications with packet retransmission. IEEE Trans Veh Technol 68(7):6219–6230CrossRef
14.
Guo S, Zhou X (2019) Robust resource allocation with imperfect channel estimation in NOMA-based heterogeneous vehicular networks. IEEE Trans Commun 67(3):2321–2332CrossRef
15.
Hasselt HV, Guez A, Silver D (2015) Deep reinforcement learning with double q-learning. In: Computer science
16.
He C, Abbas R, Peng C, Shirvanimoghaddam M, Vucetic B (2017) Ultra-reliable low latency cellular networks: use cases, challenges and approaches. IEEE Commun Mag
17.
Hessel M, Modayil J, Van Hasselt H, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2018) Rainbow: combining improvements in deep reinforcement learning. In: AAAI conference on artificial intelligence
18.
Jaderberg M, Czarnecki WM, Dunning I, Marris L, Graepel T (2019) Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364(6443):859–865MathSciNetCrossRef
19.
Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Achieving high availability in heterogeneous cellular networks via spectrum aggregation. IEEE Trans Veh Technol 66(11):10156–10169CrossRef
20.
Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Availability analysis and optimization in comp and ca-enabled hetnets. IEEE Trans Commun 65(6):2438–2450CrossRef
21.
Jie J, Deng Y, Ping S, Aghvami H, Nallanathan A (2016) High availability optimization in heterogeneous cellular networks. In: GLOBECOM 2016 IEEE global communications conference
22.
Kanervisto A, Scheller C, Hautamäki V (2020) Action space shaping in deep reinforcement learning
23.
Lee H, Vahid S, Moessner K (2014) A survey of radio resource management for spectrum aggregation in LTE-advanced. IEEE Commun Surv Tutor 16(2):745–760CrossRef
24.
Liao X, Shi J, Li Z, Zhang L, Xia B (2020) A model-driven deep reinforcement learning heuristic algorithm for resource allocation in ultra-dense cellular networks. IEEE Trans Veh Technol 69(1):983–997CrossRef
25.
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529CrossRef
26.
Nguyen HT, Murakami H, Nguyen K, Ishizu K, Hwang WJ (2019) Joint user association and power allocation for millimeter-wave ultra-dense networks. Mob Netw Appl (2)
27.
Nielsen JJ, Liu R, Popovski P (2018) Ultra-reliable low latency communication using interface diversity. IEEE Trans Commun 66(3):1322–1334CrossRef
28.
Pukite P, Pukite J (1998) Markov modeling for reliability analysis. Wiley, New YorkCrossRef
29.
Qin Z, Yue X, Liu Y, Ding Z, Nallanathan A (2018) User association and resource allocation in unified NOMA enabled heterogeneous ultra dense networks. IEEE Commun Mag 56(6):86–92CrossRef
30.
Raman RK, Jagannathan K (2018) Downlink resource allocation under time-varying interference: fairness and throughput optimality. IEEE Trans Wirel Commun 17(2):722–735CrossRef
31.
Schaul T, Quan J, Antonoglou I, Silver D (2015) Prioritized experience replay. Comput Sci
32.
Shirvanimoghaddam M, Mohamadi MS, Abbas R, Minja A, Yue C, Matuz B, Han G, Lin Z, Li Y, Johnson S (2018) Short block-length codes for ultra-reliable low-latency communications. In: IEEE communications magazine, pp 1–8
33.
Silver D, Graves A, Antonoglou I, Riedmiller M, Mnih V, Wierstra D, Kavukcuoglu K (2013) Playing atari with deep reinforcement learning. arXiv: Learning
34.
Skarin P, Tärneberg W, Årzen K, Kihl M (2018) Towards mission-critical control at the edge and over 5g. In: 2018 IEEE international conference on edge computing (EDGE), pp 50–57
35.
Sklar B (1997) Rayleigh fading channels in mobile digital communication systems part I: characterization. IEEE Commun Mag 35(9):136–146CrossRef
36.
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. IEEE Trans Neural Netw 9(5):1054–1054CrossRef
37.
38.
Wang Z, Schaul T, Hessel M, Van Hasselt H, Lanctot M, De Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on international conference on machine learning, vol 48, ICML’16, JMLR.org, pp 1995–2003
39.
Whitehead SD (1991) A complexity analysis of cooperative mechanisms in reinforcement learning. In: National conference on artificial intelligence
40.
Wu X, Ma Z, Chen X, Labeau F, Han S (2019) Energy efficiency-aware joint resource allocation and power allocation in multi-user beamforming. IEEE Trans Veh Technol 68(5):4824–4833CrossRef
41.
Yousefvand M, Hamidouche K, Mandayam NB (2019) Learning-based resource optimization in ultra reliable low latency hetnets. In: GLOBECOM 2019 IEEE global communications conference
42.
Zhang H, Venturino L, Prasad N, Li P, Rangarajan S, Wang X (2011) Weighted sum-rate maximization in multi-cell networks via coordinated scheduling and discrete power control. IEEE J Sel Areas Commun 29(6):1214–1224CrossRef
43.
44.
Zhang N, Cheng N, Gamage AT, Zhang K, Mark JW, Shen X (2015) Cloud assisted hetnets toward 5g wireless networks. IEEE Commun Mag 53(6):59–65CrossRef
45.
46.
Zhou ZH, Yu Y, Qian C.: Evolutionary learning: advances in theories and algorithms
Metadata
Title
Online reliability optimization for URLLC in HetNets: a DQN approach
Authors
Leyou Yang
Jie Jia
Jian Chen
Xingwei Wang
Publication date
16-11-2020
Publisher
Springer London
Published in
Neural Computing and Applications / Issue 12/2021
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI
https://doi.org/10.1007/s00521-020-05492-4

Other articles of this Issue 12/2021

Neural Computing and Applications 12/2021 Go to the issue

Original Article

Molten steel temperature prediction using a hybrid model based on information interaction-enhanced cuckoo search

Original Article

Window of multistability and its control in a simple 3D Hopfield neural network: application to biomedical image encryption

Original Article

Optimal power flow solution with stochastic wind power using the Lévy coyote optimization algorithm

Original Article

Design of sparse Bayesian echo state network for time series prediction

Original Article

Periodically intermittent control for finite-time synchronization of delayed quaternion-valued neural networks

Original Article

CNNFOIL: convolutional encoder decoder modeling for pressure fields around airfoils

Premium Partner

    Image Credits