Skip to main content
Top
Published in: New Generation Computing 1/2023

19-12-2022

A Clustering Offloading Decision Method for Edge Computing Tasks Based on Deep Reinforcement Learning

Authors: Zhen Zhang, Huanzhou Li, Zhangguo Tang, Dinglin Gu, Jian Zhang

Published in: New Generation Computing | Issue 1/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In many IoT scenarios, the resources of terminal devices are limited, and it is difficult to provide services with low latency and low energy consumption. Mobile edge computing is an effective solution by offloading computing tasks to edge server processing. There are some problems in the existing offloading decision algorithms: the offloading decision method based on heuristic algorithms cannot dynamically adjust the policy in the changing environment; the offloading algorithm based on deep reinforcement learning will lead to slow convergence and poor exploration effect due to the problem of dimension explosion. To solve the above problems, this paper designs an offloading decision algorithm to make dynamic decisions in a mobile edge computing network with multi-device access. The algorithm comprehensively considers the energy consumption of terminal equipment, offloading overhead, average delay and success rate of task completion, aiming to achieve the highest total revenue of the whole system in a period of time. In this work, the online offloading problem is abstracted as a Markov decision process. Based on the Double Dueling Deep Q-Network (D3QN) algorithm, the offloading decision is designed to adapt to the highly dynamic environment of the edge computing network and solve the problem of high state space complexity. In addition, this paper innovatively introduces a clustering algorithm into deep reinforcement learning (DRL) to preprocess the action space and solve the explosion problem of the action space dimension caused by the increase of terminal devices. The experimental results show that the proposed algorithm is superior to the baseline strategies such as Deep Q-Network (DQN) algorithm in convergence speed and total reward.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference He, X.F., Jin, R.C., Dai, H.Y.: Deep PDS-learning for privacy-aware offloading in MEC-enabled IoT. IEEE Internet Things J. 6(3), 4547–4555 (2019) CrossRef He, X.F., Jin, R.C., Dai, H.Y.: Deep PDS-learning for privacy-aware offloading in MEC-enabled IoT. IEEE Internet Things J. 6(3), 4547–4555 (2019) CrossRef
2.
go back to reference Mansouri, Y., Babar, M.A.: A review of edge computing: features and resource virtualization. J. Parallel Distrib. Comput. 150, 155–183 (2021) CrossRef Mansouri, Y., Babar, M.A.: A review of edge computing: features and resource virtualization. J. Parallel Distrib. Comput. 150, 155–183 (2021) CrossRef
3.
go back to reference Chen, Y., Zhang, N., Zhang, Y.C., Chen, X.: Dynamic computation offloading in edge computing for internet of things. IEEE Internet Things J. 6(3), 4242–4251 (2019) CrossRef Chen, Y., Zhang, N., Zhang, Y.C., Chen, X.: Dynamic computation offloading in edge computing for internet of things. IEEE Internet Things J. 6(3), 4242–4251 (2019) CrossRef
4.
go back to reference Jiang, C.F., Cheng, X.L., Gao, H.H., Zhou, X., Wan, J.: Toward computation offloading in edge computing: a survey. IEEE Access 7, 131543–131558 (2019) CrossRef Jiang, C.F., Cheng, X.L., Gao, H.H., Zhou, X., Wan, J.: Toward computation offloading in edge computing: a survey. IEEE Access 7, 131543–131558 (2019) CrossRef
5.
go back to reference Islam, A., Debnath, A., Ghose, M., Chakraborty, S.: A survey on task offloading in multi-access edge computing. J. Syst. Architect. 118, 16 (2021) CrossRef Islam, A., Debnath, A., Ghose, M., Chakraborty, S.: A survey on task offloading in multi-access edge computing. J. Syst. Architect. 118, 16 (2021) CrossRef
6.
go back to reference Mao, Y.Y., You, C.S., Zhang, J., Huang, K.B., Letaief, K.B.: A survey on mobile edge computing: the communication perspective. IEEE Commun. Surv. Tutorials 19(4), 2322–2358 (2017) CrossRef Mao, Y.Y., You, C.S., Zhang, J., Huang, K.B., Letaief, K.B.: A survey on mobile edge computing: the communication perspective. IEEE Commun. Surv. Tutorials 19(4), 2322–2358 (2017) CrossRef
7.
go back to reference Zhang, J., Xia, W.W., Yan, F., Shen, L.F.: Joint computation offloading and resource allocation optimization in heterogeneous networks with mobile edge computing. IEEE Access 6, 19324–19337 (2018) CrossRef Zhang, J., Xia, W.W., Yan, F., Shen, L.F.: Joint computation offloading and resource allocation optimization in heterogeneous networks with mobile edge computing. IEEE Access 6, 19324–19337 (2018) CrossRef
8.
go back to reference Lin, H., Zeadally, S., Chen, Z.H., Labiod, H., Wang, L.S.: A survey on computation offloading modeling for edge computing. J. Netw. Comput. Appl. 169, 25 (2020) CrossRef Lin, H., Zeadally, S., Chen, Z.H., Labiod, H., Wang, L.S.: A survey on computation offloading modeling for edge computing. J. Netw. Comput. Appl. 169, 25 (2020) CrossRef
9.
go back to reference Cui, L.Z., Chen, Z.T., Yang, S., Ming, Z.X., Li, Q., Zhou, Y.P., Chen, S.P., Lu, Q.H.: A blockchain-based containerized edge computing platform for the internet of vehicles. IEEE Internet Things J. 8(4), 2395–2408 (2021) CrossRef Cui, L.Z., Chen, Z.T., Yang, S., Ming, Z.X., Li, Q., Zhou, Y.P., Chen, S.P., Lu, Q.H.: A blockchain-based containerized edge computing platform for the internet of vehicles. IEEE Internet Things J. 8(4), 2395–2408 (2021) CrossRef
10.
go back to reference Huang, D., Wang, P., Niyato, D.: A dynamic offloading algorithm for mobile computing. IEEE Trans. Wireless Commun. 11(6), 1991–1995 (2012) CrossRef Huang, D., Wang, P., Niyato, D.: A dynamic offloading algorithm for mobile computing. IEEE Trans. Wireless Commun. 11(6), 1991–1995 (2012) CrossRef
11.
go back to reference You, C.S., Huang, K.B., Chae, H., Kim, B.H.: Energy-efficient resource allocation for mobile-edge computation offloading. IEEE Trans. Wireless Commun. 16(3), 1397–1411 (2017) CrossRef You, C.S., Huang, K.B., Chae, H., Kim, B.H.: Energy-efficient resource allocation for mobile-edge computation offloading. IEEE Trans. Wireless Commun. 16(3), 1397–1411 (2017) CrossRef
12.
go back to reference Ebrahimzadeh, A., Maier, M.: Distributed cooperative computation offloading in multi-access edge computing fiber-wireless networks. Opt. Commun. 452, 130–139 (2019) CrossRef Ebrahimzadeh, A., Maier, M.: Distributed cooperative computation offloading in multi-access edge computing fiber-wireless networks. Opt. Commun. 452, 130–139 (2019) CrossRef
13.
go back to reference Ale, L., Zhang, N., Wu, H.C., Chen, D.J., Han, T.: Online proactive caching in mobile edge computing using bidirectional deep recurrent neural network. IEEE Internet Things J. 6(3), 5520–5530 (2019) CrossRef Ale, L., Zhang, N., Wu, H.C., Chen, D.J., Han, T.: Online proactive caching in mobile edge computing using bidirectional deep recurrent neural network. IEEE Internet Things J. 6(3), 5520–5530 (2019) CrossRef
14.
go back to reference Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013) CrossRef Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013) CrossRef
15.
go back to reference Chen, M., Hao, Y.X.: Task offloading for mobile edge computing in software defined ultra-dense network. IEEE J. Sel. Areas Commun. 36(3), 587–597 (2018) MathSciNetCrossRef Chen, M., Hao, Y.X.: Task offloading for mobile edge computing in software defined ultra-dense network. IEEE J. Sel. Areas Commun. 36(3), 587–597 (2018) MathSciNetCrossRef
16.
go back to reference Chen, M.H., Dong, M., Liang, B.: Resource sharing of a computing access point for multi-user mobile cloud offloading with delay constraints. IEEE Trans. Mob. Comput. 17(12), 2868–2881 (2018) CrossRef Chen, M.H., Dong, M., Liang, B.: Resource sharing of a computing access point for multi-user mobile cloud offloading with delay constraints. IEEE Trans. Mob. Comput. 17(12), 2868–2881 (2018) CrossRef
17.
go back to reference Al-Asadi, M.A., Tasdemir, S.: Empirical comparisons for combining balancing and feature selection strategies for characterizing football players using FIFA video game system. IEEE Access 9, 149266–149286 (2021) CrossRef Al-Asadi, M.A., Tasdemir, S.: Empirical comparisons for combining balancing and feature selection strategies for characterizing football players using FIFA video game system. IEEE Access 9, 149266–149286 (2021) CrossRef
18.
go back to reference Qiu, X.Y., Liu, L.B., Chen, W.H., Hong, Z.C., Zheng, Z.B.: Online deep reinforcement learning for computation offloading in blockchain-empowered mobile edge computing. IEEE Trans. Veh. Technol. 68(8), 8050–8062 (2019) CrossRef Qiu, X.Y., Liu, L.B., Chen, W.H., Hong, Z.C., Zheng, Z.B.: Online deep reinforcement learning for computation offloading in blockchain-empowered mobile edge computing. IEEE Trans. Veh. Technol. 68(8), 8050–8062 (2019) CrossRef
19.
go back to reference Zhao, R., Wang, X.J., Xia, J.J., Fan, L.S.: Deep reinforcement learning based mobile edge computing for intelligent internet of things. Phys. Commun. 43, 7 (2020) CrossRef Zhao, R., Wang, X.J., Xia, J.J., Fan, L.S.: Deep reinforcement learning based mobile edge computing for intelligent internet of things. Phys. Commun. 43, 7 (2020) CrossRef
20.
go back to reference Wang, H.N., Liu, N., Zhang, Y.Y., Feng, D.W., Huang, F., Li, D.S., Zhang, Y.M.: Deep reinforcement learning: a survey. Front. Inf. Technol. Electr. Eng. 21(12), 1726–1744 (2020) CrossRef Wang, H.N., Liu, N., Zhang, Y.Y., Feng, D.W., Huang, F., Li, D.S., Zhang, Y.M.: Deep reinforcement learning: a survey. Front. Inf. Technol. Electr. Eng. 21(12), 1726–1744 (2020) CrossRef
21.
go back to reference Chen, Z., Wang, X.D.: Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach. EURASIP J. Wirel. Commun. Netw. 2020(1), 21 (2020) CrossRef Chen, Z., Wang, X.D.: Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach. EURASIP J. Wirel. Commun. Netw. 2020(1), 21 (2020) CrossRef
22.
go back to reference Zhang, Y., Yao, J.G., Guan, H.B.: Intelligent cloud resource management with deep reinforcement learning. IEEE Cloud Comput. 4(6), 60–69 (2017) CrossRef Zhang, Y., Yao, J.G., Guan, H.B.: Intelligent cloud resource management with deep reinforcement learning. IEEE Cloud Comput. 4(6), 60–69 (2017) CrossRef
23.
go back to reference Zhang, C., Zheng, Z.X.: Task migration for mobile edge computing using deep reinforcement learning. Future Gener. Comput. Syst. Int. J. Esci. 96, 111–118 (2019) CrossRef Zhang, C., Zheng, Z.X.: Task migration for mobile edge computing using deep reinforcement learning. Future Gener. Comput. Syst. Int. J. Esci. 96, 111–118 (2019) CrossRef
24.
go back to reference Chen, W.H., Liu, B.C., Huang, H.W., Guo, S., Meng, Z.B.: When UAV swarm meets edge-cloud computing: the QoS perspective. IEEE Netw. 33(2), 36–43 (2019) CrossRef Chen, W.H., Liu, B.C., Huang, H.W., Guo, S., Meng, Z.B.: When UAV swarm meets edge-cloud computing: the QoS perspective. IEEE Netw. 33(2), 36–43 (2019) CrossRef
25.
go back to reference Liu, M.T., Yu, F.R., Teng, Y.L., Leung, V.C.M., Song, M.: Computation offloading and content caching n wireless blockchain networks with mobile edge computing. IEEE Trans. Veh. Technol. 67(11), 11008–11021 (2018) CrossRef Liu, M.T., Yu, F.R., Teng, Y.L., Leung, V.C.M., Song, M.: Computation offloading and content caching n wireless blockchain networks with mobile edge computing. IEEE Trans. Veh. Technol. 67(11), 11008–11021 (2018) CrossRef
26.
go back to reference Chen, W.H., Yaguchi, Y., Naruse, K., Watanobe, Y., Nakamura, K.: QoS-aware robotic streaming workflow allocation in cloud robotics systems. IEEE Trans. Serv. Comput. 14(2), 544–558 (2021) CrossRef Chen, W.H., Yaguchi, Y., Naruse, K., Watanobe, Y., Nakamura, K.: QoS-aware robotic streaming workflow allocation in cloud robotics systems. IEEE Trans. Serv. Comput. 14(2), 544–558 (2021) CrossRef
27.
go back to reference Huang, X.Y., Leng, S.P., Maharjan, S., Zhang, Y.: Multi-agent deep reinforcement learning for computation offloading and interference coordination in small cell networks. IEEE Trans. Veh. Technol. 70(9), 9282–9293 (2021) CrossRef Huang, X.Y., Leng, S.P., Maharjan, S., Zhang, Y.: Multi-agent deep reinforcement learning for computation offloading and interference coordination in small cell networks. IEEE Trans. Veh. Technol. 70(9), 9282–9293 (2021) CrossRef
28.
go back to reference Rioul, O., Magossi, J.C.: On Shannon’s formula and Hartley’s rule: beyond the mathematical coincidence. Entropy 16(9), 4892–4910 (2014) MathSciNetCrossRefMATH Rioul, O., Magossi, J.C.: On Shannon’s formula and Hartley’s rule: beyond the mathematical coincidence. Entropy 16(9), 4892–4910 (2014) MathSciNetCrossRefMATH
29.
go back to reference Seng, D.W., Zhang, J.M., Shi, X.Y.: Visual analysis of deep Q-network. KSII Trans. Internet Inf. Syst. 15(3), 853–873 (2021) Seng, D.W., Zhang, J.M., Shi, X.Y.: Visual analysis of deep Q-network. KSII Trans. Internet Inf. Syst. 15(3), 853–873 (2021)
30.
go back to reference Zhang, R., Yu, F.R., Liu, J., Huang, T., Liu, Y.J.: Deep reinforcement learning (DRL)-based device-to-device (D2D) caching With blockchain and mobile edge computing. IEEE Trans. Wireless Commun. 19(10), 6469–6485 (2020) CrossRef Zhang, R., Yu, F.R., Liu, J., Huang, T., Liu, Y.J.: Deep reinforcement learning (DRL)-based device-to-device (D2D) caching With blockchain and mobile edge computing. IEEE Trans. Wireless Commun. 19(10), 6469–6485 (2020) CrossRef
31.
go back to reference Shi, C.M., Wei, B.T., Wei, S.L., Wang, W., Liu, H., Liu, J.L.: A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm. EURASIP J. Wirel. Commun. Netw. 2021(1), 16 (2021) CrossRef Shi, C.M., Wei, B.T., Wei, S.L., Wang, W., Liu, H., Liu, J.L.: A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm. EURASIP J. Wirel. Commun. Netw. 2021(1), 16 (2021) CrossRef
Metadata
Title
A Clustering Offloading Decision Method for Edge Computing Tasks Based on Deep Reinforcement Learning
Authors
Zhen Zhang
Huanzhou Li
Zhangguo Tang
Dinglin Gu
Jian Zhang
Publication date
19-12-2022
Publisher
Springer Japan
Published in
New Generation Computing / Issue 1/2023
Print ISSN: 0288-3635
Electronic ISSN: 1882-7055
DOI
https://doi.org/10.1007/s00354-022-00199-7

Other articles of this Issue 1/2023

New Generation Computing 1/2023 Go to the issue

Premium Partner