Top

World Wide Web

Published in:

14-03-2023

A DRL-based online VM scheduler for cost optimization in cloud brokers

Authors: Xingjia Li, Li Pan, Shijun Liu

Published in: World Wide Web | Issue 5/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The virtual machine (VM) scheduling problem in cloud brokers that support cloud bursting is fraught with uncertainty due to the on-demand nature of Infrastructure as a Service (IaaS) VMs. Until a VM request is received, the scheduler does not know in advance when it will arrive or what configurations it demands. Even when a VM request is received, the scheduler does not know when the VM’s lifecycle expires. Existing studies begin to use deep reinforcement learning (DRL) to solve such scheduling problems. However, they do not address how to guarantee the QoS of user requests. In this paper, we investigate a cost optimization problem for online VM scheduling in cloud brokers for cloud bursting to minimize the cost spent on public clouds while satisfying specified QoS restrictions. We propose DeepBS, a DRL-based online VM scheduler in a cloud broker which learns from experience to adaptively improve scheduling strategies in environments with non-smooth and uncertain user requests. We evaluate the performance of DeepBS under two request arrival patterns which are respectively based on Google and Alibaba cluster traces, and the experiments show that DeepBS has a significant advantage over other benchmark algorithms in terms of cost optimization.

previous article Spatiotemporal data management and analytics for recommender systems

next article Efficient maintenance of highway cover labelling for distance queries on large dynamic graphs

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Hsu, P.-F., Ray, S., Li-Hsieh, Y.-Y.: Examining cloud computing adoption intention, pricing mechanism, and deployment model. Int. J. Inf. Manag. 34(4), 474–488 (2014). https://doi.org/10.1016/j.ijinfomgt.2014.04.006CrossRef

Singh, B., Dhawan, S., Arora, A., Patail, A.: A view of cloud computing. Int. J. Comput. Technol. 4(2b1), 50–58 (2013)

Leavitt, N.: Is cloud computing really ready for prime time? Computer 42(1), 15–20 (2009). https://doi.org/10.1109/MC.2009.20MathSciNetCrossRef

Lucas-Simarro, J.L., Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Scheduling strategies for optimal service deployment across multiple clouds. Futur. Gener. Comput. Syst. 29(6), 1431–1441 (2013). https://doi.org/10.1016/j.future.2012.01.007. Including Special sections: High Performance Computing in the Cloud & Resource Discovery Mechanisms for P2P SystemsCrossRef

Li, X., Pan, L., Liu, S.: A survey of resource provisioning problem in cloud brokers. J. Netw. Comput. Appl. 203, 103384 (2022). https://doi.org/10.1016/j.jnca.2022.103384CrossRef

Cloud Services Brokerage Market Research Report by Service Type, by Platform, by Deployment Model, by Region - Global Forecast to 2026 - Cumulative Impact of COVID-19. https://www.researchandmarkets.com/reports/4989790/cloud-services-brokerage-market-research-report#rela0-4985613. Accessed 29 Nov 2022

State of the Cloud Report Survey. https://info.flexera.com/CM-REPORT-State-of-the-Cloud. Accessed 29 Nov 2022

Nair, S.K., Porwal, S., Dimitrakos, T., Ferrer, A.J., Tordsson, J., Sharif, T., Sheridan, C., Rajarajan, M., Khan, A.U.: Towards secure cloud bursting, brokerage and aggregation. In: 2010 Eighth IEEE European Conference on Web Services, pp. 189–196. https://doi.org/10.1109/ECOWS.2010.33 (2010)

Song, F., Xing, H., Wang, X., Luo, S., Dai, P., Li, K.: Offloading dependent tasks in multi-access edge computing: a multi-objective reinforcement learning approach. Futur. Gener. Comput. Syst. 128, 333–348 (2022). https://doi.org/10.1016/j.future.2021.10.013CrossRef

10.

Caviglione, L., Gaggero, M., Paolucci, M., Ronco, R.: 19. Soft. Comput. 25, 12569–12588 (2021). https://doi.org/10.1007/s00500-020-05462-xCrossRef

11.

Mao, H., Alizadeh, M., Menache, I., Kandula, S.: Resource management with deep reinforcement learning. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks. HotNets ’16, pp. 50–56. Association for Computing Machinery. https://doi.org/10.1145/3005745.3005750 (2016)

12.

Tong, Z., Chen, H., Deng, X., Li, K., Li, K.: A scheduling scheme in the cloud computing environment using deep q-learning. Inf. Sci. 512, 1170–1191 (2020). https://doi.org/10.1016/j.ins.2019.10.035CrossRef

13.

Rolik, O., Zharikov, E., Koval, A., Telenyk, S.: Dynamie management of data center resources using reinforcement learning. In: 2018 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommunications and Computer Engineering (TCSET), pp. 237–244. https://doi.org/10.1109/TCSET.2018.8336194 (2018)

14.

Long, S., Li, Z., Xing, Y., Tian, S., Li, D., Yu, R.: A reinforcement learning-based virtual machine placement strategy in cloud data centers. In: 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 223–230. https://doi.org/10.1109/HPCC-SmartCity-DSS50907.2020.00028 (2020)

15.

Sheng, J., Hu, Y., Zhou, W., Zhu, L., Jin, B., Wang, J., Wang, X.: Learning to schedule multi-numa virtual machines via reinforcement learning. Pattern Recognit. 121, 108254 (2022). https://doi.org/10.1016/j.patcog.2021.108254CrossRef

16.

Li, Y., Tang, X., Cai, W.: On dynamic bin packing for resource allocation in the cloud. In: Proceedings of the 26th ACM Symposium on Parallelism in Algorithms and Architectures. SPAA ’14, pp. 2–11. Association for Computing Machinery. https://doi.org/10.1145/2612669.2612675 (2014)

17.

Li, Y., Tang, X., Cai, W.: Dynamic bin packing for on-demand cloud resource allocation. IEEE Trans. Parallel Distrib. Syst. 27(1), 157–170 (2016). https://doi.org/10.1109/TPDS.2015.2393868CrossRef

18.

Azar, Y., Vainstein, D.: Tight bounds for clairvoyant dynamic bin packing. ACM Trans. Parallel Comput. 6(3). https://doi.org/10.1145/336 (2019)

19.

Gharehpasha, S., Masdari, M.: A discrete chaotic multi-objective sca-alo optimization algorithm for an optimal virtual machine placement in cloud data center. J. Ambient. Intell. Humanized Comput. 12(10), 9323–9339 (2021). https://doi.org/10.1007/s12652-020-02645-0CrossRef

20.

Ghetas, M.: A multi-objective monarch butterfly algorithm for virtual machine placement in cloud computing. Neural Comput. & Applic. 33 (17), 11011–11025 (2021). https://doi.org/10.1007/s00521-020-05559-2CrossRef

21.

Zhao, J., Rodríguez, M.A., Buyya, R.: A deep reinforcement learning approach to resource management in hybrid clouds harnessing renewable energy and task scheduling. In: 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pp. 240–249. https://doi.org/10.1109/CLOUD53861.2021.00037 (2021)

22.

Silva Filho, M. C., Monteiro, C. C., Inácio, P. R. M., Freire, M. M.: Approaches for optimizing virtual machine placement and migration in cloud environments: a survey. J. Parallel Distrib. Comput. 111, 222–250 (2018). https://doi.org/10.1016/j.jpdc.2017.08.010CrossRef

23.

Moghe, U., Lakkadwala, P., Mishra, D.K.: Cloud computing: Survey of different utilization techniques. In: 2012 CSI Sixth International Conference on Software Engineering (CONSEG), pp. 1–4. https://doi.org/10.1109/CONSEG.2012.6349524 (2012)

24.

Cong, P., Xu, G., Wei, T., Li, K.: A survey of profit optimization techniques for cloud providers. ACM Comput. Surv. 53(2), 26 (2020). https://doi.org/10.1145/3376917CrossRef

25.

Ranjana, R., Raja, J.: A survey on power aware virtual machine placement strategies in a cloud data center. In: 2013 International Conference on Green Computing, Communication and Conservation of Energy (ICGCE), pp. 747–752. https://doi.org/10.1109/ICGCE.2013.6823533 (2013)

26.

Zhang, J., Huang, H., Wang, X.: Resource provision algorithms in cloud computing: a survey. J. Netw. Comput. Appl. 64, 23–42 (2016). https://doi.org/10.1016/j.jnca.2015.12.018CrossRef

27.

Kumar, M., Sharma, S. C., Goel, A., Singh, S. P.: A comprehensive survey for scheduling techniques in cloud computing. J. Netw. Comput. Appl. 143, 1–33 (2019). https://doi.org/10.1016/j.jnca.2019.06.006CrossRef

28.

Zhou, G., Tian, W., Buyya, R.: Deep Reinforcement Learning-based Methods for Resource Scheduling in Cloud Computing: A Review and Future Directions. arXiv (2021). https://doi.org/10.48550/ARXIV.2105.04086. https://arxiv.org/abs/2105.04086

29.

Plaat, A.: Deep Reinforcement Learning. Springer (2022). https://doi.org/10.1007/978-981-19-0638-1

30.

Mell, P., Grance, T.: The NIST Definition of Cloud Computing. Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD (2011). https://doi.org/10.6028/NIST.SP.800-145

31.

Empowering App Development for Developers Docker. https://www.docker.com/. Accessed 29 Nov 2022

32.

Bhimani, J., Yang, Z., Mi, N., Yang, J., Xu, Q., Awasthi, M., Pandurangan, R., Balakrishnan, V.: Docker container scheduler for i/o intensive applications running on nvme ssds. IEEE Trans. Multi-Scale Comput. Syst. 4(3), 313–326 (2018). https://doi.org/10.1109/TMSCS.2018.2801281CrossRef

33.

Ramos, F., Viegas, E., Santin, A., Horchulhack, P., dos Santos, R.R., Espindola, A.: A machine learning model for detection of docker-based app overbooking on kubernetes. In: ICC 2021 - IEEE International Conference on Communications, pp. 1–6. (2021). https://doi.org/10.1109/ICC42927.2021.9500259

34.

AWS Wavelength brings AWS compute and storage capabilities to 5G networks to build, deploy, and scale ultra-low latency applications on edge devices. https://aws.amazon.com/wavelength/. Accessed 29 Nov 2022

35.

Szepesvári, C.: Algorithms for reinforcement learning. Synth. Lect. Artif. Intell. Mach. Learn. 4(1), 1–103 (2010)MATH

36.

Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust Region Policy Optimization. In: International Conference on Machine Learning, pp. 1889–1897. https://arxiv.org/abs/1502.05477 (2015)

37.

Sun, W., Yuan, Y.-x.: Optimization theory and methods, vol. 1 (2006)

38.

Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Proceedings of the 12th International Conference on Neural Information Processing Systems. NIPS’99, pp. 1057–1063. MIT Press. https://dl.acm.org/doi/10.5555/3009657.3009806 (1999)

39.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E.Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024–8035. https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf (2019)

40.

Borg cluster traces from Google. https://github.com/google/cluster-data/. Accessed 29 Nov 2022

41.

Alibaba cluster trace program. https://github.com/alibaba/clusterdata/. Accessed 29 Nov 2022

42.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. CoRR abs/1606.01540 (2016). https://arxiv.org/abs/1606.01540

43.

Amazon EC2 On-Demand Pricing. https://aws.amazon.com/ec2/pricing/on-demand/. Accessed 29 Nov 2022

44.

Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: Fair allocation of multiple resource types. In: 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11). USENIX Association. https://www.usenix.org/conference/nsdi11/dominant-resource-fairness-fair-allocation-multiple-resource-types (2011)

45.

Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., Baldeschwieler, E.: Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing. SOCC ’13. Association for Computing Machinery. https://doi.org/10.1145/2523616.2523633 (2013)

46.

Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A.D., Katz, R., Shenker, S., Stoica, I.: Mesos: A platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. NSDI’11, pp. 295–308. USENIX Association. https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center (2011)

47.

Bays, C.: A comparison of next-fit, first-fit, and best-fit. Commun. ACM 20(3), 191–192 (1977). https://doi.org/10.1145/359436.359453CrossRef

48.

Grandl, R., Ananthanarayanan, G., Kandula, S., Rao, S., Akella, A.: Multi-resource packing for cluster schedulers. In: Proceedings of the 2014 ACM Conference on SIGCOMM. SIGCOMM ’14, pp. 455–466. Association for Computing Machinery. https://doi.org/10.1145/2619239.2626334 (2014)

49.

Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. CoRR abs/1602.01783 (2016). https://arxiv.org/abs/1602.01783

50.

Wang, Z., de Freitas, N., Lanctot, M.: Dueling network architectures for deep reinforcement learning. CoRR abs/1511.06581 (2015). https://arxiv.org/abs/1511.06581

51.

van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. CoRR abs/1509.06461 (2015). https://arxiv.org/abs/1509.06461

52.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.A.: Playing atari with deep reinforcement learning. CoRR abs/1312.5602 (2013). https://arxiv.org/abs/1312.5602

53.

RLlib: Industry-Grade Reinforcement Learning. https://docs.ray.io/en/latest/rllib/index.html. Accessed 29 Nov 2022

Title: A DRL-based online VM scheduler for cost optimization in cloud brokers
Authors: Xingjia Li
Li Pan
Shijun Liu
Publication date: 14-03-2023
Publisher: Springer US
Published in: World Wide Web / Issue 5/2023
Print ISSN: 1386-145X
Electronic ISSN: 1573-1413
DOI: https://doi.org/10.1007/s11280-023-01145-3

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 5/2023

Explanation-based data-free model extraction attacks

Beyond model splitting: Preventing label inference attacks in vertical federated learning with dispersed training

Phrase-level attention network for few-shot inverse relation classification in knowledge graph

An efficient privacy-preserving blockchain storage method for internet of things environment

Editor’s note: Special issue on knowledge-graph-enabled methods and applications for the future web

Efficient maintenance of highway cover labelling for distance queries on large dynamic graphs

Premium Partner