Top

Published in:

2021 | OriginalPaper | Chapter

Containerization and Orchestration on HPC Systems

Author : Naweiluo Zhou

Published in: Sustained Simulation Performance 2019 and 2020

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Containerization demonstrates its efficiency in application deployment in Cloud clusters. HPC systems start to adopt containers, as containers can encapsulate complex programs with their dependencies in isolated environments making applications more portable. Nevertheless, conventional HPC workload managers lack micro-service support and deeply-integrated container management, as opposed to container orchestrators. We enable the synergy of Cloud and HPC clusters. We propose the preliminary design of a feedback control scheduler that performs efficient container scheduling meanwhile taking advantage of the scheduling policies of the container orchestrator (Kubernetes) and the HPC workload manager (TORQUE).

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter CFD Simulation with Microservices for Geoscience Applications

next chapter The Role of Machine Learning and Artificial Intelligence in High Performance Computing

Software-defined infrastructure (SDI) is the definition of technical computing infrastructure entirely under the control of software with no operator or human intervention. It operates independent of any hardware-specific dependencies and is programmatically extensible.

Marathon container orchestrator: https://mesosphere.g.ithub.io/marathon/.

Bernstein, D.: Containers and cloud: from LXC to Docker to Kubernetes. IEEE Cloud Comput. 1(3), 81–84 (2014)CrossRef

Buyya, R., Srirama, S.N.: A lightweight container middleware for edge cloud architectures, pp. 145–170. Wiley (2019)

Casalicchio, E.: Container orchestration: a survey. In: Puliafito, A., Trivedi, K.S. (eds.) Systems Modeling: Methodologies and Tools, pp. 221–235. Springer International Publishing, Cham (2019)CrossRef

Casalicchio, E., Iannucci, S.: The state-of-the-art in container technologies: application, orchestration and security. Concurr. Comput. Pract. Exp. e5668 (2020)

Ciechanowicz, P., Poldner, M., Kuchen, H.: The M$\ddot{u}$nster skeleton library Muesli: a comprehensive overview

Containers on virtual machines or bare metals? Tech. Rep., VMWare, Dec 2018

Felter, W., Ferreira, A., Rajamony, R., Rubio, J.: An updated performance comparison of virtual machines and Linux containers. In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 171–172 (2015)

Fernandez, G.P., Brito, A.: Secure container orchestration in the cloud: policies and implementation. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC 19, New York, NY, USA, pp. 138–145. Association for Computing Machinery (2019)

Gao, P.X., Narayan, A., Karandikar, S., Carreira, J., Han, S., Agarwal, R., Ratnasamy, S., Shenker, S.: Network requirements for resource disaggregation. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, USA, OSDI 16, pp. 249–264. USENIX Association (2016)

10.

Gerhardt, L., Bhimji, W., Canon, S., Fasel, M., Jacobsen, D., Mustafa, M., Porter, J., Tsulaia, V.: Shifter: containers for HPC. J. Phys. Conf. Ser. 898, 082021 (2017)CrossRef

11.

Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message-passing Interface. MIT Press, Cambridge, MA, USA (1994)MATH

12.

Hale, J.S., Li, L., Richardson, C.N., Wells, G.N.: Containers for portable, productive, and performant scientific computing. Comput. Sci. Eng. 19(6), 40–50 (2017)CrossRef

13.

Higgins, J., Holmes, V., Venters, C.: Orchestrating docker containers in the HPC environment. In: Kunkel, J.M., Ludwig, T. (eds.) High Performance Computing, pp. 506–513. Springer International Publishing, Cham (2015)CrossRef

14.

Hightower, K., Burns, B., Beda, J.: Kubernetes: Up and Running Dive into the Future of Infrastructure, 1st edn. O’Reilly Media, Inc. (2017)

15.

Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A.D., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, USA, NSDI 11, pp. 295–308. USENIX Association (2011)

16.

Hovestadt, M., Kao, O., Keller, A., Streit, A.: Scheduling in hpc resource management systems: queuing vs. planning. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) Job Scheduling Strategies for Parallel Processing. Springer, Berlin, Heidelberg (2003)

17.

Hu, G., Zhang, Y., and Chen, W. Exploring the performance of singularity for high performance computing scenarios. In: 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Aug 2019, pp. 2587–2593

18.

Jackson, D., Snell, Q., Clement, M.: Core Algorithms of the Maui scheduler. In: Feitelson, D.G., Rudolph, L. (eds.) Job Scheduling Strategies for Parallel Processing, pp. 87–102. Springer, Berlin, Heidelberg (2001)CrossRef

19.

Jette, M.A., Yoo, A.B., Grondona, M.: SLURM: simple linux utility for resource management. In: Proceedings of Job Scheduling Strategies for Parallel Processing (JSSPP) 2003. Lecture Notes in Computer Science, pp. 44–60. Springer (2002)

20.

Julian, S., Shuey, M., Cook, S.: Containers in research: initial experiences with lightweight infrastructure. In: Proceedings of the XSEDE16 Conference on Diversity, Big Data, and Science at Scale, New York, NY, USA, 2016, XSEDE16. Association for Computing Machinery (2016)

21.

Kakadia, D.: Apache Mesos Essentials. Packt Publishing (2015)

22.

Khan, M., Becker, T., Kuppuudaiyar, P., Elster, A.C.: Container-based virtualization for heterogeneous HPC clouds: insights from the EU H2020 cloudlightning project. In: 2018 IEEE International Conference on Cloud Engineering (IC2E), Apr 2018, pp. 392–397 (2018)

23.

Klusáček, D., Chlumský, V., Rudová, H.: Planning and optimization in torque resource manager. In: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing, New York, NY, USA. Association for Computing Machinery (2015)

24.

Kurtzer, G.M., Sochat, V.V., Bauer, M.: Singularity: scientific containers for mobility of compute. PloS one (2017)

25.

Liu, F., Keahey, K., Riteau, P., Weissman, J.: Dynamically negotiating capacity between on-demand and batch clusters. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 18. IEEE Press (2018)

26.

Maenhaut, P.-J., Volckaert, B., Ongenae, V., De Turck, F.: Resource management in a containerized cloud: status and challenges. J. Netw. Syst, Manag. (2019)

27.

Martin, J.P., Kandasamy, A., Chandrasekaran, K.: Exploring the support for high performance applications in the container runtime environment. Hum. Centric Comput. Inf. Sci. 8, 1 (2018)CrossRef

28.

Merkel, D.: Docker: lightweight linux containers for consistent development and deployment. Linux J. 2014, 239 (2014)

29.

Mu’alem, A.W., Feitelson, D.G.: Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans. Parallel Distrib. Syst. 12(6), 529–543 (2001)CrossRef

30.

Narkhede, N., Shapira, G., Palino, T.: Kafka: The Definitive Guide Real-Time Data and Stream Processing at Scale, 1st edn. O’Reilly Media, Inc. (2017)

31.

Pandey, S., Tokekar, V.: Prominence of MapReduce in big data processing. In: 2014 Fourth International Conference on Communication Systems and Network Technologies, pp. 555–560 (2014)

32.

Piras, M.E., Pireddu, L., Moro, M., Zanetti, G.: Container orchestration on HPC clusters. In: Weiland, M., Juckeland, G., Alam, S., Jagode, H. (eds.) High Performance Computing, pp. 25–35. Springer International Publishing, Cham (2019)CrossRef

33.

Plauth, M., Feinbube, L., Polze, A.: A performance survey of lightweight virtualization techniques. In: De Paoli, F., Schulte, S., Broch Johnsen, E. (eds.) Service-Oriented and Cloud Computing, pp. 34–48. Springer International Publishing, Cham (2017)CrossRef

34.

Prabhakaran, S.: Dynamic resource management and job scheduling for high performance computing. Ph.D. thesis, Technische Universität Darmstadt, Darmstadt, Aug 2016

35.

Priedhorsky, R., Randles, T.: Charliecloud: unprivileged containers for user-defined software stacks in HPC. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, New York, NY, USA, SC 17. Association for Computing Machinery (2017)

36.

Reuther, A., Byun, C., Arcand, W., Bestor, D., Bergeron, B., Hubbell, M., Jones, M., Michaleas, P., Prout, A., Rosa, A., Kepner, J.: Scalable system scheduling for HPC and big data. J. Parallel Distrib. Comput. 111, 76–92 (2018)CrossRef

37.

Rodriguez, M.A., Buyya, R.: Container-based cluster orchestration systems: a taxonomy and future directions. Softw. Pract. Exper. 49(5), 698–719 (2019)CrossRef

38.

Rosado, T., Bernardino, J. An overview of openstack architecture. In: Proceedings of the 18th International Database Engineering & Applications Symposium, New York, NY, USA, IDEAS ’14, pp. 366–367. Association for Computing Machinery (2014)

39.

Senthil Kumaran, S.: Practical LXC and LXD: Linux Containers for Virtualization and Orchestration, 1st edn. Apress, USA (2017)CrossRef

40.

Soppelsa, F., Kaewkasi, C.: Native Docker Clustering with Swarm. Packt Publishing (2017)

41.

Staples, G.: Torque resource manager. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, New York, NY, USA, p. 8. Association for Computing Machinery (2006)

42.

Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., Baldeschwieler, E.: Apache hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, New York, NY, USA, SOCC ’13. Association for Computing Machinery (2013)

43.

White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc. (2012)

44.

Wrede, F., and von Hof, V. Enabling efficient use of algorithmic skeletons in cloud environments: container-based virtualization for hybrid CPU-GPU execution of data-parallel skeletons. In: Proceedings of the Symposium on Applied Computing, New York, NY, USA, SAC 17, pp. 1593–1596. Association for Computing Machinery (2017)

45.

Xavier, M.G., Neves, M.V., Rossi, F.D., Ferreto, T.C., Lange, T., De Rose, C.A.F.: Performance evaluation of container-based virtualization for high performance computing environments. In: 2013 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, pp. 233–240 (2013)

46.

Younge, A.J., Pedretti, K., Grant, R.E., Brightwell, R.: A tale of two systems: using containers to deploy HPC applications on supercomputers and clouds. In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 74–81 (2017)

47.

Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, USA, HotCloud10, p. 10. USENIX Association (2010)

48.

Zaharia, M., Xin, R.S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M.J., Ghodsi, A., Gonzalez, J., Shenker, S., Stoica, I.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRef

49.

Zhang, J., Lu, X., Panda, D.K.: Is singularity-based container technology ready for running MPI applications on HPC clouds? In: Proceedings of The10th International Conference on Utility and Cloud Computing, New York, NY, USA, UCC 17. Association for Computing Machinery (2017)

50.

Zhou, N., Georgiou, Y., Zhong, L., Zhou, H., Pospieszny, M.: Container orchestration on HPC systems. In: 2020 IEEE International Conference on Cloud Computing (CLOUD) (2020)

Title: Containerization and Orchestration on HPC Systems
Author: Naweiluo Zhou
Publisher: Springer International Publishing
Book: Sustained Simulation Performance 2019 and 2020
Print ISBN: 978-3-030-68048-0

Electronic ISBN: 978-3-030-68049-7

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-3-030-68049-7_10

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner