Top

The Journal of Supercomputing

Published in:

30-11-2020

Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study

Authors: Peini Liu, Jordi Guitart

Published in: The Journal of Supercomputing | Issue 6/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The high-performance computing (HPC) community has recently started to use containerization to obtain fast, customized, portable, flexible, and reproducible deployments of their workloads. Previous work showed that deploying an HPC workload into a single container can keep bare-metal performance. However, there is a lack of research on multi-container deployments that partition the processes belonging to each application into different containers. Partitioning HPC applications has shown to improve their performance on virtual machines by allowing to set affinity to a non-uniform memory access (NUMA) domain for each of them. Consequently, it is essential to understand the performance implications of distinct multi-container deployment schemes for HPC workloads, focusing on the impact of the container granularity and its combination with processor and memory affinity. This paper presents a systematic performance comparison and analysis of multi-container deployment schemes for HPC workloads on a single-node platform, which considers different containerization technologies (including Docker and Singularity), two different platform architectures (UMA and NUMA), and two application subscription modes (exact subscription and over-subscription). Our results indicate that finer-grained multi-container deployments, on the one side, can benefit the performance of some applications with low interprocess communication, especially in over-subscribed scenarios and when combined with affinity, but, on the other side, they can incur some performance degradation for communication-intensive applications when using containerization technologies that deploy isolated network namespaces.

previous article Application of discrete event-triggered control in wireless signal transmission of IOT

next article DIESEL: A novel deep learning-based tool for SpMV computations and solving sparse linear equation systems

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

https://www.docker.com/.

https://sylabs.io/.

http://icl.cs.utk.edu/hpcc/.

https://tools.bsc.es/paraver.

http://man7.org/linux/man-pages/man1/perf.1.html.

Alam S, Barrett R, Bast M, Fahey MR, Kuehn J, McCurdy C, Rogers J, Roth P, Sankaran R, Vetter JS et al (2008) Early evaluation of IBM BlueGene/P. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (SC’08). IEEE, pp 1–12. https://doi.org/10.1109/SC.2008.5214725

Arango C, Dernat R, Sanabria J (2017) Performance evaluation of container-based virtualization for high performance computing environments. CoRR abs/1709.10140

Azab A (2017) Enabling docker containers for high-performance and many-task computing. In: Proceedings of the 2017 IEEE International Conference on Cloud Engineering (IC2E), pp 279–285. https://doi.org/10.1109/IC2E.2017.52

Bacik J Cpu scheduler imbalance with cgroups. https://josefbacik.github.io/kernel/scheduler/cgroup/2017/07/24/scheduler-imbalance.html

Banerjee A, Mehta R, Shen Z (2015) NUMA aware I/O in virtualized systems. In: Proceedings of the 2015 IEEE 23rd annual symposium on high-performance interconnects, pp 10–17 (2015). https://doi.org/10.1109/HOTI.2015.17

Bermejo B, Juiz C (2020) On the classification and quantification of server consolidation overheads. J Supercomput. https://doi.org/10.1007/s11227-020-03258-2CrossRef

Cheng Y, Chen W, Chen X, Xu B, Zhang S (2013) A user-level numa-aware scheduler for optimizing virtual machine performance. In: Revised selected papers of the 10th international symposium on advanced parallel processing technologies, APPT 2013, vol 8299, pp 32–46. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45293-2_3

Chung MT, Quang-Hung N, Nguyen M, Thoai N (2016) Using docker in high performance computing applications. In: Proceedings of the 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE), pp 52–57. https://doi.org/10.1109/CCE.2016.7562612

Felter W, Ferreira A, Rajamony R, Rubio J (2015) An updated performance comparison of virtual machines and Linux containers. In: Proceedings of the 2015 IEEE international symposium on performance analysis of systems and software (ISPASS). IEEE, pp 171–172. https://doi.org/10.1109/ISPASS.2015.7095802

10.

Google: Cgroups-cpus. https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt

11.

Halácsy G, Ádám Mann Z (2018) Optimal energy-efficient placement of virtual machines with divisible sizes. Inf Process Lett 138:51–56. https://doi.org/10.1016/j.ipl.2018.06.003MathSciNetCrossRefMATH

12.

HPC advisor council: HPCC performance benchmark and profiling (2015). https://hpcadvisorycouncil.com/pdf/HPCC_Analysis_and_Profiling_Intel_E5-2697v3.pdf

13.

HPC wire: Sylabs releases singularity 3.0 container platform; Cites AI Support (2018). https://www.hpcwire.com/2018/10/08/sylabs-releases-singularity-3-0-container-platform-cites-ai-support/

14.

Ibrahim KZ, Hofmeyr S, Iancu C (2011) Characterizing the performance of parallel applications on multi-socket virtual machines. In: Proceedings of the 2011 11th IEEE/ACM international symposium on cluster, cloud and grid computing. IEEE, pp 1–12. https://doi.org/10.1109/CCGrid.2011.50

15.

Ibrahim KZ, Hofmeyr S, Iancu C (2014) The case for partitioning virtual machines on multicore architectures. IEEE Trans Parallel Distrib Syst 25(10):2683–2696. https://doi.org/10.1109/TPDS.2013.242CrossRef

16.

Iosup A, Ostermann S, Yigitbasi MN, Prodan R, Fahringer T, Epema D (2011) Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans Parallel Distrib Syst 22(6):931–945. https://doi.org/10.1109/TPDS.2011.66CrossRef

17.

Jha DN, Garg S, Jayaraman PP, Buyya R, Li Z, Morgan G, Ranjan R (2019) A study on the evaluation of HPC microservices in containerized environment. Concurr Comput. https://doi.org/10.1002/cpe.5323CrossRef

18.

Jha DN, Garg S, Jayaraman PP, Buyya R, Li Z, Ranjan R (2018) A holistic evaluation of docker containers for interfering microservices. In: Proceedings of the 2018 IEEE International Conference on Services Computing (SCC), pp 33–40. https://doi.org/10.1109/SCC.2018.00012

19.

Kuity A, Peddoju SK (2017) Performance evaluation of container-based high performance computing ecosystem using OpenPOWER. In: Kunkel JM, Yokota R, Taufer M, Shalf J (eds) High performance computing, ISC high performance 2017, Lecture notes in computer science. Springer International Publishing, Cham, vol 10524, pp 290–308. https://doi.org/10.1007/978-3-319-67630-2_22

20.

Kurtzer GM, Sochat V, Bauer MW (2017) Singularity: scientific containers for mobility of compute. PLoS ONE 12(5):e0177459. https://doi.org/10.1371/journal.pone.0177459CrossRef

21.

Lozi JP, Lepers B, Funston J, Gaud F, Quéma V, Fedorova A (2016) The Linux scheduler: a decade of wasted cores. In: Proceedings of the Eleventh European Conference on Computer Systems, EuroSys’16. Association for Computing Machinery. https://doi.org/10.1145/2901318.2901326

22.

Luszczek PR, Bailey DH, Dongarra JJ, Kepner J, Lucas RF, Rabenseifner R, Takahashi D (2006) The HPC challenge (HPCC) benchmark suite. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC’06). https://doi.org/10.1145/1188455.1188677

23.

Luszczek P, Koester D (2005) HPC challenge v1.x benchmark suite. SC’05 Tutorial, Seattle, Washington. http://icl.cs.utk.edu/news_pub/submissions/HPCChallengeTutorialDPKPL22Nov2005.pdf

24.

Maliszewski AM, Griebler D, Schepke C, Ditter A, Fey D, Fernandes LG (2018) The NAS benchmark kernels for single and multi-tenant cloud instances with LXC/KVM. In: Proceedings of the 2018 International Conference on High Performance Computing Simulation (HPCS), pp 359–366. https://doi.org/10.1109/HPCS.2018.00066

25.

Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50–60. https://doi.org/10.1214/aoms/1177730491MathSciNetCrossRefMATH

26.

Menouer T (2020) KCSS: Kubernetes container scheduling strategy. J Supercomput. https://doi.org/10.1007/s11227-020-03427-3CrossRef

27.

OpenMPI Team: Can I force aggressive or degraded performance modes? https://www.open-mpi.org/faq/?category=running

28.

OpenMPI Team: Can I oversubscribe nodes (run more processes than processors)? https://www.open-mpi.org/faq/?category=running

29.

Perarnau S, Essen BCV, Gioiosa R, Iskra K, Gokhale MB, Yoshii K, Beckman P (2019) Argo. In: Operating systems for supercomputers and high performance computing. https://doi.org/10.1007/978-981-13-6624-6_12

30.

Pillet V, Labarta J, Cortes T, Girona S (1995) PARAVER: a tool to visualize and analyze parallel code. In: Proceedings of the 18th World Occam and Transputer User Group Technical Meeting. IOS Press, pp 9–13

31.

Rao J, Wang K, Zhou X, Xu C (2013) Optimizing virtual machine scheduling in NUMA multicore systems. In: Proceedings of the 2013 IEEE 19th international symposium on high performance computer architecture (HPCA), pp 306–317. https://doi.org/10.1109/HPCA.2013.6522328

32.

Roloff E, Diener M, Carissimi A, Navaux POA (2012) High performance computing in the cloud: deployment, performance and cost efficiency. In: Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science, pp 371–378. https://doi.org/10.1109/CloudCom.2012.6427549

33.

Rudyy O, Garcia-Gasulla M, Mantovani F, Santiago A, Sirvent R, Vázquez M (2019) Containers in HPC: a scalability and portability study in production biological simulations. In: Proceedings of the 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp 567–577. https://doi.org/10.1109/IPDPS.2019.00066

34.

Saha P, Beltre A, Govindaraju M (2019) Scylla: a mesos framework for container based MPI jobs. CoRR abs/1905.08386

35.

Saha P, Beltre A, Uminski P, Govindaraju M (2018) Evaluation of docker containers for scientific workloads in the cloud. In: Proceedings of the practice and experience on advanced research computing, PEARC’18. Association for Computing Machinery. https://doi.org/10.1145/3219104.3229280

36.

Sande Veiga V, Simon M, Azab A, Fernandez C, Muscianisi G, Fiameni G, Marocchi S (2019) Evaluation and benchmarking of singularity MPI containers on EU research e-infrastructure. In: Proceedings of the 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pp 1–10. https://doi.org/10.1109/CANOPIE-HPC49598.2019.00006

37.

Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3–4):591–611. https://doi.org/10.1093/biomet/52.3-4.591MathSciNetCrossRefMATH

38.

Sharma P, Chaufournier L, Shenoy P, Tay YC (2016) Containers and virtual machines at scale. In: Proceedings of the 17th International Conference on Middleware, pp 1–13. https://doi.org/10.1145/2988336.2988337

39.

Sterling T, Anderson M, Brodowicz M (2018) The essential resource management. In: High performance computing, chapter 5. Morgan Kaufmann, Boston, pp 141–190. https://doi.org/10.1016/B978-0-12-420158-3.00005-8

40.

Tesfatsion SK, Klein C, Tordsson J (2018) Virtualization techniques compared: performance, resource, and power usage overheads in clouds. In: Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering, ICPE ’18. Association for Computing Machinery, pp 145–156. https://doi.org/10.1145/3184407.3184414

41.

Torrez A, Randles T, Priedhorsky R (2019) HPC container runtimes have minimal or no performance impact. In: Proceedings of the 2019 IEEE/ACM international workshop on containers and new orchestration paradigms for isolated environments in HPC (CANOPIE-HPC), pp 37–42. https://doi.org/10.1109/CANOPIE-HPC49598.2019.00010

42.

Tudor BM, Teo YM (2011) A practical approach for performance analysis of shared-memory programs. In: Proceedings of the 2011 IEEE international parallel distributed processing symposium, pp 652–663.https://doi.org/10.1109/IPDPS.2011.68

43.

Vmware: virtualizing high-performance computing (HPC) environments: reference architecture (September) (2018)

44.

Wang Y, Evans RT, Huang L (2019) Performant container support for HPC applications. In: Proceedings of the practice and experience in advanced research computing on rise of the machines (learning), PEARC’19, pp 1–6. Association for Computing Machinery. https://doi.org/10.1145/3332186.3332226

45.

Welch BL (1947) The generalization of student’s problem when several different population variances are involved. Biometrika 34(1–2):28–35. https://doi.org/10.1093/biomet/34.1-2.28MathSciNetCrossRefMATH

46.

Xavier MG, Neves MV, Rossi FD, Ferreto TC, Lange T, De Rose CAF (2013) Performance evaluation of container-based virtualization for high performance computing environments. In: Proceedings of the 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, pp 233–240. https://doi.org/10.1109/PDP.2013.41

47.

Xing F, You H, Lu C (2014) HPC benchmark assessment with statistical analysis. Procedia Comput Sci 29:210–219. https://doi.org/10.1016/j.procs.2014.05.019CrossRef

48.

Yang S, Wang X, An L, Zhang G (2019) Yun: a high-performance container management service based on OpenStack. In: Proceedings of the 2019 IEEE Fourth International Conference on Data Science in Cyberspace (DSC), pp 202–209. https://doi.org/10.1109/DSC.2019.00038

49.

Younge AJ, Pedretti K, Grant RE, Brightwell R (2017) A tale of two systems: using containers to deploy HPC applications on supercomputers and clouds. In: Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp 74–81. https://doi.org/10.1109/CloudCom.2017.40

Title: Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study
Authors: Peini Liu
Jordi Guitart
Publication date: 30-11-2020
Publisher: Springer US
Published in: The Journal of Supercomputing / Issue 6/2021
Print ISSN: 0920-8542
Electronic ISSN: 1573-0484
DOI: https://doi.org/10.1007/s11227-020-03518-1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 6/2021

Performance optimization in delay tolerant networks using backtracking algorithm for fully credits distribution to contrast selfish nodes

K-DBSCAN: An improved DBSCAN algorithm for big data

New set of adapted Gegenbauer–Chebyshev invariant moments for image recognition and classification

Parallelization of the self-organized maps algorithm for federated learning on distributed sources

DIESEL: A novel deep learning-based tool for SpMV computations and solving sparse linear equation systems

Accelerating Louvain community detection algorithm on graphic processing unit

Premium Partner