Top

Published in:

2021 | OriginalPaper | Chapter

Dynamically Distributing Tasks from an Unattended Parallel Compiler with Cloudbook

Authors : José J. García-Aranda, Juan Ramos-Díaz, Sergio Molina-Cardín, Xavier Larriva-Novo, Andrés Bustos, Luis A. Galindo, Rafael Mayo-García

Published in: High Performance Computing

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

A dynamic version of Cloudbook is presented in this work, a new tool for automatically and unattendedly parallelizing codes which also lately distributes the tasks dynamically. Cloudbook is designed for Python codes and, above all, makes the parallelization in a way in which the number and main characteristics of the available infrastructure is taken into account for optimizing the execution (performance, bandwidth connection, etc.) in a dynamic way. Cloudbook is designed to allow developers to get the technical benefits of automated distribution and parallelization of programs with a very low learning cost. It only requires labelling the original code with a reduced set of pragmas located at function headers. Results of the tests carried out with Cloudbook with several codes on a real infrastructure are presented as well.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

next chapter Fostering Remote Visualization: Experiences in Two Different HPC Sites

Barve, A., Khandelwal, S., Khan, N., Keshatiwar, S., Botre, S.: Serial to parallel code converter tools: a review. Int. J. Res. Advent Tech. Special Issue National Conference “NCPCI-2016” (2016)

Varsha, K.R.: Automatic parallelization tools: a review. IJESC 7(3), 5780–5784 (2017)

Barve, A., Khomane, S., Kulkarni, B., Katare, S., Ghadage, S.: A serial to parallel C++ code converter for multi-core machines. In: Proceedings of the International Conference on ICT in Business Industry & Government (2016)

Alsubhi, K.: An architecture for translating sequential code to parallel. In: Proceedings of the 2nd International Conference on Information System and Data Mining, pp. 88–92, April 2018

André, F., Le Fur, M., Mahéo, Y., Pazat, J.-L.: The Pandore data-parallel compiler and its portable runtime. In: Hertzberger, B., Serazzi, G. (eds.) HPCN-Europe 1995. LNCS, vol. 919, pp. 176–183. Springer, Heidelberg (1995). https://doi.org/10.1007/BFb0046627CrossRef

Chapman, B., Jost, G., van der Pas, R.: Using OpenMP. The MIT Press, Cambridge (2008)

Chavarria-Miranda, D., Mellor-Crummey, J.: An evaluation of data-parallel compiler support for line-sweep applications. In: Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, pp. 7–17. IEEE, New York (2002)

Chavarría-Miranda, D., Mellor-Crummey, J., Sarang, T.: Data-parallel compiler support for multipartitioning. In: Sakellariou, R., Gurd, J., Freeman, L., Keane, J. (eds.) Euro-Par 2001. LNCS, vol. 2150, pp. 241–253. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44681-8_36CrossRefMATH

Di Martino, B., Esposito, A., D’Angelo, S., Maisto, S.A., Nacchia, S.: A compiler for agnostic programming and deployment of big data analytics on multiple platforms. IEEE Trans. Parallel Distrib. Syst. 30(9), 1920–1931 (2019). https://doi.org/10.1109/TPDS.2019.2901488CrossRef

10.

Hsu, A.W.: A Data Parallel Compiler Hosted on the GPU. Indiana University, Bloomington (2019)

11.

Kotsifakou, M.: HPVM: heterogeneous parallel virtual machine. In: Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 68–80. ACM Digital Library (2018)

12.

Iserte, S., et al.: Dynamic management of resource allocation for OmpSs jobs. In: Proceedings of the First Ph.D. Symposium on Sustainable Ultrascale Computing Systems, NESUS COST Action, Timisoara (2016)

13.

Becker, T., Karl, W., Schüle, T.: Evaluating dynamic task scheduling in a task-based runtime system for heterogeneous architectures. In: Schoeberl, M., Hochberger, C., Uhrig, S., Brehm, J., Pionteck, T. (eds.) ARCS 2019. LNCS, vol. 11479, pp. 142–155. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-18656-2_11CrossRef

14.

Ramírez-Velarde, R., Tchernykh, A., Barba-Jimenez, C., Hirales-Carbajal, A., Nolazco-Flores, J.: Adaptive resource allocation with job runtime uncertainty. J. Grid Comput. 15(4), 415–434 (2017). https://doi.org/10.1007/s10723-017-9410-6CrossRef

15.

Becker, T., Busse, P., Schuele, T.: Evaluation of dynamic task scheduling algorithms in a runtime system for heterogeneous architectures. In: 31st International Conference on Architecture of Computing Systems, Braunschweig, pp. 1–8 (2018)

16.

Rodriguez-Pascual, M., Mayo-García, R.M., Llorente, I.M.: Montera: a framework for efficient execution of Monte Carlo codes on grid infrastructure. Comput. Inform. 32, 113–144 (2013)

17.

Rubio-Montero, A.J., Rodríguez-Pascual, M.A., Mayo-García, R.: A simple model to exploit reliable algorithms in cloud federations. Soft. Comput. 21, 4543–4555 (2017). https://doi.org/10.1007/s00500-016-2143-9CrossRef

18.

Kumar, A., Shorey, R.: Performance analysis and scheduling of stochastic fork-join jobs in a multicomputer system. IEEE Trans. Parallel Distrib. Syst. 4, 1147–1164 (1993). https://doi.org/10.1109/71.246075CrossRef

19.

Heggie, D.C.: The Classical Gravitational N-Body Problem. Encyclopaedia of Mathematical Physics, Elsevier (2006)

20.

Romik, D.: Shortest paths in the Tower of Hanoi graph and finite automata. SIAM J. Discret. Math. 20, 610–622 (2006)MathSciNetCrossRef

21.

Demo of the N-Body proof of concept and how it performs. https://drive.google.com/open?id=193f30luFq22cy8QUjzzWHgMA8zKfUAv4

22.

Rodríguez-Pascual, M.A., et al.: Superconducting vortex lattice configurations on periodic potentials: simulation and experiment. J. Supercond. Nov. Magn. 25, 2127–2130 (2012). https://doi.org/10.1007/s10948-012-1636-8CrossRef

Title: Dynamically Distributing Tasks from an Unattended Parallel Compiler with Cloudbook
Authors: José J. García-Aranda
Juan Ramos-Díaz
Sergio Molina-Cardín
Xavier Larriva-Novo
Andrés Bustos
Luis A. Galindo
Rafael Mayo-García
Publisher: Springer International Publishing
Book: High Performance Computing
Print ISBN: 978-3-030-68034-3

Electronic ISBN: 978-3-030-68035-0

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-3-030-68035-0_1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner