Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Bücher
Zeitschriften
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024

Mehr Umsatz mit Top-Kontakten

Am 7. Mai erfahren Sie, wie Vertriebsteams Leadgenerierung, Leadnurturing und Leadstrategien effizient steuern können.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
An Introduction to Big Data, High Performance Computing, High-Throughput Computing, and Hadoop Kapitel lesen Erstes Kapitel lesen

2016 | OriginalPaper | Buchkapitel

6.  dispel4py: Agility and Scalability for Data-Intensive Methods Using HPC

verfasst von : Rosa Filgueira, Malcolm P. Atkinson, Amrey Krause

Erschienen in: Conquering Big Data with High Performance Computing

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Today’s data bonanza and increasing computational power provide many new opportunities for combining observations with sophisticated simulation results to improve complex models and make forecasts by analyzing their relationships. This should lead to well-presented actionable information that can support decisions and contribute trustworthy knowledge. Practitioners in all disciplines: computational scientists, data scientists and decision makers need improved tools to realize such potential. The library dispel4py is such a tool. dispel4py is a Python library for describing abstract workflows for distributed data-intensive applications. It delivers a simple abstract model in familiar development environments with a fluent path to production use that automatically addresses scale without its users having to reformulate their methods. This depends on optimal mappings to many current HPC and data-intensive platforms.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Accelerating Big Data Processing on Modern HPC Clusters
Nächstes Kapitel Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters
Literatur
1.
B. Ács, X. Llorà, L. Auvil, B. Capitanu, D. Tcheng, M. Haberman, L. Dong, T. Wentling, M. Welge, A general approach to data-intensive computing using the Meandre component-based framework, in Proceedings of 1st International Workshop on Workflow Approaches to New Data-centric Science, WANDS ’10 (ACM, New York, 2010), pp. 8:1–8:12
2.
B. Agarwalla et al., Streamline: scheduling streaming applications in a wide area environment. J. Multimedia Syst. 13, 69–85 (2007)CrossRef
3.
4.
S.G. Ahmad et al., Data-intensive workflow optimization based on application task graph partitioning in heterogeneous computing systems, in 4th IEEE International Conference on Big Data and Cloud Computing (2014)
5.
S. Aiche et al., Workflows for automated downstream data analysis and visualization in large-scale computational mass spectrometry. Proteomics 15 (8), 1443–1447 (2015)CrossRef
6.
7.
8.
9.
M.P. Atkinson, M. Parsons, The digital-data challenge, in The DATA Bonanza – Improving Knowledge Discovery for Science, Engineering and Business, Chap. 1, ed. by M.P. Atkinson et al. (Wiley, Hoboken, 2013), pp. 5–13CrossRef
10.
M.P. Atkinson, C.S. Liew, M. Galea, P. Martin, A. Krause, A. Mouat, Ó. Corcho, D. Snelling, Data-intensive architecture for scientific knowledge discovery. Distrib. Parallel Databases 30 (5–6), 307–324 (2012)CrossRef
11.
M.P. Atkinson et al., Data-Intensive thinking with Dispel, in THE DATA BONANZA: Improving Knowledge Discovery for Science, Engineering and Business, Chap. 4 (Wiley, Hoboken, 2013), pp. 61–122
12.
M.P. Atkinson, R. Baxter, P. Besana, M. Galea, M. Parsons, P. Brezany, O. Corcho, J. van Hemert, D. Snelling, The DATA Bonanza – Improving Knowledge Discovery for Science, Engineering and Business (Wiley, Hoboken, 2013)CrossRef
13.
M.P. Atkinson, M. Carpené, E. Casarotti, S. Claus, R. Filgueira, A. Frank, M. Galea, T. Garth, A. Gemünd, H. Igel, I. Klampanos, A. Krause, L. Krischer, S.H. Leong, F. Magnoni, J. Matser, A. Michelini, A. Rietbrock, H. Schwichtenberg, A. Spinuso, J.-P. Vilotte, VERCE delivers a productive e-Science environment for seismology research, in Proceedings of 11th IEEE eScience Conference (2015)
14.
15.
D. Barseghian et al., Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis. Ecol. Inform. 5, 42–50 (2010)CrossRef
16.
S. Beisken et al., KNIME-CDK: workflow-driven cheminformatics. BMC Bioinform. 14 (1), 257 (2013)
17.
K. Belhajjame, J. Zhao, D. Garijo, M. Gamble, K. Hettne, R. Palma, E. Mina, O. Corcho, J.-M. Gómez-Pérez, S. Bechhofer, G. Klyne, C. Goble, Using a suite of ontologies for preserving workflow-centric research objects, in Web Semantics: Science, Services and Agents on the World Wide Web, vol. 32 (2015), pp. 16–42. ISSN:1570-8268
18.
G.B. Berriman et al., Generating complex astronomy workflows, in Workflows for e-Science (Springer, London, 2007)
19.
G.B. Berriman, E. Deelman, P.T. Groth, G. Juve, The application of cloud computing to the creation of image mosaics and management of their provenance, in Software and Cyberinfrastructure for Astronomy, vol. 7740, ed. by N.M. Radziwill, A. Bridger (SPIE, Bellingham, 2010), p. 77401F
20.
M.R. Berthold, N. Cebron, F. Dill, T.R. Gabriel, T. Kötter, T. Meinl, P. Ohl, K. Thiel, B. Wiswedel, Knime - the konstanz information miner. SIGKDD Explor. 11, 26–31 (2009)CrossRef
21.
D. Blankenberg, G.V. Kuster, N. Coraor, G. Ananda, R. Lazarus, M. Mangan, A. Nekrutenko, J. Taylor, Galaxy: a web-based genome analysis tool for experimentalists, in Current Protocols in Molecular Biology (Wiley, New York, 2010)
22.
C. Buil-Aranda, M. Arenas, O. Corcho, A. Polleres, Federating queries in {SPARQL} 1.1: syntax, semantics and evaluation. Web Semant. Sci. Serv. Agents World Wide Web 18 (1), 1–17 (2013). Special section on the semantic and social web
23.
M. Carpené, I. Klampanos, S. Leong, E. Casarotti, P. Danecek, G. Ferini, A. Gemünd, A. Krause, L. Krischer, F. Magnoni, M. Simon, A. Spinuso, L. Trani, M.P. Atkinson, G. Erbacci, A. Frank, H. Igel, A. Rietbrock, H. Schwichtenberg, J.-P. Vilotte, Towards addressing cpu-intensive seismological applications in europe, in Supercomputing, vol. 7905, ed. by J. Kunkel, T. Ludwig, H. Meuer. Lecture Notes in Computer Science (Springer, Berlin/Heidelberg, 2013), pp. 55–66
24.
D. Churches et al., Programming scientific and distributed workflow with Triana services. Concurr. Comput. Pract. Exp. 18 (10), 1021–1037 (2006)CrossRef
25.
26.
D. De Roure, C. Goble, Software design for empowering scientists. IEEE Softw. 26 (1), 88–95 (2009)CrossRef
27.
D. De Roure et al., The design and realisation of the myexperiment virtual research environment for social sharing of workflows. Futur. Gener. Comput. Syst. 25, 561–567 (2009)CrossRef
28.
E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P.J. Maechling, R. Mayani, W. Chen, R.F. da Silva, M. Livny, K. Wenger, Pegasus, a workflow management system for science automation. Futur. Gener. Comput. Syst. 46, 17–35 (2015)CrossRef
29.
30.
31.
32.
33.
Z. Falt, D. Bednárek, M. Kruliš, J. Yaghob, F. Zavoral, Bobolang: a language for parallel streaming applications, in Proceedings of HPDC ’14 (ACM, New York, 2014), pp. 311–314
34.
R. Filgueira, A. Krause, M.P. Atkinson, I. Klampanos, A. Spinuso, S. Sanchez-Exposito, dispel4py: an agile framework for data-intensive escience, in Proceedings of IEEE eScience 2015 (2015)
35.
D. Gannon, B. Plale, S. Marru, G. Kandaswamy, Y. Simmhan, S. Shirasuna, Dynamic, adaptive workflows for mesoscale meteorology, in Workflows for e-Science: Scientific Workflows for Grids, ed. by Taylor et al. (Springer, London, 2007), pp. 126–142
36.
S. Gesing, M.P. Atkinson, R. Filgueira, I. Taylor, A. Jones, V. Stankovski, C.S. Liew, A. Spinuso, G. Terstyanszky, P. Kacsuk, Workflows in a dashboard: a new generation of usability, in Proceedings of WORKS ’14 (IEEE Press, Piscataway, 2014), pp. 82–93
37.
F. Guirado et al., Enhancing throughput for streaming applications running on cluster systems. J. Parallel Distrib. Comput. 73 (8), 1092–1105 (2013)CrossRef
38.
P. Kacsuk (ed.), Science Gateways for Distributed Computing Infrastructures: Development Framework and Exploitation by Scientific User Communities (Springer, Cham, 2014)
39.
S. Kelling, D. Fink, W. Hochachka, K. Rosenberg, R. Cook, T. Damoulas, C. Silva, W. Michener, Estimating species distributions – across space, through time and with features of the environment, in The DATA Bonanza – Improving Knowledge Discovery for Science, Engineering and Business, Chap. 22, ed. by M.P. Atkinson et al. (Wiley, Hoboken, 2013), pp. 441–458CrossRef
40.
H. Koepke, Why Python rocks for research. Technical report, University of Washington (2014)
41.
S. Kohler, S. Gulati, G. Cao, Q. Hart, B. Ludascher, Sliding window calculations on streaming data using the kepler scientific workflow system. Proc. Comput. Sci. 9, 1639–1646 (2012)CrossRef
42.
M. Kozlovszky, K. Karóczkai, I. Márton, P. Kacsuk, T. Gottdank, DCI bridge: executing WS-PGRADE workflows in distributed computing infrastructures, in Science Gateways for Distributed Computing Infrastructures: Development Framework and Exploitation by Scientific User Communities, Chap. 4, ed. by P. Kacsuk (Springer, Cham, 2014), pp. 51–67
43.
L. Lefort et al., W3C Incubator Group Report – review of Sensor and Observation ontologies. Technical report, W3C (2010)
44.
45.
B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E.A. Lee, J. Tao, Y. Zhao, Scientific workflow management and the Kepler system. Concurr. Comput. Pract. Exp. 18 (10), 1039–1065 (2006)CrossRef
46.
P. Maechling, E. Deelman, L. Zhao, R. Graves, G. Mehta, N. Gupta, J. Mehringer, C. Kesselman, S. Callaghan, D. Okaya, H. Francoeur, V. Gupta, Y. Cui, K. Vahi, T. Jordan, E. Field, SCEC CyberShake workflows—automating probabilistic seismic hazard analysis calculations, in Workflows for e-Science: Scientific Workflows for Grids, ed. by I.J. Taylor et al. (Springer London, 2007), pp. 143–163CrossRef
47.
P. Martin, G. Yaikhom, Definition of the DISPEL language, in THE DATA BONANZA: Improving Knowledge Discovery for Science, Engineering and Business, Chap. 10 (Wiley, Hoboken, 2013), pp. 203–236
48.
T. Megies, M. Beyreuther, R. Barsch, L. Krischer, J. Wassermann, ObsPy—What can it do for data centers and observatories? Ann. Geophys. 54 (1), 47–58 (2011)
49.
50.
51.
MPI Forum, MPI: a message-passing interface standard. Int. J. Supercomput. Appl. 8, 165–414 (1994)
52.
53.
54.
55.
I.S. Pérez, M.S. Pérez-Hernández, Towards reproducibility in scientific workflows: an infrastructure-based approach. Sci. Program. 2015, 243180:1–243180:11 (2015)
56.
D. Rogers, I. Harvey, T.T. Huu, K. Evans, T. Glatard, I. Kallel, I. Taylor, J. Montagnat, A. Jones, A. Harrison, Bundle and pool architecture for multi-language, robust, scalable workflow executions. J. Grid Comput. 11 (3), 457–480 (2013)CrossRef
57.
M. Rynge et al., Producing an infrared multiwavelength galactic plane atlas using montage, pegasus and Amazon web services, in ADASS Conference (2013)
58.
Y. Simmhan et al., Building the trident scientific workflow workbench for data management in the cloud, in ADVCOMP (IEEE, Sliema, 2009)
59.
A. Spinuso et al., Provenance for seismological processing pipelines in a distributed streaming workflow, in Proceedings of EDBT ’13 (ACM, New York, 2013), pp. 307–312
60.
M. Stonebraker, P. Brown, D. Zhang, J. Becla, SciDB: a database management system for applications with complex analytics. Comput. Sci. Eng. 15 (3), 54–62 (2013)CrossRef
61.
G. Terstyanszky, T. Kukla, T. Kiss, P. Kacsuk, A. Balasko, Z. Farkas, Enabling scientific workflow sharing through coarse-grained interoperability. Futur. Gener. Comput. Syst. 37, 46–59 (2014)CrossRef
62.
63.
K. Vahi, M. Rynge, G. Juve, R. Mayani, E. Deelman, Rethinking data management for big data scientific workflows, in Workshop on Big Data and Science: Infrastructure and Services (2013)
64.
65.
C. Walter, Kryder’s law: the doubling of processor speed every 18 months is a snail’s pace compared with rising hard-disk capacity, and Mark Kryder plans to squeeze in even more bits. Sci. Am. 293 (2), 32–33 (2005)CrossRef
66.
M. Wilde, M. Hategan, J.M. Wozniak, B. Clifford, D.S. Katz, I. Foster, Swift: a language for distributed parallel scripting. Parallel Comput. 37 (9), 633–652 (2011)CrossRef
67.
K. Wolstencroft, R. Haines, D. Fellows, A. Williams, D. Withers, S. Owen, S. Soiland-Reyes, I. Dunlop, A. Nenadic, P. Fisher, J. Bhagat, K. Belhajjame, F. Bacall, A. Hardisty, A. Nieva de la Hidalga, M.P. Balcazar Vargas, S. Sufi, C. Goble, The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucleic Acids Res. 41 (W1), W557–W561 (2013)CrossRef
68.
J.M. Wozniak, T.G. Armstrong, K. Maheshwari, E.L. Lusk, D.S. Katz, M. Wilde, I.T. Foster, Turbine: a distributed-memory dataflow engine for high performance many-task applications. Fundam. Inform. 128 (3), 337–366, 01 (2013)
Metadaten
Titel
dispel4py: Agility and Scalability for Data-Intensive Methods Using HPC
verfasst von
Rosa Filgueira
Malcolm P. Atkinson
Amrey Krause
Copyright-Jahr
2016
Buch
Conquering Big Data with High Performance Computing

Print ISBN: 978-3-319-33740-1

Electronic ISBN: 978-3-319-33742-5

Copyright-Jahr: 2016

DOI
https://doi.org/10.1007/978-3-319-33742-5_6