Skip to main content

2016 | OriginalPaper | Buchkapitel

Simulation of Runtime Performance of Big Data Workflows on the Cloud

verfasst von : Faris Llwaah, Jacek Cała, Nigel Thomas

Erschienen in: Computer Performance Engineering

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Big data analysis has become a vital tool in many disciplines. Due to its intensive nature, big data analysis is often performed in cloud computing environments. Cloud computing offers the potential for large scale parallelism and scalable provision. However, determining an optimal deployment can be an expensive operation and therefore some form of prediction of performance prior to deployment would be extremely useful. In this paper we explore the deployment of one complex such problem, the NGS pipeline. We use provenance execution data to populate models simulated in WorkflowSim and CloudSim. This allows us to explore different scenarios for runtime properties.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Cała, J., Marei, E., Xu, Y., Takeda, K., Missier, P.: Scalable and efficient whole-exome data processing using workflows on the cloud. Future Gener. Comput. Syst. (2016, in press) Cała, J., Marei, E., Xu, Y., Takeda, K., Missier, P.: Scalable and efficient whole-exome data processing using workflows on the cloud. Future Gener. Comput. Syst. (2016, in press)
2.
Zurück zum Zitat Cała, J., Xu, Y., Wijaya, E., Missier, P.: From scripted HPC-based NGS pipelines to workflows on the cloud. In: 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid (2014) Cała, J., Xu, Y., Wijaya, E., Missier, P.: From scripted HPC-based NGS pipelines to workflows on the cloud. In: 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid (2014)
3.
Zurück zum Zitat Calheiros, R., Ranjan, R., Beloglazov, A., De Rose, C., Buyya, R.: CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw.: Pract. Exp. 41, 23–50 (2010) Calheiros, R., Ranjan, R., Beloglazov, A., De Rose, C., Buyya, R.: CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw.: Pract. Exp. 41, 23–50 (2010)
4.
Zurück zum Zitat Chen, W., Deelman, E.: WorkflowSim: a toolkit for simulating scientific workflows in distributed environments. In: 2012 IEEE 8th International Conference on E-Science (2012) Chen, W., Deelman, E.: WorkflowSim: a toolkit for simulating scientific workflows in distributed environments. In: 2012 IEEE 8th International Conference on E-Science (2012)
5.
Zurück zum Zitat Deelman, E., Gil, Y.: Managing large-scale scientific workflows in distributed environments: experiences and challenges. In: 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science 2006) (2006) Deelman, E., Gil, Y.: Managing large-scale scientific workflows in distributed environments: experiences and challenges. In: 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science 2006) (2006)
6.
Zurück zum Zitat Fan, C., Chang, Y., Wang, W., Yuan, S.: Execution time prediction using rough set theory in hybrid cloud. In: 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing (2012) Fan, C., Chang, Y., Wang, W., Yuan, S.: Execution time prediction using rough set theory in hybrid cloud. In: 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing (2012)
7.
Zurück zum Zitat Iverson, M., Ozguner, F., Potter, L.: Statistical prediction of task execution times through analytic benchmarking for scheduling in a heterogeneous environment. IEEE Trans. Comput. 48, 1374–1379 (1999)CrossRef Iverson, M., Ozguner, F., Potter, L.: Statistical prediction of task execution times through analytic benchmarking for scheduling in a heterogeneous environment. IEEE Trans. Comput. 48, 1374–1379 (1999)CrossRef
8.
Zurück zum Zitat Li, A., Zong, X., Kandula, S., Yang, X., Zhang, M.: CloudProphet. ACM SIGCOMM Comput. Commun. Rev. 41, 426 (2011)CrossRef Li, A., Zong, X., Kandula, S., Yang, X., Zhang, M.: CloudProphet. ACM SIGCOMM Comput. Commun. Rev. 41, 426 (2011)CrossRef
9.
Zurück zum Zitat Long, W., Yuqing, L., Qingxin, X.: Using CloudSim to model and simulate cloud computing environment. In: 2013 Ninth International Conference on Computational Intelligence and Security (2013) Long, W., Yuqing, L., Qingxin, X.: Using CloudSim to model and simulate cloud computing environment. In: 2013 Ninth International Conference on Computational Intelligence and Security (2013)
10.
Zurück zum Zitat Pabinger, S., Dander, A., Fischer, M., Snajder, R., Sperk, M., Efremova, M., Krabichler, B., Speicher, M., Zschocke, J., Trajanoski, Z.: A survey of tools for variant analysis of next-generation genome sequencing data. Brief. Bioinform. 15, 256–278 (2014)CrossRef Pabinger, S., Dander, A., Fischer, M., Snajder, R., Sperk, M., Efremova, M., Krabichler, B., Speicher, M., Zschocke, J., Trajanoski, Z.: A survey of tools for variant analysis of next-generation genome sequencing data. Brief. Bioinform. 15, 256–278 (2014)CrossRef
11.
Zurück zum Zitat Rak, M., Cuomo, A., Villano, U.: Cost/performance evaluation for cloud applications using simulation. In: 2013 Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (2013) Rak, M., Cuomo, A., Villano, U.: Cost/performance evaluation for cloud applications using simulation. In: 2013 Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (2013)
12.
Zurück zum Zitat Rozinat, A., Wynn, M.T., van der Aalst, W.M.P., ter Hofstede, A.H.M., Fidge, C.J.: Workflow simulation for operational decision support using design, historic and state information. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 196–211. Springer, Heidelberg (2008)CrossRef Rozinat, A., Wynn, M.T., van der Aalst, W.M.P., ter Hofstede, A.H.M., Fidge, C.J.: Workflow simulation for operational decision support using design, historic and state information. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 196–211. Springer, Heidelberg (2008)CrossRef
13.
Zurück zum Zitat Rak, M., Turtur, M., Villano, U.: Early prediction of the cost of HPC application execution in the cloud. In: 2014 16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (2015) Rak, M., Turtur, M., Villano, U.: Early prediction of the cost of HPC application execution in the cloud. In: 2014 16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (2015)
14.
Zurück zum Zitat Achour, S., Ammar, M., Khmili, B., Nasri, W.: MPI-PERF-SIM: towards an automatic performance prediction tool of MPI programs on hierarchical clusters. In: 2011 19th International Euromicro Conference on Parallel, Distributed and Network-Based Processing (2011) Achour, S., Ammar, M., Khmili, B., Nasri, W.: MPI-PERF-SIM: towards an automatic performance prediction tool of MPI programs on hierarchical clusters. In: 2011 19th International Euromicro Conference on Parallel, Distributed and Network-Based Processing (2011)
Metadaten
Titel
Simulation of Runtime Performance of Big Data Workflows on the Cloud
verfasst von
Faris Llwaah
Jacek Cała
Nigel Thomas
Copyright-Jahr
2016
DOI
https://doi.org/10.1007/978-3-319-46433-6_10

Neuer Inhalt