Skip to main content
Erschienen in: Cluster Computing 3/2017

01.07.2017

Adaptive hybrid storage systems leveraging SSDs and HDDs in HPC cloud environments

verfasst von: Donghun Koo, Jik-Soo Kim, Soonwook Hwang, Hyeonsang Eom, Jaehwan Lee

Erschienen in: Cluster Computing | Ausgabe 3/2017

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Cloud computing should inherently support various types of data-intensive workloads with different storage access patterns. This makes a high-performance storage system in the Cloud an important component. Emerging flash device technologies such as solid state drives (SSDs) are a viable choice for building high performance computing (HPC) cloud storage systems to address more fine-grained data access patterns. However, the bit-per-dollar SSD price is still higher than the prices of HDDs. This study proposes an optimized progressive file layout (PFL) method to leverage the advantages of SSDs in a parallel file system such as Lustre so that small file I/O performance can be significantly improved. A PFL can dynamically adjust chunk sizes and stripe patterns according to various I/O traffics. Extensive experimental results show that this approach (i.e. building a hybrid storage system based on a combination of SSDs and HDDs) can actually achieve balanced throughput over mixed I/O workloads consisting of large and small file access patterns.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
3.
Zurück zum Zitat Paciucci, G., Paper, S., Meyers, I., Ballantyne, D.: Developing High-Performance, Scalable, cost effective storage solutions with Intel Cloud Edition Lustre and Amazon Web Services. Reference Architecture: Developing Storage Solutions with Intel Cloud Edition for Lustre and Amazon Web Services (2015) Paciucci, G., Paper, S., Meyers, I., Ballantyne, D.: Developing High-Performance, Scalable, cost effective storage solutions with Intel Cloud Edition Lustre and Amazon Web Services. Reference Architecture: Developing Storage Solutions with Intel Cloud Edition for Lustre and Amazon Web Services (2015)
4.
Zurück zum Zitat Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers 2008. In: Workshop on Many-Task Computing on Grids and Supercomputers (2008) Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers 2008. In: Workshop on Many-Task Computing on Grids and Supercomputers (2008)
6.
Zurück zum Zitat Hammond, J.L.: Intel high performance data division. Progressive file layouts prototype. LUG (2015) Hammond, J.L.: Intel high performance data division. Progressive file layouts prototype. LUG (2015)
7.
Zurück zum Zitat Mohr, R., Brim, M., Oral, S., Dilger, A.: Evaluating progressive file layouts for Lustre. LUG (2016) Mohr, R., Brim, M., Oral, S., Dilger, A.: Evaluating progressive file layouts for Lustre. LUG (2016)
8.
Zurück zum Zitat Koo, D., Kim, J.-S., Hwang, S., Eom, H., Lee, J.: Utilizing Progressive File Layout Leveraging SSDs in HPC Cloud Environments. In: Proceedings of the IEEE International Workshops on Foundations and Applications of Self* Systems, September 2016 Koo, D., Kim, J.-S., Hwang, S., Eom, H., Lee, J.: Utilizing Progressive File Layout Leveraging SSDs in HPC Cloud Environments. In: Proceedings of the IEEE International Workshops on Foundations and Applications of Self* Systems, September 2016
9.
Zurück zum Zitat Benchmarking Working Group, OpenSFS. I/O Characterization of Large-Scale HPC Centers. Reference Architecture: the Supercomputing Conference (2012) Benchmarking Working Group, OpenSFS. I/O Characterization of Large-Scale HPC Centers. Reference Architecture: the Supercomputing Conference (2012)
10.
Zurück zum Zitat Lee, J., Koo, D., Park, K., Kim, J., Hwang, S.: Performance analysis of Lustre file system using high performance storage devices. KIISE Trans. Comput. Pract. 22(4), 163–169 (2016)CrossRef Lee, J., Koo, D., Park, K., Kim, J., Hwang, S.: Performance analysis of Lustre file system using high performance storage devices. KIISE Trans. Comput. Pract. 22(4), 163–169 (2016)CrossRef
12.
Zurück zum Zitat Prabhakar, R., Vazhkudai, S.S., Kim, Y., Butt, A.R., Li, M., Kandemir, M.: Provisioning a multi-tiered data staging area for extreme-scale machines. In: Proceedings of the 2011 31st International Conference on Distributed Computing Systems, ICDCS’11, pp. 1–12, IEEE Computer Society, Washington, DC, USA (2011) Prabhakar, R., Vazhkudai, S.S., Kim, Y., Butt, A.R., Li, M., Kandemir, M.: Provisioning a multi-tiered data staging area for extreme-scale machines. In: Proceedings of the 2011 31st International Conference on Distributed Computing Systems, ICDCS’11, pp. 1–12, IEEE Computer Society, Washington, DC, USA (2011)
14.
Zurück zum Zitat Liu, N., Cope, J., Carns, P., Carothers, C., Ross, R., Grider, G., Crume, A., Maltzahn, C.: On the role of burst buffers in leadership-class storage systems. In: IEEE 28th Symposium on MSST/SNAPI. IEEE (2012) Liu, N., Cope, J., Carns, P., Carothers, C., Ross, R., Grider, G., Crume, A., Maltzahn, C.: On the role of burst buffers in leadership-class storage systems. In: IEEE 28th Symposium on MSST/SNAPI. IEEE (2012)
Metadaten
Titel
Adaptive hybrid storage systems leveraging SSDs and HDDs in HPC cloud environments
verfasst von
Donghun Koo
Jik-Soo Kim
Soonwook Hwang
Hyeonsang Eom
Jaehwan Lee
Publikationsdatum
01.07.2017
Verlag
Springer US
Erschienen in
Cluster Computing / Ausgabe 3/2017
Print ISSN: 1386-7857
Elektronische ISSN: 1573-7543
DOI
https://doi.org/10.1007/s10586-017-1002-5

Weitere Artikel der Ausgabe 3/2017

Cluster Computing 3/2017 Zur Ausgabe