Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Behavioral Emulation for Scalable Design-Space Exploration of Algorithms and Architectures Read chapter Read first chapter

2016 | OriginalPaper | Chapter

Motivation and Implementation of a Dynamic Remote Storage System for I/O Demanding HPC Applications

Authors : Matthias Neuer, Jürgen Salk, Holger Berger, Erich Focht, Christian Mosch, Karsten Siegmund, Volodymyr Kushnarenko, Stefan Kombrink, Stefan Wesner

Published in: High Performance Computing

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

I/O subsystem performance is becoming increasingly important for a wide range of applications. The demand can be met with a large memory capacity or fast local SSDs, but such solutions cause very high investment costs and are rather inflexible. Here, we investigate a multi-tiered approach, combining memory, local SSDs and InfiniBand-attached block storage connected via SRP (SCSI RDMA protocol). Different variants of this hybrid storage system are evaluated, and we present a method to analyze the I/O patterns of applications for choosing the best approach. We also demonstrate the integration of the dynamic remote storage facility into the scheduling system Moab. Our method allows for an on-demand provisioning of multi-tiered file systems of varying sizes, managing storage resources automatically with a broker software.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Early Evaluation of the “Infinite Memory Engine” Burst Buffer Solution
next chapter Parallel I/O Architecture Modelling Based on File System Counters
Literature
1.
Zhang, G., Chiu, L., Liu, L.: Adaptive data migration in Multi-tiered storage based cloud environment. In: Proceedings of the IEEE 3rd International Conference on Cloud Computing. IEEE Computer Society (2010)
2.
Werner, H.-J., Knowles, P.J., Knizia, G., Manby, F.R., Schütz, M.: Molpro: a general-purpose quantum chemistry program package. Wiley Interdisc. Rev. Comput. Mol. Sci. 2(2), 242–253 (2012)CrossRef
3.
Byna, S., Chen, Y., Sun, X.-H., Thakur, R., Gropp, W.: Parallel I/O prefetching using MPI file caching and I/O signatures. In: Proceedings of the ACM/IEEE Conference on Supercomputing. IEEE Press, Article 44 (2008)
4.
5.
6.
7.
8.
Neuer, M., Mosch, C., Salk, J., Siegmund, K., Kushnarenko, V., Kombrink, S., Nau, T., Wesner, S.: Storage systems for I/O-intensive applications in computational chemistry. In: Resch, M.M., Bez, W., Focht, E., Kobayashi, H., Qi, J., Roller, S. (eds.) Sustained Simulation Performance, pp. 51–60. Springer, Heidelberg (2015)
9.
Hollowell, C., Hogue, R., Smith, J., Strecker-Kellogg, W., Wong, A., Zaytsev, A.: The effect of flashcache and bcache on I/O performance. J. Phys. Conf. Ser. 513, 2023 (2014)CrossRef
10.
Sweeney, A., Doucette, D., Hu, W., Anderson, C., Nishimoto, M., Peck, G.: Scalability in the XFS file system. In: Proceedings of the USENIX 1996 Technical Conference, San Diego, CA, USA, pp. 1–14 (1996)
11.
Mathur, A., Cao, M., Bhattacharya, S., Dilger, A., Tomas, A., Vivier, L.: The new ext4 filesystem: current status and future plans. In: Proceedings of the Linux Symposium (2007)
12.
Schütz, M., Werner, H.-J.: Low-order scaling local electron correlation methods. IV. linear scaling local coupled-cluster (LCCSD). J. Chem. Phys. 114(2), 661–681 (2001)CrossRef
13.
Holland, M., Gibson, G.A.: Parity declustering for continuous operation in redundant disk arrays. In: Proceeding of the 5th Conference on Architectural Support for Programming Languages and Operating Systems (1992)
14.
Metadata
Title
Motivation and Implementation of a Dynamic Remote Storage System for I/O Demanding HPC Applications
Authors
Matthias Neuer
Jürgen Salk
Holger Berger
Erich Focht
Christian Mosch
Karsten Siegmund
Volodymyr Kushnarenko
Stefan Kombrink
Stefan Wesner
Copyright Year
2016
Book
High Performance Computing

Print ISBN: 978-3-319-46078-9

Electronic ISBN: 978-3-319-46079-6

Copyright Year: 2016

DOI
https://doi.org/10.1007/978-3-319-46079-6_42