Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
MTZ-Fachkongress

Antriebe und Energiesysteme von morgen


Austausch von Automobil- und Energiewirtschaft

Am 14./15. Mai geht es in Chemnitz um die Mobilitätswende durch Elektro- und Wasserstofffahrzeuge.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
GraphScSh: Efficient I/O Scheduling and Graph Sharing for Concurrent Graph Processing Kapitel lesen Erstes Kapitel lesen

2019 | OriginalPaper | Buchkapitel

Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning

verfasst von : Peng Cheng, Yutong Lu, Yunfei Du, Zhiguang Chen, Yang Liu

Erschienen in: Network and Parallel Computing

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

As storage hierarchies are getting deeper on modern high-performance computing systems, intelligent data placement strategies that can choose the optimal storage tier dynamically is the key to realize the potential of hierarchical storage architecture. However, providing a general solution that can be applied in different storage architectures and diverse applications is challenging. In this paper, we propose adaptive storage learner (ASL), which explores the idea of using machine learning techniques to mine the relationship between data placement strategies and I/O performance under varied workflow characteristics and system status, and uses the learned model to choose the optimal storage tier intelligently. We implement a prototype and integrate it into an existing data management system. Empirical comparison based on real scientific workflows tests shows that ASL is capable of combining workflow characteristics and real-time system status to make optimal data placement decisions.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Deep Fusion: A Software Scheduling Method for Memory Access Optimization
Nächstes Kapitel I/O Optimizations Based on Workload Characteristics for Parallel File Systems
Literatur
1.
Habib, S., et al.: Hacc: simulating sky surveys on state-of-the-art supercomputing architectures. New Astron. 42, 49–65 (2016)CrossRef
2.
Kurth, T., et al.: Exascale deep learning for climate analytics. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, TX, USA, 11–16 November 2018, pp. 51:1–51:12 (2018)
3.
Miyoshi, T., et al.: Big data assimilation toward post-petascale severe weather prediction: an overview and progress. Proc. IEEE 104(11), 2155–2179 (2016)CrossRef
4.
Liu, N., Cope, J., Carns, P.H., Carothers, C.D., Ross, R.B., et al.: On the role of burst buffers in leadership-class storage systems. In: IEEE 28th Symposium on Mass Storage Systems and Technologies, MSST 2012, 16–20 April 2012, Asilomar Conference Grounds, pp. 1–11. Pacific Grove, CA, USA (2012)
5.
Docan, C., Parashar, M., Klasky, S.: Dataspaces: an interaction and coordination framework for coupled simulation workflows. Cluster Comput. 15(2), 163–181 (2012)CrossRef
6.
Bhimji, W., Bard, D., Romanus, M.: Accelerating science with the nersc burst buffer early user program. In: LBNL LBNL-1005736, May 2016
7.
8.
9.
Swami, S., Mohanram, K.: Reliable non-volatile memories: techniques and measures. IEEE Des. Test 99, 1 (2017)
10.
Li, H., Ghodsi, A., Zaharia, M., Shenker, S., Stoica, I.: Tachyon: reliable, memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 6:1–6:15. Seattle, WA, USA (2014)
11.
Kakoulli, E., Herodotou, H.: Octopusfs: a distributed file system with tiered storage management. In: Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD 2017, pp. 65–78 (2017)
12.
Dong, B., Byna, S., Wu, K.P., Johansen, H., Johnson, J.N., Keen, N.: Data elevator: low-contention data movement in hierarchical storage system. In: 23rd IEEE International Conference on High Performance Computing (HiPC 2016), pp. 152–161. Hyderabad, India (2016)
13.
Jin, T., et al.: Exploring data staging across deep memory hierarchies for coupled data intensive simulation workflows. In: 2015 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015, pp. 1033–1042 (2015)
14.
Cheng, P., Lu, Y., Du, Y., Chen, Z.: Accelerating scientific workflows with tiered data management system. In: IEEE International Conference on High Performance Computing and Communications (2018)
15.
Subedi, P., Davis, P.E., Duan, S., Klasky, S., Kolla, H., Parashar, M.: Stacker: an autonomic data movement engine for extreme-scale data staging-based in-situ workflows. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2018), pp. 73:1–73:11 (2018)
16.
17.
Deelman, E., Gannon, D., Shields, M.S., Taylor, I.J.: Workflows and e-science: an overview of workflow system features and capabilities. Future Gener. Comp. Syst. 25(5), 528–540 (2009)CrossRef
18.
Deelman, E., et al.: Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46, 17–35 (2015)CrossRef
19.
Wilde, M., Hategan, M., Wozniak, J.M., Clifford, B., Katz, D.S., Foster, I.: Swift: a language for distributed parallel scripting. Parallel Comput. 37(9), 633–652 (2011)CrossRef
20.
Chen, W., Deelman, E.: Workflowsim: a toolkit for simulating scientific workflows in distributed environments. In: 8th IEEE International Conference on E-Science, pp. 1–8 (2012)
21.
Hazekamp, N., et al.: Combining static and dynamic storage management for data intensive scientific workflows. IEEE Trans. Parallel and Distrib. Syst. 99, 1 (2018)
22.
23.
Liao, X., Xiao, L., Yang, C., Yutong, L.: Milkyway-2 supercomputer: system and application. Front. Comput. Sci. 8(3), 345–356 (2014)MathSciNetCrossRef
24.
Taft, R., Vartak, M., Satish, N.R., Sundaram, N., Madden, S., Stonebraker, M.:. Genbase: a complex analytics genomics benchmark. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SGIMOD 2014). ACM (2014)
25.
Krish, K.R., Anwar, A., Butt, A.R.: hats: a heterogeneity-aware tiered storage for hadoop. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 502–511 (2014)
26.
Wang, T., Byna, S., Dong, B., Tang, H.: Univistor: integrated hierarchical and distributed storage for HPC. In: IEEE International Conference on Cluster Computing, CLUSTER 2018, Belfast, UK, 10–13 September 2018, pp. 134–144 (2018)
27.
Kougkas, A., Devarajan, H., Sun, X.H.: Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system. In: Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2018), pp. 219–230 (2018)
28.
Dai, D., Bao, F.S., Zhou, J., Shi, X., Chen, Y.: Vectorizing disks blocks for efficient storage system via deep learning. Parallel Comput. 82, 75–90 (2019)CrossRef
29.
Tomes, E., Rush, E.N., Altiparmak, N.: Towards adaptive parallel storage systems. IEEE Trans. Comput. 67(12), 1840–1848 (2018)MathSciNetCrossRef
30.
Zheng, S., Hoseinzadeh, M., Swanson, S.: Ziggurat: a tiered file system for non-volatile main memories and disks. In: 17th USENIX Conference on File and Storage Technologies, FAST 2019, Boston, MA, 25–28 February 2019, pp. 207–219 (2019)
Metadaten
Titel
Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning
verfasst von
Peng Cheng
Yutong Lu
Yunfei Du
Zhiguang Chen
Yang Liu
Copyright-Jahr
2019
Buch
Network and Parallel Computing

Print ISBN: 978-3-030-30708-0

Electronic ISBN: 978-3-030-30709-7

Copyright-Jahr: 2019

DOI
https://doi.org/10.1007/978-3-030-30709-7_23

Premium Partner

    Bildnachweise