Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
GraphScSh: Efficient I/O Scheduling and Graph Sharing for Concurrent Graph Processing Read chapter Read first chapter

2019 | OriginalPaper | Chapter

Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning

Authors : Peng Cheng, Yutong Lu, Yunfei Du, Zhiguang Chen, Yang Liu

Published in: Network and Parallel Computing

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

As storage hierarchies are getting deeper on modern high-performance computing systems, intelligent data placement strategies that can choose the optimal storage tier dynamically is the key to realize the potential of hierarchical storage architecture. However, providing a general solution that can be applied in different storage architectures and diverse applications is challenging. In this paper, we propose adaptive storage learner (ASL), which explores the idea of using machine learning techniques to mine the relationship between data placement strategies and I/O performance under varied workflow characteristics and system status, and uses the learned model to choose the optimal storage tier intelligently. We implement a prototype and integrate it into an existing data management system. Empirical comparison based on real scientific workflows tests shows that ASL is capable of combining workflow characteristics and real-time system status to make optimal data placement decisions.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Deep Fusion: A Software Scheduling Method for Memory Access Optimization
next chapter I/O Optimizations Based on Workload Characteristics for Parallel File Systems
Literature
1.
Habib, S., et al.: Hacc: simulating sky surveys on state-of-the-art supercomputing architectures. New Astron. 42, 49–65 (2016)CrossRef
2.
Kurth, T., et al.: Exascale deep learning for climate analytics. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, TX, USA, 11–16 November 2018, pp. 51:1–51:12 (2018)
3.
Miyoshi, T., et al.: Big data assimilation toward post-petascale severe weather prediction: an overview and progress. Proc. IEEE 104(11), 2155–2179 (2016)CrossRef
4.
Liu, N., Cope, J., Carns, P.H., Carothers, C.D., Ross, R.B., et al.: On the role of burst buffers in leadership-class storage systems. In: IEEE 28th Symposium on Mass Storage Systems and Technologies, MSST 2012, 16–20 April 2012, Asilomar Conference Grounds, pp. 1–11. Pacific Grove, CA, USA (2012)
5.
Docan, C., Parashar, M., Klasky, S.: Dataspaces: an interaction and coordination framework for coupled simulation workflows. Cluster Comput. 15(2), 163–181 (2012)CrossRef
6.
Bhimji, W., Bard, D., Romanus, M.: Accelerating science with the nersc burst buffer early user program. In: LBNL LBNL-1005736, May 2016
7.
8.
9.
Swami, S., Mohanram, K.: Reliable non-volatile memories: techniques and measures. IEEE Des. Test 99, 1 (2017)
10.
Li, H., Ghodsi, A., Zaharia, M., Shenker, S., Stoica, I.: Tachyon: reliable, memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 6:1–6:15. Seattle, WA, USA (2014)
11.
Kakoulli, E., Herodotou, H.: Octopusfs: a distributed file system with tiered storage management. In: Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD 2017, pp. 65–78 (2017)
12.
Dong, B., Byna, S., Wu, K.P., Johansen, H., Johnson, J.N., Keen, N.: Data elevator: low-contention data movement in hierarchical storage system. In: 23rd IEEE International Conference on High Performance Computing (HiPC 2016), pp. 152–161. Hyderabad, India (2016)
13.
Jin, T., et al.: Exploring data staging across deep memory hierarchies for coupled data intensive simulation workflows. In: 2015 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015, pp. 1033–1042 (2015)
14.
Cheng, P., Lu, Y., Du, Y., Chen, Z.: Accelerating scientific workflows with tiered data management system. In: IEEE International Conference on High Performance Computing and Communications (2018)
15.
Subedi, P., Davis, P.E., Duan, S., Klasky, S., Kolla, H., Parashar, M.: Stacker: an autonomic data movement engine for extreme-scale data staging-based in-situ workflows. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2018), pp. 73:1–73:11 (2018)
16.
17.
Deelman, E., Gannon, D., Shields, M.S., Taylor, I.J.: Workflows and e-science: an overview of workflow system features and capabilities. Future Gener. Comp. Syst. 25(5), 528–540 (2009)CrossRef
18.
Deelman, E., et al.: Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46, 17–35 (2015)CrossRef
19.
Wilde, M., Hategan, M., Wozniak, J.M., Clifford, B., Katz, D.S., Foster, I.: Swift: a language for distributed parallel scripting. Parallel Comput. 37(9), 633–652 (2011)CrossRef
20.
Chen, W., Deelman, E.: Workflowsim: a toolkit for simulating scientific workflows in distributed environments. In: 8th IEEE International Conference on E-Science, pp. 1–8 (2012)
21.
Hazekamp, N., et al.: Combining static and dynamic storage management for data intensive scientific workflows. IEEE Trans. Parallel and Distrib. Syst. 99, 1 (2018)
22.
23.
Liao, X., Xiao, L., Yang, C., Yutong, L.: Milkyway-2 supercomputer: system and application. Front. Comput. Sci. 8(3), 345–356 (2014)MathSciNetCrossRef
24.
Taft, R., Vartak, M., Satish, N.R., Sundaram, N., Madden, S., Stonebraker, M.:. Genbase: a complex analytics genomics benchmark. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SGIMOD 2014). ACM (2014)
25.
Krish, K.R., Anwar, A., Butt, A.R.: hats: a heterogeneity-aware tiered storage for hadoop. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 502–511 (2014)
26.
Wang, T., Byna, S., Dong, B., Tang, H.: Univistor: integrated hierarchical and distributed storage for HPC. In: IEEE International Conference on Cluster Computing, CLUSTER 2018, Belfast, UK, 10–13 September 2018, pp. 134–144 (2018)
27.
Kougkas, A., Devarajan, H., Sun, X.H.: Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system. In: Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2018), pp. 219–230 (2018)
28.
Dai, D., Bao, F.S., Zhou, J., Shi, X., Chen, Y.: Vectorizing disks blocks for efficient storage system via deep learning. Parallel Comput. 82, 75–90 (2019)CrossRef
29.
Tomes, E., Rush, E.N., Altiparmak, N.: Towards adaptive parallel storage systems. IEEE Trans. Comput. 67(12), 1840–1848 (2018)MathSciNetCrossRef
30.
Zheng, S., Hoseinzadeh, M., Swanson, S.: Ziggurat: a tiered file system for non-volatile main memories and disks. In: 17th USENIX Conference on File and Storage Technologies, FAST 2019, Boston, MA, 25–28 February 2019, pp. 207–219 (2019)
Metadata
Title
Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning
Authors
Peng Cheng
Yutong Lu
Yunfei Du
Zhiguang Chen
Yang Liu
Copyright Year
2019
Book
Network and Parallel Computing

Print ISBN: 978-3-030-30708-0

Electronic ISBN: 978-3-030-30709-7

Copyright Year: 2019

DOI
https://doi.org/10.1007/978-3-030-30709-7_23

Premium Partner

    Image Credits