Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Cluster Computing
Published in: Cluster Computing 4/2018

25-06-2018

Optimizing of metadata management in large-scale file systems

Authors: Nae Young Song, Hwajung Kim, Hyuck Han, Heon Young Yeom

Published in: Cluster Computing | Issue 4/2018

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

As modern computer systems face the challenge of large data, filesystems have to deal with a large number of files. This leads to amplified concerns of metadata operations as well as data operations. Most filesystems manage metadata of files by constructing in-memory data structures, such as directory entry (dentry) and inode. We found inefficiencies on management of metadata in existing filesystems, such as path traversal mechanism. In this article, we optimize the metadata operations by (1) looking up dentry cache (dcache) hash table in backward manner. To adopt the backward finding mechanism, we devise the rename and permission-granted mechanism. We also propose (2) compacting the metadata into dentry structures for in-memory space efficiency. We evaluate our optimized metadata managing mechanisms with several benchmarks, including a real-world workload. These optimizations significantly reduce dcache lookup latency by up to 40% and improve overall throughput by up to 72% in a real-world benchmark.
previous article Enabling secure auditing and deduplicating data without owner-relationship exposure in cloud storage
next article CECT: computationally efficient congestion-avoidance and traffic engineering in software-defined cloud data centers

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
Literature
1.
2.
Ghemawat, S., Gobioff, H., Leung, S.T.: The google file system. ACM SIGOPS Operat. Syst. Rev. 37, 29–43 (2003)CrossRef
3.
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass storage Systems and Technologies (MSST), pp. 1–10. IEEE (2010)
4.
Muralidhar, S., Lloyd, W., Roy, S., Hill, C., Lin, E., Liu, W., Pan, S., Shankar, S., Sivakumar, V., Tang, L., et al.: f4: Facebook’s warm blob storage system. In: Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pp. 383–398 (2014)
5.
Beaver, D., Kumar, S., Li, H.C., Sobel, J., Vajgel, P., et al.: Finding a needle in haystack: Facebook’s photo storage. OSDI 10, 1–8 (2010)
6.
Hua, Y., Zhu, Y., Jiang, H., Feng, D., Tian, L.: Scalable and adaptive metadata management in ultra large-scale file systems. In: Proceedings of the 28th International Conference on Distributed Computing Systems, 2008. ICDCS’08. pp. 403–410. IEEE (2008),
7.
Lensing, P.H., Cortes, T., Hughes, J., Brinkmann, A.: File system scalability with highly decentralized metadata on independent storage devices. In: Proceedings of the 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 366–375, IEEE (2016)
8.
Ren, K., Zheng, Q., Patil, S., Gibson, G.: Indexfs: Scaling file system metadata performance with stateless caching and bulk insertion. In: Proceedings of the SC14: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 237–248. IEEE (2014)
9.
Weil, S.A., Pollack, K.T., Brandt, S.A., Miller, E.L.: Dynamic metadata management for petabyte-scale file systems. In: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, p. 4. IEEE Computer Society (2004)
10.
Zhang, S., Catanese, H., Wang, A.A.I.: The composite-file file system: decoupling the one-to-one mapping of files and metadata for better performance. In: Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST 16), pp. 15–22 (2016)
11.
Ren, K., Gibson, G.A.: Tablefs: Enhancing metadata efficiency in the local file system. In: Proceedings of the USENIX Annual Technical Conference, pp. 145–156 (2013)
12.
Patil, S., Gibson, G.A.: Scale and concurrency of giga+: File system directories with millions of files. FAST 11, 13–13 (2011)
13.
Song, N.Y., Kim, H., Yeom, H.Y.: Efficient metadata management in large-scale systems. In: International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia), ACM (2018)
14.
Lee, C., Sim, D., Hwang, J.Y., Cho, S.: F2fs: A new file system for flash storage. In: Proceedings of the 13th Conference on FAST, pp. 273–286 (2015)
15.
16.
Lensing, P.H., Cortes, T., Brinkmann, A.: Direct lookup and hash-based metadata placement for local file systems. In: Proceedings of the 6th International Systems and Storage Conference, p. 5, ACM (2013)
17.
Tsai, C.C., Zhan, Y., Reddy, J., Jiao, Y., Zhang, T., Porter, D.E.: How to get more value from your file system directory cache. In: Proceedings of the 25th Symposium on Operating Systems Principles, pp. 441–456. ACM (2015)
18.
Yuan, J., Zhan, Y., Jannen, W., Pandey, P., Akshintala, A., Chandnani, K., Deo, P., Kasheff, Z., Walsh, L., Bender, M., et al.: Optimizing every operation in a write-optimized file system. In: Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST 16), pp. 1–14 (2016)
19.
McVoy, L.W., Staelin, C., et al.: lmbench: Portable tools for performance analysis. In: Proceedings of the USENIX Annual Technical Conference, pp. 279–294. San Diego, CA, USA (1996)
20.
Katcher, J.: Postmark: A new file system benchmark. Technical Report TR3022, Network Appliance (1997)
21.
22.
Jannen, W., Yuan, J., Zhan, Y., Akshintala, A., Esmet, J., Jiao, Y., Mittal, A., Pandey, P., Reddy, P., Walsh, L., et al.: Betrfs: Write-optimization in a kernel file system. ACM Trans. Storage 11(4), 18 (2015)CrossRef
23.
Zhan, Y., Conway, A., Jiao, Y., Knorr, E., Bender, M.A., Farach-Colton, M., Jannen, W., Johnson, R., Porter, D.E., Yuan, J.: The full path to full-path indexing. In: Proceedings of the 16th USENIX Conference on File and Storage Technologies, pp. 123–138. USENIX Association (2018)
24.
Welch, B., Noer, G.: Optimizing a hybrid ssd/hdd hpc storage system based on file size distributions. In: Proceedings of the 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–12. IEEE (2013)
25.
Bisson, T., Patel, Y., Pasupathy, S.: Designing a fast file system crawler with incremental differencing. ACM SIGOPS Operat. Syst. Rev. 46(3), 11–19 (2012)CrossRef
26.
O’Neil, P., Cheng, E., Gawlick, D., O’Neil, E.: The log-structured merge-tree (lsm-tree). Acta Informa. 33(4), 351–385 (1996)CrossRef
27.
Esmet, J., Bender, M.A., Farach-Colton, M., Kuszmaul, B.C.: The TokuFS streaming file system. In: Proceedings of the HotStorage (2012)
28.
Duchamp, D.: Optimistic lookup of whole nfs paths in a single operation. In: Proceedings of the USENIX Summer, pp. 161–169 (1994)
29.
Welch, B.: A comparison of three distributed file system architectures: Vnode, sprite, and plan 9. Comput. Syst. 7(2), 175–199 (1994)
Metadata
Title
Optimizing of metadata management in large-scale file systems
Authors
Nae Young Song
Hwajung Kim
Hyuck Han
Heon Young Yeom
Publication date
25-06-2018
Publisher
Springer US
Published in
Cluster Computing / Issue 4/2018
Print ISSN: 1386-7857
Electronic ISSN: 1573-7543
DOI
https://doi.org/10.1007/s10586-018-2814-7

Other articles of this Issue 4/2018

Cluster Computing 4/2018 Go to the issue

OriginalPaper

A novel task scheduling approach based on dynamic queues and hybrid meta-heuristic algorithms for cloud computing environment

OriginalPaper

A new MapReduce associative classifier based on a new storage format for large-scale imbalanced data

OriginalPaper

DEFAD: ensemble classifier for DDOS enabled flood attack defense in distributed network environment

OriginalPaper

Secure outsourcing algorithms of modular exponentiations with optimal checkability based on a single untrusted cloud server

OriginalPaper

An implementation of matrix–matrix multiplication on the Intel KNL processor with AVX-512

OriginalPaper

Towards operational cost minimization for cloud bursting with deadline constraints in hybrid clouds

Premium Partner

    Image Credits