Qiumin Xu
Huzefa Siyamwala
ABSTRACT
The storage subsystem has undergone tremendous innovation in order to keep up with the ever-increasing demand for throughput. Non Volatile Memory Express (NVMe) based solid state devices are the latest development in this domain, delivering unprecedented performance in terms of latency and peak bandwidth. NVMe drives are expected to be particularly beneficial for I/O intensive applications, with databases being one of the prominent use-cases.
This paper provides the first, in-depth performance analysis of NVMe drives. Combining driver instrumentation with system monitoring tools, we present a breakdown of access times for I/O requests throughout the entire system. Furthermore, we present a detailed, quantitative analysis of all the factors contributing to the low-latency, high-throughput characteristics of NVMe drives, including the system software stack. Lastly, we characterize the performance of multiple cloud databases (both relational and NoSQL) on state-of-the-art NVMe drives, and compare that to their performance on enterprise-class SATA-based SSDs. We show that NVMe-backed database applications deliver up to 8× superior client-side performance over enterprise-class, SATA-based SSDs.
- J. Axboe. FIO. http://git.kernel.dk/?p=fio.git;a=summary, 2014.Google Scholar
- J. Axboe and A. D. Brunelle. blktrace User Guide. http://www.cse.unsw.edu.au/~aaronc/iosched/doc/blktrace.html, 2007.Google Scholar
- S. Bates. Accelerating Data Centers Using NVMe and CUDA. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2014/20140805_D11_Bates.pdf, 2014.Google Scholar
- M. Bjørling, J. Axboe, D. Nellans, and P. Bonnet. Linux block IO: Introducing multi-queue SSD access on multi-core systems. In Proceedings of SYSTOR, 2013. Google ScholarDigital Library
- K. Busch. Linux NVMe Driver. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130812_PreConfD_Busch.pdf, 2013.Google Scholar
- A. M. Caulfield, A. De, J. Coburn, T. I. Mollow, R. K. Gupta, and S. Swanson. Moneta: A high-performance storage array architecture for next-generation, non-volatile memories. In Proceedings of MICRO, 2010. Google ScholarDigital Library
- B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of SoCC, 2010. Google ScholarDigital Library
- J. Corbet. A new deadline I/O scheduler. http://lwn.net/Articles/10874/, 2002.Google Scholar
- Datastax. Companies using nosql apache cassandra. http://planetcassandra.org/companies/, 2010.Google Scholar
- DataStax. Cassandra -- About the write path. http://www.datastax.com/documentation/cassandra/1.2/cassandra/dml/dml_write_path_c.html?scroll=concept_ds_wt3_32w_zj__about-the-write-path, 2014.Google Scholar
- Dell. OLTP I/O Profile Study with Microsoft SQL 2012 Using EqualLogic PS Series Storage. http://i.dell.com/sites/doccontent/business/large-business/en/Documents/BP1046_OLTP_IO_Profile_Study.pdf, 2013.Google Scholar
- F. X. Diebold. Big Data Dynamic Factor Models for Macroeconomic Measurementand Forecasting. In Eighth World Congress of the Econometric Society, 2000.Google Scholar
- A. P. Foong, B. Veal, and F. T. Hady. Towards SSD-Ready Enterprise Platforms. In Proceedings of ADMS at VLDB, pages 15--21, 2010.Google Scholar
- FusionIO. Fusion-io Flash Memory as RAM Relief. 2013.Google Scholar
- FusionIO. 1.1 a scalable block layer for high-performance ssd storage. http://kernelnewbies.org/Linux_3.13, 2014a.Google Scholar
- FusionIO. Fusion Power for MongoDB. http://www.fusionio.com/white-papers/fusion-power-for-mongodb, 2014b.Google Scholar
- HammerDB. Hammerdb, version 2.16. http://hammerora.sourceforge.net/, 2014.Google Scholar
- U. Hoelzle and L. A. Barroso. The Datacenter As a Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan and Claypool Publishers, 1st edition, 2009. Google ScholarDigital Library
- X.-Y. Hu, E. Eleftheriou, R. Haas, L. LLiadis, and R. Pletka. Write Amplification Analysis in Flash-Based Solid State Drives. In Proceedings of SYSTOR, 2009. Google ScholarDigital Library
- A. Huffman. NVM Express Revision 1.1. http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1.pdf, 2012.Google Scholar
- Huffman, A. and Juenemann, D. The Nonvolatile Memory Transformation of Client Storage. Computer, 46(8): 38--44, August 2013. Google ScholarDigital Library
- J. Corbet. Anticipatory I/O Scheduling. http://lwn.net/Articles/21274/, 2003.Google Scholar
- K. Jo. Scaling from Datacenter to Client. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2014/20140805_A12_All.pdf, 2014.Google Scholar
- M. Larabel. SCSI Multi-Queue Performance Appears Great For Linux 3.17, 2014.Google Scholar
- MongoDB. MongoDB Storage. http://docs.mongodb.org/manual/faq/storage/, 2014a.Google Scholar
- MongoDB. MongoDB 2.6 Manual. http://docs.mongodb.org/manual/, 2014b.Google Scholar
- MongoDB. Mongodb -- write operations overview. http://docs.mongodb.org/manual/core/write-operations-introduction/, 2014c.Google Scholar
- MongoDB. Performance Considerations for MongoDB. http://info.mongodb.com/rs/mongodb/images/MongoDB-Performance-Considerations_2.4.pdf, 2014d.Google Scholar
- NVM-Express. NVM Express Explained. http://nvmexpress.org/wp-content/uploads/2013/04/NVM_whitepaper.pdf, 2012.Google Scholar
- Oracle. Mysql, version 5.5. http://www.mysql.com/, 2014.Google Scholar
- S. L. Pratt and D. A. Heger. Workload Dependent Performance Evaluation of the Linux 2.6 I/O Schedulers. In Linux Symposium, Ottawa, Canada, July 2004.Google Scholar
- M. Rice. Tuning Linux I/O Scheduler for SSDs, 2013. URL http://dev.nuodb.com/techblog/tuning-linux-io-scheduler-ssds.Google Scholar
- Samsung. SM843 Pro Data Center Series. http://www.samsung.com/global/business/semiconductor/file/media/SM843_Brochure-0.pdf, 2013.Google Scholar
- Samsung. XS1715 Ultra-fast Enterprise Class 1.6TB SSD. http://www.samsung.com/global/business/semiconductor/file/product/XS1715_ProdOverview_2014_1.pdf, 2014.Google Scholar
- SanDisk. The SanDisk Solution for Scaling Cassandra. http://www.sandisk.com/assets/docs/The_Sandisk_Solution_for_Scaling_Cassandra.pdf, 2013.Google Scholar
- SanDisk. MongoDB Product Brief. 2014.Google Scholar
- P. Snyder. tmpfs: A virtual memory file system. http://wiki.deimos.fr/images/1/1e/Solaris_tmpfs.pdf, 1990.Google Scholar
- S. Swanson and A. Caulfield. Refactor, reduce, recycle: Restructuring the i/o stack for the future of storage. IEEE Computer, 46(8): 52--59, August 2013. Google ScholarDigital Library
- R. Thummarukudy. Designing a Configurable NVM Express Controller/Subsystem. http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2014/20140807_I31_Thummarukudy.pdf, 2014.Google Scholar
- TPC. TPC-C Benchmark Standard Specification, Revision 5.11. http://www.tpc.org/tpcc/spec/tpcc_current.pdf, 2010.Google Scholar
- D. H. Walker. A Comparison of NVMe and AHCI. In The Serial ATA International Organization, 2012.Google Scholar
- J. Yang, D. B. Minturn, and F. Hady. When Poll is Better than Interrupt. In Proceedings of FAST, 2012. Google ScholarDigital Library
- Y. J. Yu, D. I. Shin, W. Shin, N. Y. Song, J. W. Choi, H. S. Kim, H. Eom, and H. Y. Yeom. Optimizing the Block I/O Subsystem for Fast Storage Devices. ACM Transactions on Computer Systems (TOCS), 32(2), 2014. Google ScholarDigital Library
Index Terms
- Performance analysis of NVMe SSDs and their implication on real world databases
Recommendations
Improving read performance by isolating multiple queues in NVMe SSDs
IMCOM '17: Proceedings of the 11th International Conference on Ubiquitous Information Management and CommunicationAs NVMe SSDs have become viable, recent researches have focused on optimizing the performance of NVMe SSDs, which provide multiple I/O queues to maximize the I/O parallelism of flash-chip, while traditional operating systems are designed ordinarily for ...
Performance Characterization of Hyperscale Applicationson on NVMe SSDs
SIGMETRICS '15: Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer SystemsThe storage subsystem has undergone tremendous innovation in order to keep up with the ever-increasing demand for throughput. NVMe based SSDs are the latest development in this domain, delivering unprecedented performance in terms of both latency and ...
Performance Characterization of NVMe-over-Fabrics Storage Disaggregation
Special Section on Systor 2017 and Regular PapersStorage disaggregation separates compute and storage to different nodes to allow for independent resource scaling and, thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-...
Comments