Djellel Eddine Difallah
Carlo Curino
Philippe Cudre-Mauroux
Abstract
Benchmarking is an essential aspect of any database management system (DBMS) effort. Despite several recent advancements, such as pre-configured cloud database images and database-as-a-service (DBaaS) offerings, the deployment of a comprehensive testing platform with a diverse set of datasets and workloads is still far from being trivial. In many cases, researchers and developers are limited to a small number of workloads to evaluate the performance characteristics of their work. This is due to the lack of a universal benchmarking infrastructure, and to the difficulty of gaining access to real data and workloads. This results in lots of unnecessary engineering efforts and makes the performance evaluation results difficult to compare. To remedy these problems, we present OLTP-Bench, an extensible "batteries included" DBMS benchmarking testbed. The key contributions of OLTP-Bench are its ease of use and extensibility, support for tight control of transaction mixtures, request rates, and access distributions over time, as well as the ability to support all major DBMSs and DBaaS platforms. Moreover, it is bundled with fifteen workloads that all differ in complexity and system demands, including four synthetic workloads, eight workloads from popular benchmarks, and three workloads that are derived from real-world applications. We demonstrate through a comprehensive set of experiments conducted on popular DBMS and DBaaS offerings the different features provided by OLTP-Bench and the effectiveness of our testbed in characterizing the performance of database services.
- JPA Performance Benchmark. http://www.jpab.org.Google Scholar
- OLTPBenchmark.com. http://oltpbenchmark.com.Google Scholar
- pgbench. http://postgresql.org/docs/9.2/static/pgbench.html.Google Scholar
- PolePosition: The Open Source Database Benchmark. http://polepos.org.Google Scholar
- SysBench: A System Performance Benchmark. http://sysbench.sourceforge.net.Google Scholar
- V. Angkanawaraphan and A. Pavlo. AuctionMark: A Benchmark for High-Performance OLTP Systems. http://hstore.cs.brown.edu/projects/auctionmark.Google Scholar
- A. Arasu, M. Cherniack, E. F. Galvez, D. Maier, A. Maskey, E. Ryvkina, M. Stonebraker, and R. Tibbetts. Linear road: A stream data management benchmark. In VLDB, 2004. Google ScholarDigital Library
- T. G. Armstrong, V. Ponnekanti, D. Borthakur, and M. Callaghan. Linkbench: a database benchmark based on the facebook social graph. In SIGMOD Conference, pages 1185--1196, 2013. Google ScholarDigital Library
- S. Babu, N. Borisov, S. Duan, H. Herodotou, and V. Thummala. Automated experiment-driven management of (database) systems. In HotOS, 2009. Google ScholarDigital Library
- D. Bitton, D. J. DeWitt, and C. Turbyfill. Benchmarking database systems a systematic approach. In VLDB, 1983. Google ScholarDigital Library
- M. J. Cahill, U. Röhm, and A. D. Fekete. Serializable isolation for snapshot databases. SIGMOD, pages 729--738, 2008. Google ScholarDigital Library
- M. J. Cahill, U. Röhm, and A. D. Fekete. Serializable isolation for snapshot databases. ACM Transactions on Database Systems (TODS), 34(4): 20, 2009. Google ScholarDigital Library
- R. Cattell. Scalable SQL and NoSQL data stores. SIGMOD Rec., 39: 12--27, 2011. Google ScholarDigital Library
- M. Cha, H. Haddadi, F. Benevenuto, and K. P. Gummadi. Measuring user influence in Twitter: The million follower fallacy. In ICWSM, May 2010.Google ScholarCross Ref
- R. Cole, F. Funke, L. Giakoumakis, W. Guy, A. Kemper, S. Krompass, H. Kuno, R. Nambiar, T. Neumann, M. Poess, et al. The mixed workload ch-benchmark. In Proceedings of the Fourth International Workshop on Testing Database Systems, page 8. ACM, 2011. Google ScholarDigital Library
- B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with ycsb. In SoCC, pages 143--154, 2010. Google ScholarDigital Library
- C. Curino, E. Jones, R. A. Popa, N. Malviya, E. Wu, S. Madden, H. Balakrishnan, and N. Zeldovich. Relational Cloud: A Database Service for the Cloud. In CIDR, pages 235--240, 2011.Google Scholar
- F. Funke, A. Kemper, and T. Neumann. Benchmarking hybrid OLTP & OLAP database systems. In BTW, pages 390--409, 2011.Google Scholar
- J. Gray. Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., 1992. Google ScholarDigital Library
- W. H. Highleyman. Performance Analysis of Transaction Processing Systems. Prentice Hall, 1989. Google ScholarDigital Library
- D. Kossmann, T. Kraska, and S. Loesing. An evaluation of alternative architectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pages 579--590. ACM, 2010. Google ScholarDigital Library
- M. Lehn, T. Triebel, C. Gross, D. Stingl, K. Saller, W. Effelsberg, A. Kovacevic, and R. Steinmetz. Designing benchmarks for p2p systems. In From Active Data Management to Event-Based Systems and More. 2010. Google ScholarDigital Library
- P. Massa and P. Avesani. Controversial users demand local trust metrics: an experimental study on epinions.com community. In AAAI-05, pages 121--126, 2005. Google ScholarDigital Library
- S. Patil, M. Polte, K. Ren, W. Tantisiriroj, L. Xiao, J. López, G. Gibson, A. Fuchs, and B. Rinaldi. YCSB++: benchmarking and performance debugging advanced features in scalable table stores. SOCC, pages 9:1--9:14, 2011. Google ScholarDigital Library
- A. Pavlo, E. P. Jones, and S. Zdonik. On predictive modeling for optimizing transaction execution in parallel OLTP systems. Proc. VLDB Endow., 5: 85--96, October 2011. Google ScholarDigital Library
- D. R. Ports and K. Grittner. Serializable snapshot isolation in postgresql. Proceedings of the VLDB Endowment, 5(12): 1850--1861, 2012. Google ScholarDigital Library
- S. Ray, B. Simion, and A. Brown. Jackpine: A benchmark to evaluate spatial database performance. In ICDE, 2011. Google ScholarDigital Library
- E. Sarhan, A. Ghalwash, and M. Khafagy. Specification and implementation of dynamic web site benchmark in telecommunication area. In WEAS, pages 863--867, 2008. Google ScholarDigital Library
- Scalyr. Even Stranger than Expected: a Systematic Look at EC2 I/O. http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-io/.Google Scholar
- J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. PVLDB, 3(1), 2010. Google ScholarDigital Library
- A. Schmidt, F. Waas, M. Kersten, M. J. Carey, I. Manolescu, and R. Busse. Xmark: a benchmark for xml data management. In VLDB, 2002. Google ScholarDigital Library
- B. Schroeder, A. Wierman, and M. Harchol-Balter. Open versus closed: a cautionary tale. NSDI, pages 18--18, 2006. Google ScholarDigital Library
- M. Seltzer, D. Krinsky, K. Smith, and X. Zhang. The case for application-specific benchmarking. In HotOS, 1999. Google ScholarDigital Library
- P. Shivam, V. Marupadi, J. Chase, T. Subramaniam, and S. Babu. Cutting corners: workbench automation for server benchmarking. In USENIX, 2008. Google ScholarDigital Library
- M. Stonebraker and A. Pavlo. The SEATS Airline Ticketing Systems Benchmark. http://hstore.cs.brown.edu/projects/seats.Google Scholar
- The Transaction Processing Council. TPC-C Benchmark (Revision 5.9.0). http://www.tpc.org/tpcc/spec/tpcc_current.pdf, June 2007.Google Scholar
- P. Tözün, I. Pandis, C. Kaynak, D. Jevdjic, and A. Ailamaki. From A to E: analyzing TPC's OLTP benchmarks: the obsolete, the ubiquitous, the unexplored. EDBT, pages 17--28, 2013. Google ScholarDigital Library
- G. Urdaneta, G. Pierre, and M. van Steen. Wikipedia workload analysis for decentralized hosting. Comput. Netw., 53: 1830--1845, July 2009. Google ScholarDigital Library
- G. Weikum. Where is the Data in the Big Data Wave? http://wp.sigmod.org/?p=786.Google Scholar
- A. Wolski. TATP Benchmark Description (Version 1.0). http://tatpbenchmark.sourceforge.net, March 2009.Google Scholar
- W. Zheng, R. Bianchini, G. J. Janakiraman, J. R. Santos, and Y. Turner. Justrunit: experiment-based management of virtualized data centers. In USENIX, 2009. Google ScholarDigital Library
Recommendations
Benchmarking OLTP/web databases in the cloud: the OLTP-bench framework
CloudDB '12: Proceedings of the fourth international workshop on Cloud data managementBenchmarking is a key activity in building and tuning data management systems, but the lack of reference workloads and a common platform makes it a time consuming and painful task. The need for such a tool is heightened with the advent of cloud ...
Stream Bench: Towards Benchmarking Modern Distributed Stream Computing Frameworks
UCC '14: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud ComputingWhile big data is becoming ubiquitous, interest in handling data stream at scale is also gaining popularity, which leads to the sprout of many distributed stream computing systems. However, complexity of stream computing and diversity of workloads ...
Comments