Florian Funke
Abstract
Growing main memory sizes have facilitated database management systems that keep the entire database in main memory. The drastic performance improvements that came along with these in-memory systems have made it possible to reunite the two areas of online transaction processing (OLTP) and online analytical processing (OLAP): An emerging class of hybrid OLTP and OLAP database systems allows to process analytical queries directly on the transactional data. By offering arbitrarily current snapshots of the transactional data for OLAP, these systems enable real-time business intelligence.
Despite memory sizes of several Terabytes in a single commodity server, RAM is still a precious resource: Since free memory can be used for intermediate results in query processing, the amount of memory determines query performance to a large extent. Consequently, we propose the compaction of memory-resident databases. Compaction consists of two tasks: First, separating the mutable working set from the immutable "frozen" data. Second, compressing the immutable data and optimizing it for efficient, memory-consumption-friendly snapshotting. Our approach reorganizes and compresses transactional data online and yet hardly affects the mission-critical OLTP throughput. This is achieved by unburdening the OLTP threads from all additional processing and performing these tasks asynchronously.
- D. J. Abadi, S. Madden, and M. Ferreira. Integrating Compression and Execution in Column-Oriented Database Systems. In SIGMOD, pages 671--682, 2006. Google Scholar
Digital Library
- G. Antoshenkov, D. B. Lomet, and J. Murray. Order Preserving Compression. In ICDE, pages 655--663, 1996. Google ScholarDigital Library
- C. Binnig, S. Hildenbrand, and F. Färber. Dictionary-based Order-preserving String Compression for Main Memory Column Stores. In SIGMOD, pages 283--296, 2009. Google ScholarDigital Library
- P. A. Boncz, M. Zukowski, and N. Nes. MonetDB/X100: Hyper-Pipelining Query Execution. In CIDR, pages 225--237, 2005.Google Scholar
- T. Cao, M. A. V. Salles, B. Sowell, Y. Yue, A. J. Demers, J. Gehrke, and W. M. White. Fast Checkpoint Recovery Algorithms for Frequently Consistent Applications. In SIGMOD, pages 265--276, 2011. Google ScholarDigital Library
- Z. Chen, J. Gehrke, and F. Korn. Query Optimization In Compressed Database Systems. In SIGMOD, pages 271--282, 2001. Google ScholarDigital Library
- C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In NSDI, pages 273--286, 2005. Google ScholarDigital Library
- R. Cole, F. Funke, L. Giakoumakis, W. Guy, A. Kemper, S. Krompass, H. A. Kuno, R. O. Nambiar, T. Neumann, M. Poess, K.-U. Sattler, M. Seibold, E. Simon, and F. Waas. The mixed workload CH-benCHmark. In DBTest, page 8, 2011. Google ScholarDigital Library
- G. P. Copeland and S. Khoshafian. A Decomposition Storage Model. In SIGMOD, pages 268--279, 1985. Google ScholarDigital Library
- P. J. Denning. The Working Set Model for Program Behaviour. Communications of the ACM, 11(5):323--333, 1968. Google ScholarDigital Library
- F. Färber, S. K. Cha, J. Primsch, C. Bornhövd, S. Sigg, and W. Lehner. SAP HANA Database: Data Management for Modern Business Applications. SIGMOD Record, 40(4):45--51, 2011. Google ScholarDigital Library
- F. Funke. {s390} introduce dirty bit for kvm live migration. Patch integrated in Linux kernel 2.6.28, Oct 2008. http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=15e86b0c752d50e910b2cca6e83ce74c4440d06c.Google Scholar
- M. Gorman. Understanding the Linux Virtual Memory Manager. Prentice Hall PTR, 2004. Google ScholarDigital Library
- G. Graefe and L. D. Shapiro. Data Compression and Database Performance. In In ACM/IEEE-CS Symposium On Applied Computing, pages 22--27, 1991.Google Scholar
- S. Héman, M. Zukowski, N. J. Nes, L. Sidirourgos, and P. A. Boncz. Positional Update Handling in Column Stores. In SIGMOD, pages 543--554, 2010. Google ScholarDigital Library
- A. L. Holloway, V. Raman, G. Swart, and D. J. DeWitt. How to Barter Bits for Chronons: Compression and Bandwidth Trade Offs for Database Scans. In SIGMOD, pages 389--400, 2007. Google ScholarDigital Library
- S. Idreos, M. L. Kersten, and S. Manegold. Database Cracking. In CIDR, pages 68--78, 2007.Google Scholar
- R. Kallman, H. Kimura, J. Natkins, A. Pavlo, A. Rasin, S. B. Zdonik, E. P. C. Jones, S. Madden, M. Stonebraker, Y. Zhang, J. Hugg, and D. J. Abadi. H-Store: A High-Performance, Distributed Main Memory Transaction Processing System. PVLDB, 1(2):1496--1499, 2008. Google ScholarDigital Library
- A. Kemper and T. Neumann. HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots. In ICDE, pages 195--206, 2011. Google ScholarDigital Library
- A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: the Linux Virtual Machine Monitor. kernel.org, 2007.Google Scholar
- J. Krüger, M. Grund, C. Tinnefeld, H. Plattner, A. Zeier, and F. Faerber. Optimizing Write Performance for Read Optimized Databases. In Database Systems for Advanced Applications, pages 291--305. Springer, 2010. Google ScholarDigital Library
- J. Krüger, C. Kim, M. Grund, N. Satish, D. Schwalb, J. Chhugani, H. Plattner, P. Dubey, and A. Zeier. Fast Updates on Read-Optimized Databases Using Multi-Core CPUs. PVLDB, 5(1):61--72, 2012. Google ScholarDigital Library
- R. A. Lorie. Physical Integrity in a Large Segmented Database. TODS, 2(1):91--104, 1977. Google ScholarDigital Library
- S. Manegold, P. A. Boncz, and M. L. Kersten. What Happens During a Join? Dissecting CPU and Memory Optimization Effects. In VLDB, pages 339--350, 2000. Google ScholarDigital Library
- H. Mühe, A. Kemper, and T. Neumann. How to Efficiently Snapshot Transactional Data: Hardware or Software Controlled? In DaMoN, pages 17--26, 2011. Google ScholarDigital Library
- T. Neumann. Efficiently Compiling Efficient Query Plans. In VLDB, pages 539--550, 2011. Google ScholarDigital Library
- Oracle. Advanced Compression with Oracle Database 11g. Whitepaper, 2011.Google Scholar
- V. Raman, G. Swart, L. Qiao, F. Reiss, V. Dialani, D. Kossmann, I. Narang, and R. Sidle. Constant-Time Query Processing. In ICDE, pages 60--69, 2008. Google ScholarDigital Library
- Transaction Processing Performance Council. TPC-C specification. www.tpc.org/tpcc/spec/TPC-C\_v5-11.pdf, 2010.Google Scholar
- VoltDB. Technical Overview. http://www.voltdb.com, March 2010.Google Scholar
- VoltDB Community. VoltDB TPC-C-like Benchmark Comparison-Benchmark Description. http://community.voltdb.com/node/134, 2010.Google Scholar
- T. Westmann, D. Kossmann, S. Helmer, and G. Moerkotte. The Implementation and Performance of Compressed Databases. SIGMOD Record, 29(3):55--67, 2000. Google ScholarDigital Library
- M. Zukowski, S. Héman, N. Nes, and P. A. Boncz. Super-Scalar RAM-CPU Cache Compression. In ICDE, page 59, 2006. Google ScholarDigital Library
Recommendations
Persistent hybrid transactional memory for databases
Processors with hardware support for transactional memory (HTM) are rapidly becoming commonplace, and processor manufacturers are currently working on implementing support for upcoming non-volatile memory (NVM) technologies. The combination of HTM and ...
Hybrid transactional memory
PPoPP '06: Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programmingHigh performance parallel programs are currently difficult to write and debug. One major source of difficulty is protecting concurrent accesses to shared data with an appropriate synchronization mechanism. Locks are the most common mechanism but they ...
Hybrid transactional memory
Proceedings of the 2006 ASPLOS ConferenceTransactional memory (TM) promises to substantially reduce the difficulty of writing correct, efficient, and scalable concurrent programs. But "bounded" and "best-effort" hardware TM proposals impose unreasonable constraints on programmers, while more ...
Comments