Micro-architectural analysis of in-memory OLTP: Revisited

Recent years have witnessed the rise of in-memory or main-memory optimized OLTP systems [10, 33, 56]. Traditional OLTP engines are disk-based since they are designed in an era where the main-memory size was in megabytes. Today, however, a server hardware with 1TB main-memory is a commodity. Therefore, the database management systems (DBMS) are able to process the data working set of most OLTP applications in-memory. This has led various vendors and researchers to design brand-new OLTP engines optimized for the case where the dataset resides in memory [23, 26, 27, 52].

In-memory OLTP systems have significant differences compared to disk-based systems. First, since the data working set resides mostly in memory, in-memory OLTP systems omit the buffer pool component, which acts as the virtual memory of a DBMS and is, therefore, essential for the disk-based systems. Then, they tend to adopt more lightweight concurrency control mechanisms to avoid the scalability bottlenecks that arise due to traditional centralized locking. They also opt for cache-conscious indexes instead of the disk-optimized B-trees. Finally, since their codebases are written from scratch, they tend to have lighter storage engines in terms of the instruction footprint.

OLTP benchmarks are famous for their suboptimal micro-architectural behavior. There is a large body of work that characterizes OLTP benchmarks at the micro-architectural level [5, 12, 22, 41, 50, 53, 54]. They all conclude that OLTP exhibits high stall time (\(>50\%\) of the execution cycles) and a low instructions-per-cycle (IPC) value (\(<1\) IPC on machines that can retire up to 4 instructions in a cycle) [12]. The instruction cache misses that mainly stem from the large instruction footprint of transactions are the main source of the stall time, while the next contributing factor is the long-latency data misses from the last-level cache (LLC) [53].

All the previous workload characterization studies, however, run the OLTP benchmarks on a disk-based OLTP engine. Considering the lighter components, cache-friendly data structures, and cleaner codebase of in-memory systems, one expects them to exhibit better cache locality (especially for the instruction cache) and less memory stall time. Due to the distinctive design features of the in-memory systems from the disk-based ones, however, it is not straightforward to extrapolate how OLTP benchmarks behave at the micro-architectural level when run on an in-memory engine solely by looking at the results of previous studies.

In this paper, we perform a detailed analysis of the micro-architectural behavior of the in-memory OLTP systems. More specifically, we compare three in-memory OLTP systems (an in-memory OLTP engine of a popular commercial vendor, a ground-up designed in-memory OLTP system, and an open-source OLTP engine, Silo [55]) to two disk-based OLTP systems (a popular commercial DBMS and an open-source OLTP engine, Shore-MT [43]). We examine CPU cycles breakdowns while running simple micro-benchmarks as well as the more complex TPC benchmarks (TPC-B and TPC-C) [1]. Our analysis demonstrates the following:This paper revisits [48] by using the methodology proposed by [49] on a later generation Intel processor. The newly used methodology allows end-to-end execution time profiling rather than merely relying on cache miss counts. The new generation of Intel processor reveals a significant change in the micro-architectural behavior of the OLTP systems. In particular, the main performance bottleneck of the disk-based and ground-up designed in-memory OLTP system shifts from instruction cache miss stalls to data cache miss stalls thanks to the improvements in the core micro-architecture. We update the contributions and the text according to the newly seen micro-architectural behavior. Furthermore, we compare and contrast Intel’s two successive processor generations and reason about the significant change in the micro-architectural behavior. We use Silo kernel OLTP engine instead of HyPer, as HyPer is not publicly available anymore. Using Silo allows assessing the complexity of the core of an in-memory OLTP system and hence making the difference between an end-to-end OLTP system and an OLTP kernel engine. Finally, we analyze memory bandwidth and commonly available acceleration features—hyper-threading, turbo-boost, and hardware prefetchers—completing the hardware–software interaction analysis of OLTP on modern processors.

The rest of the paper is organized as follows. Section 2 gives an overview of the in-memory OLTP systems- and surveys-related work on workload characterization and micro-architectural analysis studies. Section 3 describes the experimental methodology. Sections 4 and 5 present the analysis results with a micro-benchmark and TPC benchmarks, respectively. Section 6 analyzes the effects of transaction compilation, index structures, and data types, whereas Sect. 7 investigates the impact of multi-threading on the micro-architectural behavior. Section 8 presents the analysis of the memory bandwidth consumptions. Section 9 analyzes the acceleration features such as hyper-threading, turbo-boost, and hardware prefetchers. Section 10 compares two latest generations of Intel’s successive micro-architectures. Finally, Sect. 11 discusses the results and Sect. 12 concludes.

In-memory DBMS gained a lot of popularity in the last decade. In Sect. 2.1, we detail the underlying factors for this trend and the main design characteristics of in-memory OLTP systems. Then, in Sect. 2.2, we go over the recent workload characterization studies that focus on OLTP applications and highlight why they are not representative for in-memory OLTP systems.

The experiments presented in this paper are executed on real hardware. The rest of this section details the setup and methodology for our study.

Hardware We run experiments on a modern commodity server with Intel’s Broadwell processors. Table 1 shows the architectural details of this server. To collect numbers about various hardware events and break down the time spent in specific code modules, we use Intel VTune Amplifier XE 2018 [17], which provides an API for lightweight hardware counter sampling. We disable hyper-threading and turbo-boost to obtain more precise hardware sampling values and increase predictability in measurements.

OS & Compiler We use Ubuntu 16.04.6 LTS and gcc 5.4.0 on the Broadwell server.

Benchmarks We run two types of benchmarks: micro-benchmarks and TPC benchmarks [1]. Our goal is to perform sensitivity analysis and have a more detailed understanding of the systems using the micro-benchmark, while the experiments using the TPC benchmarks serve to give an idea about the behavior of the systems when running well-known real-world applications.

The micro-benchmark uses a randomly generated table with two columns (key and value) of the type Long. It has two versions: read-only and read-write. The read-only version reads N random rows from the table, whereas the read-write version updates N random rows. Both versions use an index lookup operation on the randomly picked key value to reach the row to be read or updated. We also use a modified version of the micro-benchmark where we use strings of 50 bytes for both columns to quantify the impact of data type on micro-architectural utilization in Sect. 6.2.

As for the TPC benchmarks, we use TPC-B and TPC-C. We omit the TPC-E benchmark since recent workload characterization studies demonstrate that TPC-E exhibits similar micro-architectural behavior to the TPC-B and TPC-C benchmarks [12, 54].

Analyzed systems We analyze three in-memory OLTP systems: the in-memory OLTP engine of a closed-source commercial vendor (DBMS M), an open-source commercial OLTP system (DBMS N), and an open-source OLTP engine (Silo [55]).

We pick these three systems as they are well known in the community and their design characteristics represent a good variety among today’s in-memory OLTP systems. While DBMS M adopts multi-versioned concurrency control, DBMS N uses physical data partitioning, and Silo uses optimistic concurrency control. DBMS M implements both hash index and a variant of cache-conscious B-tree index similar to [28, 29]. DBMS N uses a variant of a self-balancing binary search tree, red-black tree. Silo implements Masstree, a highly parallel in-memory index structure [32]. For DBMS M, we use the B-tree index. Moreover, DBMS M use transaction compilation techniques for the stored procedures, whereas DBMS N and Silo do not. We evaluate DBMS M always having its transaction compilation feature turned on in our analyses, except in Sect. 6.1, where we evaluate the transaction compilation feature.

In order to gain better insights about the differences between the in-memory and disk-based OLTP systems, we also include two disk-based systems: a popular, commercial system (DBMS D) and the open-source Shore-MT [43] storage manager.

To implement benchmarks, we use the SQL frontend of the commercial systems, DBMS D, DBMS M, and DBMS N, and Silo’s benchmarks in C++ and Shore-MT’s Shore-Kits suite that provides an environment to implement benchmarks for Shore-MT in C++.

For all the systems, we use asynchronous logging. Therefore, there is no delay due to I/O in the critical path of the transaction execution.

We profile the database server process by attaching VTune to it during a 120-second benchmark run following a 60-second warm-up period. We repeat every experiment three times and report the average result.

In terms of micro-architectural efficiency, our goal is to observe how well each system exploits the resources of a single core regardless of the parallelism in the system. Therefore, all the experiments except for the ones in Sects. 7 and 8 use a single worker thread executing the transactions.

The choice of a single worker thread also eliminates contention due to several threads trying to access the shared data in the case of non-partitioned systems and distributed transactions in the case of partitioning-based systems. This way we avoid possible misleading micro-architectural conclusions. For example, high contention for a shared data page could lead to multiple threads spinning on a latch for that data page, thus artificially increasing the cache hit ratio.

We use one client to generate request in the single-threaded experiments. Shore-MT, DBMS D, Silo, and DBMS M assign one worker thread per client. DBMS N, on the other hand, generates one worker thread per data partition, so we configure it to have only one partition. From VTune, we filter the hardware counter results particularly for the identified worker thread excluding the other threads that are responsible for background tasks, e.g., communication between the server and client, parsing transactions, etc.

In multi-threaded experiments (Sects. 7 and 8), we use multiple clients to generate requests for all systems. For DBMS N, we also use multiple data partitions and ensure that all transactions access only a single partition. For each system, we gradually increase the number of clients, and profile the execution with the number of clients that give the highest aggregate throughput. From VTune, we filter hardware counter results for each worker thread separately and report their average.

VTune We use Intel VTune 2018. We use VTune’s built-in general-exploration analysis for the breakdown of the CPU cycles per the Top-down Micro-architecture Analysis Method. We use VTune’s built-in memory-access analysis to measure the consumed memory bandwidth. As we numa-localize our experiments on a single socket, we report average bandwidth per-socket values. We use VTune’s built-in advanced-hotspots analysis to perform function call trace breakdown.

VTune’s general-exploration provides the breakdown of the CPU cycles [49, 58]. We use a simplified version of VTune’s original CPU cycles categorization to be able to interpret the results more easily. We follow the same simplified categorization used in our previous work [46]. In Table 18, Section C of the “Appendix”, we provide how each individual CPU cycles component that VTune reports maps to the CPU cycles category that we use.

Each CPU pipeline slot is categorized into one of two components: retiring and stalling. A retiring cycle is a cycle where the processor finishes the execution of an instruction, i.e., retires an instruction. A stalling cycle is a cycle where the processor has to wait, i.e., stall, (e.g., to perform a read from the caches). In an ideal scenario, all CPU cycles would be retiring. Stalling cycles can be further decomposed into five components: (i) branch misprediction, (ii) Icache, (iii) decoding, (iv) Dcache, and (v) resource/dependency. Today’s processors use a hardware unit called branch predictor; it predicts the outcome of a branch instruction (i.e., an if() statement) and speculatively executes instructions per the predicted branch direction and/or target. If the processor then realizes the prediction is not correct, it undoes whatever it has been doing and starts executing the correct set of instructions. This cost is defined as the branch misprediction and can be very costly, as it requires canceling a large amount of work. Icache defines the cost of instruction cache and instruction translation lookaside buffer misses. Decoding defines the cost of sub-optimal micro-architectural implementation of the instruction decoding unit. Dcache defines the cost of data-cache misses. Resource/dependency defines the cost of executing instruction that has resource and/or data dependencies. For example, if two instructions require using the same arithmetic-logic unit, one has to wait for the other. This time is identified as the resource dependency time. Or, if an instruction’s operand depends on the result of another instruction, the instruction with the dependent operand has to wait for the other instruction to finish. This time is identified as the data dependency time.

Before performing an analysis using the community standard TPC benchmarks, we devise a sensitivity study using the micro-benchmark. The goal of this study is to answer the following questions:To answer these questions, we break the analysis into two parts. The first part (Sect. 4.1) varies the database size by varying the number of rows in the table while keeping the amount of work done per transaction constant. On the other hand, the second part (Sect. 4.2) varies the amount of work done per transaction by increasing the number of rows read in a transaction while keeping the database size constant.

Section 4 performs a sensitivity analysis using a simple micro-benchmark to gain a fine-grained understanding of the the micro-architectural behavior of the OLTP systems. This section investigates the behavior of the same systems while running the more complex and community standard TPC-B (Sect. 5.1) and TPC-C (Sect. 5.2) benchmarks. All the experiments in this section use a database of size 100 GB, except DBMS M for which we use the maximum allowed database size of 32 GB. Similar to Sect. 4, we analyze the CPU cycles breakdowns and normalized throughput values.

DBMS D and DBMS M are instruction-stalls-bound. Therefore, their delivered throughput does not drop significantly as the data size is increased as described in Sect. 4.2, and we assume that DBMS M’s throughput for 100 GB is similar to its throughput for 32 GB both for TPC-B and TPC-C.

This section analyzes the impact of index and compilation optimizations the in-memory systems adopt, as well as the impact of the data types, at the micro-architectural level. Among the systems used in this study, DBMS M is the only one that allows enabling/disabling the compilation optimizations and using two different index structures: hash index and a variant of cache-conscious B-tree index similar to [28, 29]. Therefore, while we use DBMS M for analyzing the impact of index and compilation optimizations, we experiment with all the three in-memory systems (DBMS M, DBMS N, and Silo) to quantify the effect of different data types.

This section analyzes the effect of running multiple server side threads on the micro-architectural behavior. The single-threaded experiments aim to present an idealized case since it avoids cache invalidations due to data sharing across different worker threads or misleading artificially high IPC values due to threads spinning under possible contention. On the other hand, multi-threaded experiments aim to investigate a more realistic scenario where systems are loaded with multiple threads executing transactions from multiple clients.

Figures 13 and 14 show the CPU cycles breakdowns while running the read-only version of the micro-benchmark when reading 1 row and TPC-C benchmark, respectively. We use a database of size 100 GB in both of the experiments for all the systems except DBMS M. We use 10 GB of database for the micro-benchmark, and 32 GB of database for the TPC-C benchmark for DBMS M. We observe that the micro-architectural behavior of the individual systems is similar to the single-threaded executions when running the micro-benchmark. DBMS D mainly suffers from the Icache stalls, whereas Shore-MT, DBMS N and Silo majorly suffer from the Dcache stalls. DBMS M suffers less from the Icache stalls compared to the single-threaded execution. We examined the code modules breakdown of DBMS M and observed that the amount of time it spends inside the OLTP engine remains the same across the single- and multi-threaded executions. However, the amount of time DBMS M spends in different modules that are outside the OLTP engine changes significantly across the single- and multi-threaded executions. Hence, the reduced Icache stalls in the multi-threaded execution is likely due to the higher instruction locality of the code modules that are more active during multi-threaded execution.

Micro-architectural behavior follows similar trends for TPC-C for the multi-threaded execution compared to the single-threaded execution. DBMS D largely suffers from Icache stalls, and Shore-MT, DBMS N, and Silo mainly suffer from Dcache stalls. DBMS M, once again, suffers less from the Icache stalls than it does for the single-threaded execution. This is, once again, likely due to the instruction locality of the modules that are more active during the multi-threaded execution.

This section presents the consumed memory bandwidth for the sensitivity to data size and to work per transaction micro-benchmarks and TPC-C benchmark. We measured both single- and multi-threaded consumed memory bandwidth. We observed that the consumed single-threaded bandwidth is always less than 1 GB/s for all the systems. Hence, we omit the single-threaded bandwidth results, and focus on the multi-threaded ones.

In this section, we examine three popular acceleration features that today’s processors provide: hyper-threading, turbo-boost, and hardware prefetchers. We present normalized throughput values.

We use DBMS M and Silo when running the read-only micro-benchmark while probing 1 row per transaction, and when running the TPC-C benchmark for single- and multi-threaded executions. We use 10 GB of database for the micro-benchmark and 32 GB of database for TPC-C when profiling DBMS M. We use 100 GB of database for Silo.

In this section, we compare two Intel generations in terms of their micro-architectural behavior when running the micro-benchmark that randomly reads 1 row and the TPC-C benchmark. We use 100 GB of database. We use 10 GB of a database for the micro-benchmark and 32 GB of a database for TPC-C for DBMS M. We examine the Intel Xeon v2 line Ivy Bridge micro-architecture and Intel Xeon v4 line Broadwell micro-architecture. We choose these two generations as there is major micro-architectural change from the Ivy Bridge to the Haswell micro-architecture, especially in the instruction fetch unit of the processors [14] (see Sect. 4.1.1, paragraph 2). Broadwell is slightly improved version of the Haswell micro-architecture. As OLTP systems are known to severely suffer from Icache stalls, we examine how effective the instruction fetch unit improvement for OLTP systems.

Figures 15 and 16 show the results. We observe that there is a significant change in the micro-architectural behavior of the OLTP systems across the two processor generations. On the Ivy Bridge micro-architecture, all the systems except for Silo mainly suffer from Icache stalls. On the Broadwell micro-architecture, Shore-MT and DBMS N’s main micro-architectural bottlenecks shift from Icache stalls to Dcache stalls. Similarly, DBMS D and M suffer significantly less from the Icache stalls on the Broadwell micro-architecture compared to the Ivy Bridge micro-architecture. This shows that the advances in the instruction fetch unit of the processor significantly help for reducing the Icache stalls.

Our finding on Ivy Bridge vs. Broadwell corroborates the recent work of Yasin et al. [60] where SPEC benchmarks are evaluated across Ivy Bridge and Skylake (the generation after Broadwell) micro-architectures. Yasin et al. also show that the improvement on the Skylake micro-architecture, which inherits the improvements from the Broadwell micro-architecture, significantly reduces the Icache stalls.

Table 13 shows the throughput of the Broadwell machine normalized to the throughput of the Ivy Bridge machine. As expected, the Broadwell machine delivers significantly higher throughput than the Ivy Bridge machine thanks to its micro-architectural improvements.

Existing work on micro-architectural analysis of OLTP systems has mostly used an Ivy Bridge or an earlier micro-architecture generation. As a result, their conclusions were mostly referring to the instruction overheads of disk-based and in-memory OLTP systems [48, 49, 53, 54]. In this paper, we take the existing work one step ahead and provide conclusions on one of the latest generations of Intel micro-architectures.

This section summarizes the highlights of our work. In-memory OLTP systems implement a series of optimizations to reduce the instruction footprint and improve cache utilization. Despite all of the optimizations, they severely under-utilize the micro-architectural features similarly to the traditional disk-based systems.

Instruction stalls DBMS D and M incur the highest number of instruction stalls due to the large amount of legacy code they use. Hence, OLTP systems relying on legacy code modules should first optimize their instruction footprint. Transaction compilation is a promising technique that can be used to reduce the instruction footprint size and complexity of an OLTP system as shown by [34, 48] (see Sect. 6.1).

Shore-MT, despite being a disk-based system, does not suffer from instruction cache miss stalls. Hence, today’s processors are powerful enough to fetch and execute instruction stream of complex disk-based systems without stalling the instruction fetch unit. DBMS N and Silo do not suffer from the instruction stalls either. This shows that ground-up designed OLTP systems’ instruction footprint is simple enough for today’s processors to fetch instructions without stalling in the instruction cache (see Sect. 4.1.1).

Data stalls DBMS N and Silo mainly suffer from data cache stalls. DBMS M suffers from data stalls only when the instruction cache misses are mitigated by an increased instruction locality in their instruction stream (see Sect. 4.2). The data cache misses for DBMS N and Silo are mostly due to the random data accesses made during the index traversal. Hence, ground-up designed in-memory systems should firstly optimize their index structures. We have seen that Masstree [32] used by Silo significantly outperforms the red-black tree used by DBMS N (see Sect. 4.1.3, 4.2.1).

While using an efficient index structure improves the performance, its execution time is still dominated by data cache misses caused by the random data accesses during the index traversal. Hence, ground-up designed in-memory OLTP systems should adopt techniques that can mitigate the negative performance effect of random data accesses (see Sect. 4.2.1).

Mitigating data stalls One promising way to mitigate the random data accesses is using co-routines. Co-routines is a cheap thread interleaving mechanism that allows interleaving long-latency data stalls with computation. Psaropoulos et al. [38‐40] and Jonathan et al. [19] have shown that co-routines can successfully be used to improve index join and index lookup.

Another promising technique to mitigate the random data accesses is using machine learning to learn the distribution of the keys and jump to the index location that the key belongs to without actually performing the index traversal. Kraska et al. [25], Sirin et al. [30], and Ding et al. [11] have shown that machine-learned indexes can successfully replace/accelerate the index search operation.

Row- versus column-oriented storage All the systems we profile use the row-oriented storage format. However, recently, several systems adopted the column-oriented storage, mostly to use a single format for both transactional and analytical processing [36]. Using column-oriented storage as opposed to row-oriented storage requires making multiple random data accesses per row access, as different attributes of the same row will be spread around the main-memory.

Today’s processors are capable of overlapping random data accesses if their locations in the instruction stream are close to each other. Hence, making multiple random data accesses might not hurt the performance, if the data access primitives are well-implemented.

Alternatively, today’s processors have the software prefetching capability. Programs can prefetch those memory blocks ahead of the access such that the memory access times are overlapped with useful work. Developers can use software prefetching to mitigate the negative effect of making multiple random data accesses.

On the other hand, bringing multiple cache lines to the processor cache might result in inefficiently using the processor caches. If an attribute is 8 bytes and we access four attributes of the same row, we bring \(64\times 4=256\) bytes (as each cache line is 64 bytes), instead of \(8\times 4=32\) bytes. The \(256-32=224\) bytes of the data that is brought to the cache pollutes the cache, which can hurt the overall performance of the system.

Exhibiting low instruction- and memory-level parallelism, OLTP workloads are unable to utilize the wide-issue, complex out-of-order cores. Most of the time goes to the memory stalls for bringing either instructions or data items from the memory. Instruction cache sizes have been unchanged for the last decade due to the strict latency limitations, and we cannot expect them to increase. On the other hand, improved instruction fetch units can make a significant change in the micro-architectural behavior of the OLTP systems (see Sect. 10). Further advancements at the micro-architectural level, especially at the instruction fetch unit, can still have further potential impact to improve OLTP system performance, as popular OLTP systems such as DBMS D and M that we profiled still highly suffer from Icache stalls. Profile-guided (i.e., feedback-directed) optimization via the compiler and hardware support for software code prefetching are shown to be effective for reducing Icache stalls for server workloads, which also significantly suffer from Icache stalls [4, 8, 59].

As ground-up designed OLTP systems mainly suffer from long-latency data cache stalls, hardware designers can invest more on hardware mechanisms that can overlap long-latency data stalls. Hyper-threading improves performance up to 70% in a carefully designed and implemented in-memory OLTP system (see Sect. 9.1). Turbo-boost and hardware prefetchers are modestly useful for OLTP workloads, as OLTP workloads are memory-latency-bound (see Sects. 9.2, 9.3). As the processor’s power budget is limited, hardware designers can invest more power budget on the features that would overlap long-latency data stalls such as larger number of hyper-threads per physical core.

On the other hand, whatever the size of the last-level cache (LLC) is, megabytes of LLC will not be enough to keep the gigabytes of the data footprint of most standard OLTP benchmarks. Hence, instead of using beefy and complex out-of-order cores consuming large amount of energy, using simpler cores with intelligent hyper-threading mechanisms can improve the throughput of OLTP applications with a smaller power budget [12, 13, 31]. Sirin et al. [47] have shown that low-power ARM processor can provide 1.7 to 3 times lower throughput with 3 to 15 times less power consumption than a state-of-the-art Intel Xeon processor, achieving up to 9 times higher energy efficiency.

With that said, Kanev et al. [20] have shown that server workloads partially benefit from the wide-issue out-of-order execution. Hence, the use of wimpy cores with narrow-issue execution engines might produce a suboptimal performance and may not satisfy some application requirements. Similarly, Sirin et al. [47] have shown that ARM processors’ quantified latency can be up to 11 times higher than Intel Xeon towards the tail of the latency distribution, which makes Intel Xeon more suitable for tail-latency-critical applications.

GPU/FPGA-based acceleration of database systems is another line of research that allows improving database performance by using alternative computing devices to power-hungry processors. [9, 42, 45] present techniques on using GPUs for accelerating analytical processing queries such as hash join. Kim et al. [24] have proposed a transaction processing engine architecture that exploits the wide-parallelism. Alonso et al. [3] present an open-source hardware-software co-design platform for database systems, which uses CPU and FPGA as the main building blocks. [21, 44] present techniques on integrating FPGAs into common database operations such as data partitioning and regular expression. These studies highlight the opportunities to enrich the traditional computing space of database systems by alternative computing devices such as FPGAs and GPUs. As these computing devices provide massive parallelism and/or low-power consumption, they allow investigating the energy-efficiency space and potentially serve as the processors of the future database systems.

In this paper, we perform a detailed micro-architectural analysis of the in-memory OLTP systems contrasting them to the disk-based OLTP systems. Our study demonstrates that in-memory OLTP systems spend most of their time in stalls similarly to the disk-based OLTP systems despite all the design differences and lighter storage manager components of the memory-optimized systems. The lighter storage manager components reduce the instruction footprint at the storage manager layer, but the overall instruction footprint of an in-memory OLTP system is still large if the code base relies on legacy code modules. This leads to a poor instruction locality and high number of instruction cache misses. Ground-up designed in-memory OLTP systems can eliminate the instruction cache misses. In the absence of the instruction cache misses, the impact of long-latency data misses surfaces up, resulting in spending \(\sim \)70% of the execution time in stalls.