Which Category Is Better: Benchmarking Relational and Graph Database Management Systems

E.F. Codd introduced the relational data model with relations to represent data, with relational algebra and relational calculus to operate data, and relational integrity constraint to control the consistency and completeness of data. Since then, various RDBMSs have been developed with standard SQL to support data definition, data manipulation, and data control operations. The relational data model achieves great success to support a wide spectrum of applications that are related to financial, personnel, manufacturing, and logistical data management. Even until now, RDBMSs still remain as mainstreaming data management systems.

In recent years, graphs have been shown increasingly important to big data applications such as social network analysis, spatiotemporal analysis and navigation, and consumer analytics, as it is able to capture complex relationships and data dependencies. For example, in social networks, users, pictures, and events are modeled as vertices, and relationships between them are modeled as edges. So far, RDBMSs have been shown to be capable of dealing with graph processing and analysis. RDF-3X [1], Hexastore [2], and SW-store [3] are commonly used RDF stores based on RDBMSs to manage semantic Web ontology and RDF knowledge bases. They transform SPARQL [4] queries over RDF [5] data to SQL using sort-merge joins in the relational world, and various relational optimization techniques are utilized to speed up the query processing.

As another alternative solution, Neo4j [6] was proposed as a graph database management system based on the graph model to manage graph data, and many other graph database management systems including ArangoDB [7], gStore [8, 9] are efficient in processing SPARQL queries over RDF datasets. Titan has been designed for graph data management and analysis. As a typical application example, in finance service industry, behaviors (like account statement, calling list, loan history) of enterprises and customers are collected and modeled as graphs. Because GDBMSs such as Neo4j and ArangoDB often have rich graph algorithms, they are used to not only manage these behavior data, but also help do the analysis over these data using the algorithms, like personal or enterprise credit analysis.

Thus far, great controversies have been raised for the comparison between RDBMSs and GDBMSs: which category is better. On the one hand, from the perspective of GDBMSs, their advantage lies in schema-less property. They are able to manage structured, unstructured, or semi-structured data and thus are more flexible than RDBMSs. Given that GDBMSs are able to manage relational data, it is argued that RDBMSs may be replaced by GDBMSs. On the other hand, from the perspective of RDBMSs, RDBMSs have been proved to be capable of dealing with graph data management and analysis, and a few recent works have shown that by simply extending SQL languages in RDBMSs to support graph operations, the performance is comparable between RDBMSs and GDBMSs [10‐14], verifying this capacity of RDBMSs. Furthermore, due to the lack of a unified graph model and query language across GDBMSs, which often incurs an extra programming and maintenance overhead for users, the necessity of GDBMSs is disputed.

To clearly answer this question, we propose a unified benchmark for both RDBMSs and GDBMSs. Because RDBMSs and GDBMSs have different data models and different query languages, in this benchmark, we target to evaluate RDBMSs and GDBMSs over the same datasets using the same query workload under the same metrics. First, to address the issue that RDBMSs and GDBMSs have different data models, we propose a relation-to-graph mapping scheme, under which relational data are able to be transformed to graph data. In this way, we use TPC-H [15], which is a commonly accepted benchmark in RDBMSs, and extend it to evaluate GDBMSs. Similarly, we propose a graph-to-relation mapping scheme, under which graph data are able to be transformed to relational data. We use LDBC [16], which is commonly used in GDBMSs, and extend it to evaluate RDBMSs. Second, to address the issue that RDBMSs and GDBMSs have different query languages, we transform all 22 SQL queries in TPC-H to graph queries and transform all 5 graph queries in LDBC into SQL queries. Third, consider that GDMBSs can adopt different back-end storage engines, which could potentially affect the performance of GDMBSs. We then evaluate the GDBSMs using different storage engines and report our findings. We select Postgresql, a popular open-source RDBMS, as the representative of RDBMSs and two popular GDBMSs Neo4j, ArangoDB as the representatives. We conduct extensive experimental evaluations to compare GDBMSs and RDBMSs over TPC-H and LDBC under the metrics, including query processing time, memory utilization ratio, and CPU utilization ratio.

In summary, our contributions are as follows:The remainder of this paper is organized as follows: We review the related work in Sect. 2 and elaborate our unified benchmark in Sect. 3, following which we report experimental results and our findings in Sect. 4, before concluding the paper in Sect. 5.

Our work is related to both RDBMSs benchmark and GDBMSs benchmark.

RDBMSs Benchmark As the mainstream commercial database systems, there exist rich RDBMSs benchmarks. Among them, the most well accepted are the TPC series. TPC-C [17] is an online transaction processing (OLTP) benchmark, which involves a mix of five concurrent transactions of different types and complexity. It models order management and extracts the workload. TPC-C is mainly used to test the capacity of transaction processing. TPC-H is a decision support benchmark which models business procurement, whose datasets consist of 8 tables representing general business procedure. The 22 queries and the data populating the database have been designed to evaluate the capacity of handling critical business questions. TPC-DS [18] is also a decision support benchmark which models several generally applicable aspects of a decision support system, including 24 tables, and 99 randomly replaceable SQL queries. It focuses on emerging technologies, such as big data systems, to execute the benchmark.

GDBMSs Benchmark Unlike RDBMSs benchmarks which are proposed by authoritative organization, GDBMSs benchmarks come from some database companies. NoSQL performance benchmark [19] is proposed by ArangoDB [7], with its prime target to compare the performance among MongoDB, PostgreSQL, OrientDB, ArangoDB, and Neo4j under the metrics including read/write performance test, memory utilization ratio. GDBMSs benchmark [20] is created by TigerGraph [21], mainly evaluating the data loading and query performance of TigerGraph, Neo4j, Amazon Neptune [22], JanusGraph [23], and ArangoDB. The LDBC graph analytics benchmark [16] is an industrial-grade benchmark for graph analysis platforms. It consists of several typical graph algorithms, standard datasets, data generators, and reference outputs, enabling the objective comparison of graph analysis platforms.

As pointed out in existing literature [10], both RDBMSs and GDBMSs are shown to be capable of managing relational and graph data. Nevertheless, the boundary of using RDBMSs and GDBMSs still remains unclear, i.e., for RDBMSs and GDBMSs, which category should be properly used under a given application scenario. To cope with this issue, in this paper, we then propose a unified benchmark for both RDBMSs and GDBMSs.

In this section, we present the unified benchmark that is applicable for both RDBMSs and GDBMSs. The main idea can be described as follows:By doing this, we can evaluate RDBMSs and GDBMSs on the same datasets with the same query workloads under the same metrics.

In this section, we first introduce the experimental setup. We then conduct extensive analysis of GDBMSs and RDBMSs over the same datasets, using the same query workload, under the same metrics. We finally summarize our findings.

RDBMSs are ubiquitously used for storing, manipulating, and retrieving data over past decades. Meanwhile, the diversity and complexity of the applications led to challenges for RDBMSs to deal with sophisticated relationships between entities. For this reason, GDBMSs have lately received considerable attention which employs graph structures with vertices, edges, and properties to represent and store data, whereas both RDBMSs and GDBMSs are capable of managing relational data and graph data, making the boundaries of them unclear. To clearly answer the question, we have proposed a unified benchmark referring to existing benchmarks for RDBMSs and GDBMSs, which consist of relational workloads and graph algorithms, to evaluate databases on the same datasets under the same metrics. We have implemented an inter-transfer between SQL and graph query languages for querying and importing data in RDBMSs and GDBMSs, respectively. We have conducted extensive experimental evaluations for existing popular RDBMSs and GDBMSs over both standard TPC-H and LDBC and reported our findings in detail. Meanwhile, we have conducted further comparative experiments to find out the effect of different data models and back-end storage engines on query performance for GDBMSs. We can conclude that, shown in Tables 10 and 11, RDBMSs outperform GDMBSs by a substantial margin under the workloads which mainly consist of group by, sort, and aggregation operations, and their combinations. GDMBSs show their superiority under the workloads that mainly consist of multi-table join, pattern match, path identification, and their combinations. For GDBMSs with different storage engines, key-value storage engines with LSM-tree outperform for write operations, while native graph storage engine has an advantage for read operations. Row storage engines do not stand out in either read or write operations. As a future work, we will study the optimization on both data models and back-end storage engines.