A comparative survey of recent natural language interfaces for databases

Living in a digital age, the global amount of data generated is increasing rapidly. A good part of this data is (semi-)structured and stored in some kind of database, which is usually accessed using query languages such as SQL or SPARQL. Structured query languages, however, are difficult to understand for non-experts. Even though SQL was initially developed to be used by business people, reality shows that even technically skilled users often experience problems putting together correct queries [8], because the user is required to know the exact schema of the databases, the roles of various entities in the query and the precise join paths to be followed. Non-technical (or casual) users are usually overwhelmed by the technical hurdles of formal query languages.

One often-mentioned approach to facilitate database querying even for casual users is the use of natural language interfaces (NLI) for databases. These allow users to access information stored in databases by typing questions expressed in natural language [24]. Some NLIs restrict the use of the natural language to a sub-language of the domain or to a natural language restricted (and sometimes controlled) by grammatical constraints. For example, SODA [6] is based on English keywords, ATHENA [48] handles full sentences in English, and Ginseng [5] strictly guides the user to a correct full sentence in English. If users want to find all movies with the actor Brad Pitt, for instance, the input question in the different systems could be as follows:Depending on the database schema, the corresponding SQL statement might be as follows:

As we can see in this example, the users of NLIs do not have to know the underlying structure or query language to formulate the question in English.

Critiques of NLIs often highlight that natural language is claimed to be too verbose and too ambiguous. If it were possible to identify the different types of linguistic problems, then the system could support the user better, for example, with a clarification dialog. This would not only help the NLI to translate the natural language question into a formal query language, but would also assist the users in formulating correct queries. For example, if the users ask a question like ‘all movies from Columbia Pictures or 20th Century Fox,’ the system should ask for clarification of the ambiguous token ‘Columbia’ which can be either a location or a movie company (as part of the bi-gram ‘Columbia Pictures’). If the users choose movie company, then the system could directly suggest that ‘20th Century Fox’ is also a movie company and not the concept for films from 1900 until 1999.

Additionally, natural language is not only ambiguous on word level, but there can be also multiple interpretations of the meaning of a sentence. In the example input question ‘all cinemas in cities with parking lots’ the constraint ‘with parking lots’ can either refer to the cinemas or to the cities. Both interpretations are grammatically correct, but probably the intent is to find cinemas with a parking area. In some cases, like ‘all cinemas in cities with love seats’ there is only one interpretation valid on the data set: only cinemas can have a constraint on ‘love seats,’ even if it would be grammatically correct for cities. Therefore, an NLI should first check if multiple interpretations exist. Next, it should verify if they can be applied on the dataset. If there are still multiple interpretations left, the users should have the possibility to choose the most relevant one.

The above-mentioned examples are only some of the linguistic problems that NLIs must deal with. Furthermore, users are not perfect and tend to make mistakes. These range from spelling errors to the use of colloquial language, which includes syntactically ill-formed input. These examples do, however, highlight that precisely understanding the expressiveness of questions that an NLI is able to interpret is of paramount importance both for developers and users of NLIs. To address this need, this paper makes the following contributions:To help the reader understand the differences between current NLIs, in Sect. 2, we first describe a sample world, which we use as an example running consistently through the paper, in order to evaluate and benchmark the systems. This is different to previous surveys on NLIs [20, 37, 43, 44, 49], where the systems are only summarized or categorized. Based on our sample world, we introduce ten representative sample questions of increasing expressive complexity and illustrate their representativeness using question-answering corpora. These questions are used to compare the NLIs and highlight the various difficulties the NLIs face. In Sect. 3, we give a brief overview of the most important technology for natural language processing and discuss the limitations of our evaluation in Sect. 4. The sample world and the questions are then used in Sect. 5 to explain the process of translating from natural language questions to a formal query language for each NLI surveyed. We provide a systematic analysis of the major systems used in academia (Sect. 6.1) as well as three major commercial systems (Sect. 6.2). In Sect. 7, we discuss newer developments of NLIs based on machine learning. Note that these systems are not the main focus of the paper but serve more as a brief insight into new research avenues. Finally, we derive lessons learned that are vital for designing NLIs that can answer a wide range of user questions (Sect. 8).

Note that the detailed evaluation of the systems in Sect. 6 omits the approaches based on machine learning. The rationale for this is twofold. First, the functionality of the machine learning-based approaches is heavily dependent on the training data. Most papers present a system trained with some dataset and then show the capabilities of the system in the context of that training. We found no exploration of the general capabilities or robustness of a given approach when varying the input data. Hence, but is difficult to say if these systems could cover all requirements when given suitable training data or not. Second, little is known how domain-dependent those systems are on the training data. Usually, they require a lot of training examples making the comparison to the other systems—that mainly require some metadata—difficult. We, hence, concluded that the categorization of the capability of these systems is an open problem that needs to be addressed in its own survey.

In this section, we present a small sample world that serves as the basis for analyzing different NLIs. We first introduce the database schema (Sect. 2.1) and afterward discuss ten input questions of increasing complexity that pose different challenges to NLIs (Sect. 2.2). Finally, we will perform an analysis of different question-answering corpora to better understand what types of questions real users pose and how we can map them to our ten input questions. Our analysis in Sect. 2.3 indicates that our ten sample questions represent a large range of questions typically queried in question-answering systems.

This survey’s evaluation focuses on the ten sample questions introduced in Sect. 2.2. Those sample questions are based on the operators of the formal query languages SQL and SPARQL. This leads to the following limitations of the evaluation.

Limitation 1—Theoretical Our evaluation is theoretical and only based on the papers. A few systems have online demos (e.g., SPARKLIS [19]), but others—especially older ones—are not available any more. Therefore, to handle all systems equally, we based our evaluation fully on the papers of the systems.

Our approach is based on the assumption that the papers contain enough information in order to reproduce the reported experimental results. Hence, when analyzing the papers against our ten queries of increasing complexity, we explicitly checked if these papers contain hints about being able to answer these specific queries, if their architecture is in principle able to answer them or if the papers discuss explicit constraints about not being able to answer the respective queries. In short, we performed a very systematic approach of checking each of the 24 papers based on these ten queries. One potential limitation of our approach is that the actual implementation of the paper could differ from the description in the paper, and hence some of the results might be incorrect.

Limitation 2—Computational performance We completely ignored the computational performance of the systems. This limitation is based on the different focuses of the systems. For example, Walter et al. [58] (BELA) propose an approach to increase the speed and computational efficiency. Zheng et al. [69] (NLQ/A) and Zenz et al. [68] (QUICK) propose algorithms to optimize the number of user interactions for solving ambiguity problems.

Limitation 3—Accuracy In our evaluation, we ignored the accuracy given in the papers. There are multiple reasons for this decision: (a) Depending on the focus of the system, not all papers include evaluations based on accuracy. (b) Different usages of the same metrics, for example, Saha et al. [48] (ATHENA) only consider questions where the system could calculate an answer, but Damljanovic et al. [12] (FREyA) evaluate the system against the same dataset but based on needed user interactions. (c) The number of questions in different datasets can be highly different, for example, GeoData250 consists of 250 questions, Task 2 of QALD⁴-4 only of 25. (d) Even if two systems are using the same dataset, two system can preform different preprocessing, for example, Kaufmann et al. [31] (QUERIX) reduces the GeoData880 to syntactic patterns instead of all questions. All those factors make it impossible to directly compare the systems based on the given evaluation metrics in the paper.

In this section, we will focus on more recent NLIs starting from 2005. We will not discuss about older systems like BASEBALL [21], LUNAR [64], RENDEZVOUS [9], LADDER [47], Chat-80 [61], ASK [57] and JANUS [62], which are often quoted in the research field of NLIs. We will systematically analyze 24 recently developed systems in Sects. 5.1–5.4 based on the sample world introduced in Sect. 2. The main goal is to highlight strengths and weaknesses of the different approaches based on a particular data model and particular questions to be able to directly compare the systems. The explanation is based on the information describing the systems found in the papers. Finally, in Sect. 6.1, we will evaluate the systems against the sample questions and give an overall interpretation of the system.

There are different ways to classify NLIs. In this survey, we divide the NLIs into four main groups based on the technical approach they use:Table 2 gives an overview of the most representative NLIs of these four categories that we will discuss in Sects. 5.1–5.4. The table also shows which kind of query languages the systems support (e.g., SQL) as well as which NLP technologies are used. In summary, our approach of systematically evaluating these systems on a sample world with queries of increasing complexity enables a better comparison of the different approaches.

In the next subsections, we will systematically analyze the systems in more detail.

In this section, we first evaluate the 24 recently developed NLIs which were summarized before and systematically analyze if they are able to handle the ten sample questions of increasing complexity. This approach enables us to directly compare these systems which was previously not possible due to different datasets or evaluation measures used in the original papers. Afterward, we evaluate three commercial systems by asking them the same sample questions and analyzing their responses.

In current research, more and more systems include machine learning (ML) in their translation process (e.g., MEANS) or ranking (e.g., Aqqu).

KBQA [10] learns templates as a kind of question representation. It supports binary factoid questions as well as complex questions which are composed of binary factoid questions. OQA [16] approaches to leverage both curated and extracted KBs, by mining millions of rules from an unlabeled question corpus and across multiple KBs.

AMUSE [22] uses ML to determine the most probable meaning of a question and can handle different languages at the same time. Xser [65] divides the translation task into a KB-independent and a KB-related task. In the KB-independent task, they are developing a Directed Acyclic Graph parser to capture the structure of the query intentions and trained it on human-labeled data.

A new promising avenue of research is to use deep learning techniques as the foundation for NLIDBs. The basic idea is to formulate the translation of natural language (NL) to SQL as an end-to-end machine translation problem [13, 28, 54]. The approach is often called neural semantic parsing [60]. In other words, translating from NL to SQL can be formulated as a supervised machine learning problem on pairs of natural language and SQL queries. In particular, machine translation can be modeled as a sequence-to-sequence problem where the input sequence is represented by the words (tokens) of the NL and the output sequence by the tokens of SQL. The main goal is given an input sequence of tokens, predict the output sequence based on observed patterns in the past.

The main advantage of machine learning-based approaches over traditional NLIDBs is that they support a richer linguistic variability in query expressions, and thus users can formulate queries with greater flexibility. However, one of the major challenges of supervised machine learning approaches is that they require a large training data set in order to achieve good accuracy on the translation task (see further details below).

The most commonly used approach for sequence-to-sequence modeling is based on recurrent neural networks (RNNs, [15]) with an input encoder and an output decoder. The encoder and decoder are implemented as bi-directional LSTMs (Long Short Term Memory) by Hochreiter and Schmidhuber [25]. However, before an NL can be encoded, the tokens need to be represented as a vector that in turn can be processed by the neural network. A widely used approach is to use word embeddings where the vocabulary of the NL is mapped to a high-dimensional vector [45]. In order to improve the accuracy of the translation, attention models are often applied [39].

One of the currently most advanced neural machine translation systems was introduced by Iyer et al. [26]. Their approach uses an encoder–decoder model with global attention similar to Luong et al. [39] and a bi-directional LSTM network to encode the input tokens. In order to improve the translation accuracy from NL to SQL compared to traditional machine translation approaches, the database schema is taken into account. Moreover, external paraphrases are used to increase the linguistic variance of the training data. In the first step of training the neural machine translation system, the process of generating training data is bootstrapped by manually handcrafted templates for the NL and SQL query pairs. In the next phase, the training set is extended by adding linguistic variations of the input questions and parts to the query are replaced with synonyms or paraphrases of the query. The advantage of this approach is that it is query language independent and could in principle also be used to translate from NL to SPARQL. However, the disadvantage is that a large, manually handcrafted a training set is necessary. The authors reported their results using two different data sets with a training data size of 600 and 4,473 utterances, respectively.

Zhong et al. [70] introduce a system called Seq2SQL. Their approach uses a deep neural network architecture with reinforcement learning to translate from NL to SQL. The authors released WikiSQL—a new data set based on Wikipedia consisting of 24,241 tables and 80,654 hand-annotated NL-SQL-pairs. However, their approach was only demonstrated to work on simple single-table queries without joins. SQLNet by Xu et al. [66] uses a more traditional machine translation approach without reinforcement learning. However, even though SQLNet shows better accuracy than Seq2SQL, the experiments are also based on the WikiSQL data set and it is not clear how this approach would handle join queries against more realistic database settings with multiple tables. Finally, Yavuz et al. [67] show further improvements over SQLNet by incorporating both the information about the database schema as well as the base data. The paper in particular focuses on the generation of WHERE-clauses, which the authors identified as major problem of the relative low accuracy of SQL queries generated by Seq2SQL and SQLNet.

DBPal by Basik et al. [1] overcomes shortcomings of manually labeling large training data sets by synthetically generating a training set that only requires minimal annotations in the database. Similar to Iyer et al. [26], DBPal uses the database schema and query templates to describe NL/SQL-pairs. Moreover, inspired by Wang et al. [60], DBPal augments an initial set of NL queries using a paraphrasing approach based on a paraphrase dictionary. The results show that on a single-table data set DPal performs better than the semantic parsing approach of Iyer et al. [26]. However, for a multi-table data set with join queries, DBPal performs worse. The reason is that the limited training set of 600 examples does not seem to generalize well for join queries. The authors attributed the good performance of the system introduced by Iyer et al. [26] to overfitting.

Soru et al. [53] use neural machine translation approach similar to Iyer et al. [26]. However, the major difference is that they translate natural language to SPARQL rather than to SQL. Moreover, they do not apply an attention mechanism. In a subsequent white paper, Hartmann et al. [23] present an approach to automatically generate a large set of training data consisting of 894,499 training examples based on set of 5000 natural language queries. The approach is very similar to the approach used by DBPal.

In general, these new approaches show promising results, but they have either only been demonstrated to work for single-table data sets or require large amounts training data. Hence, the practical usage in realistic database settings still needs to be shown.

Another interesting new trend is in the area of conversational systems such as Williams et al. [63], John et al. [29] that often apply neural machine translation techniques. However, a detailed discussion on these systems is beyond the scope of this paper.

In this paper, we summarized 24 recently developed natural language interfaces for databases. Each of them was evaluated using ten sample questions to show their strengths and weaknesses. Based on this evaluation, we identified the following lessons learned that are relevant for the development of NLIs.

Lesson 1—Use distinct mechanisms for handling simple versus complex questions Users like to pose questions differently depending on the complexity of the question. Simple questions will often be asked with keywords, while complex questions are posed in grammatically correct sentences. For example, if users search for information about Brad Pitt, it is more likely that the users ask ‘Brad Pitt’ as an input question and not ‘Give me information about Brad Pitt.’ This lesson is highlighted in the paper by Waltinger et al. [59] (USI Answer) and related to the finding that casual users are more at ease at posing complex questions in natural language than in other formats [30]. Generally, pattern-based systems are solving this problem, but are often limited in the understanding of complex questions (i.e., subqueries). Walter et al. [58] (BELA) propose a layered system which could also be used to solve this problem of distinction. Grammar-based systems could be taught to support such non-natural language question patterns when included in the grammar. To the best of our knowledge, none of the systems we looked at supports this feature.

Lesson 2—Identifying subqueries still is a significant challenge for NLIs The identification of subqueries seems to be one of the most difficult problems for NLIs. The sample input questions Q9 and Q10 are two examples for questions composed of one and multiple subqueries, respectively. The most common NLP technology that is able to solve this problem is a parse tree. This can either be a general dependency or constituency parse tree provided, for example, by the Stanford Parser (e.g., NaLIR), or a parse tree self-learned with the rules of a grammar-based NLI (e.g., SQUALL). An alternative is the use of templates (e.g., AskNow). Li and Jagadish [35] (NaLIR) mention that the identification alone is not enough: after the identification of the subqueries, the necessary information needs to be propagated to each part of the subquery.

Lesson 3—Optimize the number of user interactions Natural language is ambiguous and even in a human-to-human conversation ambiguities occur, which are then clarified with questions. The same applies to NLIs: when an ambiguity occurs, the system needs to clarify with the user. This interaction should be optimized in such a way that the number of needed user interactions is minimized. To address this issue, Zenz et al. [68] (QUICK) and Zheng et al. [69] (NLQ/A) developed a minimization algorithm. The idea is to identify the ambiguity which has the most impact on the other ambiguities and clarify it first.

Lesson 4—Use grammar for guidance The biggest advantage of grammar-based NLIs is that they can use their grammar rules to guide the users while they are typing their questions. This improves the interaction between system and users in two ways: first, the system will understand each question the users ask; second, the users will learn how certain questions have to be asked to receive a good result. For the second part, Li et al. [38] (NaLIX) propose a solution by using a question historization and templates. This also helps the users understand what type of questions the system can understand and how the users need to ask.

New NLIs based on neural machine translation show promising results. However, the practical usage in realistic database settings still needs to be shown. Using a hybrid approach of traditional NLIs that are enhanced by neural machine translation might be a good approach for the future. Traditional approaches would guarantee better accuracy while neural machine translation approaches would increase the robustness to language variability.

In summary, our evaluation of various systems against ten sample questions shows that NLIs have made significant progress over the last decade. However, our lessons also show that more research is required to increase the expressive power of NLIs. This paper gives a guide for researchers and practitioners highlighting the strengths and weaknesses of current approaches as well as helps them design and implement NLIs that are not only of theoretical value but have impact in industry.