What is in the KGQA Benchmark Datasets? Survey on Challenges in Datasets for Question Answering on Knowledge Graphs

The more ambiguous the question in a dataset, the harder it is to retrieve the correct answer. For our analysis, we examined several aspects of ambiguity:

As for the datasets, we do not have information which textual parts of the NL question refer to which part in the given SPARQL query (if any). Especially for the description of relationships, respectively, reference of ontology properties, this is a difficult task for a KGQA system.

For the analysis of the dataset, we therefore took a detailed look at the surface forms¹², the entity candidates and the respective SPARQL query. For each question, we took into account only specific POS tags (cf. Table 6) to identify the mentioned named entities and considered the following POS sequences: These, POS sequences have been detected for Steinmetz [10] from the most common POS sequences of DBpedia labels.

Here, N refers to any noun which can be a singular or mass noun (NN), a plural noun (NNS), singular proper noun (NNP), or a plural proper noun (NNPS). Each POS sequence might be followed by more nouns which are also taken into account as part of the mentioned entity. For each identified sequence in the question, we retrieve potential entity candidates from DBpedia. For the dictionary, disambiguation and redirect labels are utilized. Then, this extracted list of entity candidates for the complete question is compared to the entities contained in the provided SPARQL query¹³

In addition, we utilized the Falcon 2.0 API to identify surface forms and entity candidates in the questions. The API has been introduced by Sakor et al. [9] and proposes to identify entities and relations within short texts or questions over Wikidata and DBpedia. For the analysis, we requested the API with the following parameters:

Tables 7 and 8 show the result of our analysis. For each approach to identify the entities in the NL question, we detect the following measures, as shown in the table: As there is no DBpedia-based SPARQL query provided in WebQuestions and SimpleQuestions, the information about entities in SPARQL queries and if the most popular entity candidate is the correct one cannot be examined and is marked as not applicable (n/a) in the tables.

Our analysis shows, that ambiguity is a serious challenge for QA systems throughout all datasets. There are mentions of named entities having more than 100 entity candidates w.r.t. DBpedia.

For our approach, the most ambiguous term within all QALD datasets is Lincoln, having 215 entity candidates¹⁵ The most ambiguous term for all datasets is contained in the SimpleQuestions datasets: pilot with 479 entity candidates¹⁶

We also analyzed, how hard the disambiguation process for the detected entities would be. For this, we detected, if the required entity is the most popular among the list of candidates for the respective surface form. A disambiguation or ranking process can be considered more simple, if the NL questions always mention very popular entities with the respective surface forms.

However, our analysis shows that in many cases the relevant entity is not the most popular among the candidates of the list. The SDBQA datasets seem to be very hard to disambiguate, as we detected the lowest amount of entities that are the most popular for both datasets and both entity detection approaches. For the Falcon 2.0 API, the required entity is the most popular in only 31% of the cases. According to our analysis, the QALD 8 train dataset requires the least elaborate disambiguation process as up to 70% (for the Falcon 2.0 API, 67% for our approach) of the required entities are the most popular among the candidates.

Overall this means, a QA system must be able to disambiguate the mentioned entities—either using an answer ranking or according to the given context. Or, the system provides queries for all (or a subset of) relevant entities and provides the results to the user to receive feedback which entity and result is the demanded one.

In knowledge graphs, facts are described using subject, property, and object. Properties serve as descriptions of relationships between subject and object, whereas subject and object represent entities (resp. sometimes objects are literals). As natural language is very expressive, names for entities can vary and relationships can be phrased in many different ways. The lexical gap refers to missing links between an entity or relationship described in natural language and the labels available for that entity, a property or a class in the underlying knowledge base.

For the analysis of the extent of the lexical gap within the datasets, we used different approaches to detect entities and relations within the NL and compared the candidate lists with the entities and properties of the respective SPARQL queries. We count all entities and properties from the SPARQL query that are not found in the candidate lists from the NL question. We assume, for these entities/properties a lexical gap exists regarding labels and potential mentions in natural language.

In addition to our own approach and the Falcon 2.0 API to identify entities, we also utilized the Spotlight API¹⁷ For our approach and the Falcon 2.0 API, we considered all entity candidates for an identified surface form. In this way, we analyzed, if the relevant entity can be identified at all. Unfortunately, Spotlight API returns only the most relevant entity for the given context—not a candidate list. Therefore, we could only consider this one entity for the analysis. We compared the list of entities (candidates) with the list of entities extracted from the SPARQL query. The query entities not contained in the (candidate) list are summed up over the complete dataset.

Table 9 shows the results for the lexical gap analysis. The table contains the named entities extracted from the SPARQL query (Entities SPARQL), the number of entities from the SPARQL queries that were not found in the NL question (Entities not found) and the percentage of entities that were not found to the overall number of entities in the SPARQL queries (Percentage not found) (by our approach, Falcon 2.0 API and the Spotlight API).

We also analyzed the extent of the lexical gap for the required properties of the SPARQL queries. For this, we also utilized the Falcon 2.0 API. For the analysis, we extracted the DBpedia ontology properties from the SPARQL query¹⁸ We then compared this list to the list of the extracted relations by the Falcon 2.0 API from the NL question. We counted the properties that were not found by the API in proportion to the number of properties required for the SPARQL query.

Table 10 shows the amount of extracted properties from the SPARQL query (Properties SPARQL), and the results of the Falcon API for the extraction of relations (Relations Not Found). The number depicts the total number of properties not found by the API. The percentage depicts the proportion of relations not found compared to the number of required properties as extracted from the SPARQL query.

Regarding the analysis of the identification of the required entities in the NL question, the amount of entities not found is remarkably high. By our approach, a minimum of 25% of the entities contained in the SPARQL queries of the QALD datasets could not be found. For the QALD 8 test dataset, the percentage is as high as 46%. For instance, the question What was the university of the rugby player who coached the Stanford rugby teams during 1906–1917? requires the entity dbr:1906-17_Stanford_rugby_teams. For this, different parts of the question (and also numbers besides nouns) must be combined to find the label for this entity.

In comparison, the Spotlight API achieved even a lower rate of correct entities detected for most datasets (respectively, a higher percentage of entities not identified correctly from the NL question). The Spotlight API only returns the most likely entity for each identified surface form according to the given context. But, with the questions the context is meager and a disambiguation is apparently not successful in many cases. This experiment shows that the disambiguation process should not be considered before creating the SPARQL queries during the QA pipeline. A sample question where the API fails is Does the Toyota Verossa have the front engine design platform?. The required entities here are dbr:Toyota_Verossa and dbr:Front-engine_design. The API only detects the first one.

The Falcon 2.0 API performs similar or slightly better than our approach on the QALD datasets. The results for the LC-QuAD datasets are very good—only 9% of the entities are not among the candidates extracted by the API. In contrast, the API performs worse than our approach on the SDBQA datasets—the amount of entities that could not be identified is as high as 40%. A sample question where the Falcon 2.0 API fails to identify the relevant entities is Which computer scientist won an oscar?. Here, the required entities are dbr:Computer_Science and dbr:Academy_Award.

As Table 10 shows, the correct identification of DBpedia ontology properties is even harder than the entity identification.

The amount of properties not detected by the Falcon 2.0 API is remarkably high for all datasets, but especially for the SDBQA datasets with 73%. Mostly, this results from the fact that DBpedia facts and subgraphs are modeled along the ontology and not directly as expressed in natural language. For instance, the question Give me English actors starring in Lovesick. requires the properties dbo:country and dbo:birthPlace to create the English heritage of the requested actors. Obviously, these relations cannot be deduced from the NL alone. But, the API also fails to detect the property dbo:knownFor within the the question What is Elon Musk famous for?.

We provide our analysis results for properties not identified by the Falcon 2.0 API as JSON dataset¹⁹ Future mapping processes to identity alternative labels for DBpedia ontology properties might benefit from this dataset.

Our analyses show that the datasets contain a high number of questions where the correct entities and properties required for the SPARQL query cannot be detected by all of the approaches considered for our analyses. Furthermore, this means that for many questions the correct SPARQL query cannot be created using the correct entities which results in incorrect answers.

Apparently, the lexical gap is a significant challenge not only for mapping of relationship descriptions to ontology properties, but even for the identification of the correct entities mentioned in the NL question. But obviously, there are significant differences between the datasets.

The expressiveness of semantic knowledge bases is based on the rather simple data structure having facts stored as triples and the effective approach of using these graph patterns in the SPARQL query to access the knowledge. However, SPARQL supports several operators which might lead to rather complex queries. Obviously, more complex queries result from complex questions and are certainly a challenge for developers of KGQA systems.

For our analysis, we examined the datasets (that provide a SPARQL query) on the existence of the following query operators: FILTER, OFFSET, LIMIT, ORDER, GROUP, UNION, OPTIONAL, Subquery, HAVING, ASK type. Detailed information how often each operator occurs in each dataset is given in Table 11. As none of the datasets contain SPARQL queries with subqueries, we left that out in the table.

As another parameter for complexity, we also counted the maximum/minimum number of entities extracted from the SPARQL query and the maximum/average/median number of triples in the SPARQL query. The results are also shown in Table 12.

A further essential process step within KGQA systems is the identification of the correct focus in the NL question. The challenge here is to examine which part of the question is the subject of interest and how it relates to the rest of the question. For template-based KGQA systems, the graph patterns of the SPARQL query are constructed around this focus. Sequence-to-sequence systems can benefit from a preceding focus identification, as the trained model might make use of a preprocessed input question where the focus is tagged. In some cases, there are more than one focus to be identified in the question. Mostly, the focus(es) are represented by a named entity in the question which results in a resource URI in the SPARQL query.

To examine this aspect of complexity, we analyzed and compared the number of named entities in the NL question and the SPARQL query. If the NL question contains more than the SPARQL query, the process of focus identification is an essential step. If the numbers of entities in the NL question and the SPARQL query are equal, this might be a hint, that all entities found in the natural language can be adopted for the SPARQL query. If there are more entities in the SPARQL query than in the NL question, an analysis process might be required to deduce the additional entities from the focus(es) and relationships extracted from the linguistics of the question.

Table 13 shows the results for all datasets and contrasts the results for our approach and the results of Falcon 2.0 API. The table contains the information if more named entities have been found in the SPARQL query (More in SPARQL) compared to the NL question or in the NL question (More in NL) compared to the SPARQL query, or if the number of identified named entities are equal in the SPARQL query and the NL question.

Our analysis shows, that for all datasets the test datasets reflect the complexity of the training datasets or they contain even less complex queries, because sometimes operators are not present in the test dataset although they occurred in the training dataset. The HAVING and OFFSET operator is utilized only rarely. None of the SPARQL operators is present in any query of the LC-QuAD datasets. Only, the QALD datasets contain all operators to some extent. The SDBQA datasets naturally do not contain ASK queries or any other SPARQL operators other than UNION queries. The UNION queries are only utilized to model the SPARQL query with alternative properties—as described in Sect. 3.3.

For almost all datasets, the minimum number of named entities contained in the SPARQL queries is zero. An example for a question resulting in zero named entities in the SPARQL query is: Which actors have the last name “Affleck”?. Here, the query only asks for a specific type of entities that contain the string “Affleck” as object for a property lastName (i.e., foaf:lastName). Figure 2 shows this sample question and the resulting SPARQL query without a named entity in it.

Regarding maximum/minimum number of entities, the dataset QALD 8 test stands out among the others. All SPARQL queries comprise exactly one entity²⁰ which could be a hint that this dataset is a bit easier to process in terms of evaluation results than the others. This assumption can be confirmed by the analysis results shown in Table 11. QALD 8 test comprises only a few SPARQL operators and no ASK question. On the other hand, our analysis process was not able to find a relatively high number of entities (between 17 and 19 out of 41), as shown in Table 9.

In most cases, the number of entities detected in the NL question is higher than the number of entities extracted from the SPARQL query—as shown in Table 13. For our approach, this results from the detection process itself rather aiming at high recall than high precision. As described in Sect. 4.3, we detected entities in the NL question according to several POS sequences—all patterns include at least one noun. This procedure extracts all (combined) nouns from the question, even though it is not relevant as entity for the query. But, the Falcon 2.0 API also extracts in many cases too many entities, especially for the LC-QuAD datasets.

An example for a question having more named entities in the NL than in the SPARQL query is: How many gold medals did Michael Phelps win at the 2008 Olympics?. Here, both our algorithm and the Falcon 2.0 API detect Michael Phelps and 2008 Olympics as named entities, but the SPARQL query only asks for the gold medalist dbr:Michael_Phelps and filters the respective events for the strings “2008” and “Olympics”:

Nevertheless, the number of questions where there are more entities in the SPARQL query is also reasonable high for all datasets. In these cases, the additional entities must be deduced from the linguistics of the question or along the edges of the knowledge graph. An example for such a case that often occurs is the mistreatment of an apparent type information using a property and a resource in the SPARQL query. For instance, in many cases a type constraint is expressed in the way Which [ontology class name] was [...]?. So, for these cases the phrase following the word which must be used to identify the correct ontology class from the KG. But in some cases—specifically for the DBpedia—such class membership is modeled using a property. For instance, the question Which professional surfers were born in Australia? might ask for instances of the class dbo:Surfer. But, the given SPARQL query in the dataset models the fact using the property dbo:occupation and the resource dbr:Surfer:

This example shows, that an obvious class membership can also be modeled as relationship between entities. This circumstance must be taken into account, when transforming NL questions to SPARQL (for DBpedia).

As described in [6], template-based approaches try to identify patterns within the natural language and transform them to SPARQL query templates. The relevant parts of the templates are then mapped to the underlying knowledge base, and the complete query is created. Most approaches use linguistic and syntactic parsers to identify similar natural language patterns that lead to the same SPARQL query template. For the analysis of the datasets regarding templates, we followed the assumption that the amount of different patterns is limited. Of course, natural language can be very expressive (also depending on the language), but in terms of KGQA, we assumed that a SPARQL query template can only be deduced from a limited number of NL patterns. Therefore, we extracted the POS sequences of the NL questions and performed a normalization step.

Furthermore, templates can also be found regarding the SPARQL query. The query represents a subgraph of the complete knowledge graph. Depending on the different options how the subjects and objects of the triples are connected, different graph patterns are depicted. Therefore, we analyzed the SPARQL queries of the datasets in order to detect the amount of different graph patterns.

We retrieved the Part-of-Speech (POS) patterns for all questions of all datasets. Which means, we annotated a question with POS tags—utilizing the Stanford POS tagger—and retrieved the patterns by only using the tags in the order they occur in the question. Furthermore, we processed the POS sequences in terms of normalization. After the identification of named entities in the NL question²¹, we replaced all POS tags that belong to this entity with the placeholder RESOURCE. Consecutive RESOURCE occurrences are replaced by only one RESOURCE. In that way, the two questions (initially having different POS sequences): are linked to the same normalized POS sequence: WRB VBD RESOURCE VBN. After this normalization step, we counted the occurrences of the patterns in the datasets again. The numbers for the extracted (normalized) sequences are shown in Table 5.

In addition to the POS sequences of the NL questions, we also analyzed what type of subgraphs require to be constructed for the SPARQL queries. Therefore, we extracted the graph patterns and counted the occurrence of the patterns per dataset. For the extraction of the graphs, we considered the following principles: After extraction of all graphs from the queries, we analyzed the set of graphs for isomorphism and counted the occurrence of the graph patterns for each dataset.

As shown in Table 5, our assumption (of a limited number of POS sequences compared to different questions) is certainly rebutted by the numbers of different sequences we found within the datasets. Altogether, we found 56,844 different POS sequences (out of 171,487 different questions for all datasets) for the questions in English language. Noteworthy is the fact, that not one of these sequences occurs in all datasets.

The sequence with the most occurrences of 1,612 is WP VBZ DT NN IN NNP NNP. An example question for that sequence is Who is the owner of Universal Studios?. But it only occurs in 4 of the 26 datasets. The most frequent sequences in terms of different datasets are WP VBD NNP²², WP VBZ DT NN IN NNP²³, and WP VBZ DT NN IN NNP NNP²⁴ They all occur in 21 of the 26 datasets.

Utilizing the normalization step, the overall number of sequences is reduced to 50,455. But still, there is no normalized POS sequence that occurs in all datasets. The most frequent normalized sequence with 2601 occurrences is WP VBZ DT NN IN RESOURCE which also originates from questions like Who is the owner of Universal Studios? or What is the revenue of IBM?. This sequence only occurs in 4 of the 26 datasets. The most frequent sequence in terms of different datasets is WP VBD RESOURCE²⁵ This sequence occurs in 24 of the 26 datasets.

Obviously, the number of different POS sequences that must be taken into account might be limited, but on a very high level.

Overall, we identified 22 different graph patterns for QALD 1–9, LC-QuAD and SDBQA. The patterns are shown in Fig. 3.

In addition, we analyzed by how many different normalized POS sequences the graph patterns are represented within each dataset. The results for this analysis are shown in Table 14.

As already detected for the aspect of complex queries, the SPARQL query graphs require at most 5 triples. These few graph patterns are represented by two different subgraphs—graph IDs 15 and 22 in Fig. 3. These patterns are only contained in the QALD 8 (both) and QALD 9 train datasets. All other datasets contain 4 triples at most.

Within the QALD datasets, only 5 different graphs (graph IDs 1–5) are remarkably present, while the other graph patterns are only used sparsely or not at all. The LC-QuAD datasets contain 7 different graph patterns that are mainly used for the queries. Only one further pattern is used a few times. That means, LC-QuAD only utilizes 8 different patterns with 3 triples at most.

The difficulty to identify the correct formal query for a given NL question is also dependent on the specific domain of the question. For some domains (technical), terms are nearly unique. That means, the disambiguation task can be omitted and the mapping of surface forms to properties, classes and entities is straightforward. For other domains, these tasks might be much more difficult which hinders the overall task of question answering. Therefore, we analyzed the datasets for the ontology classes that inherit from the entities used in the SPARQL queries. These ontology classes give a hint about the domain of the question. For instance, the SPARQL query contains an entity of class dbo:Athlete²⁶ the question is most likely from the sports domain.

For the analysis, we extracted the entities of the given SPARQL queries and retrieved the respective ontology classes via rdf:type information of the DBpedia knowledge graph. We took into account all assigned classes along the class hierarchy of the DBpedia ontology. Table 15 shows the top 10 DBpedia ontology classes and their frequency belonging to named entities of the SPARQL queries distributed over all datasets. The table also lists the top 5 classes for each dataset group (train and test dataset together) separately.

DBpedia does not provide specific domain information for the resources, and most of the ontology classes are too general to hint a certain domain for the question. Therefore, a domain assignment of the datasets cannot be performed based on these results.

However, Table 15 lists for the SDBQA datasets 4 of 5 ontology classes (Film, Band, MusicalArtist, and Album) that might hint that the contained questions are mostly from the entertainment domain.

Recently, a challenge on answer type prediction has been published as part of the International Semantic Web Conference 2020 (ISWC)²⁷ The task of this challenge is to predict the answer type of the question according to the structure of the NL question. For instance, the question Who is the heaviest player of the Chicago Bulls? requires the answer to be of type dbo:BasketballPlayer or the question How many employees does IBM have? requires the answer to be of type xsd:integer.

For the analysis of the datasets regarding answer types, we defined 10 different types:

The QALD challenge provides a hint about the answer type in the datasets, but only for the latest editions. Also, the provided answer types are more general than the types we included for our survey. Therefore, we performed an analysis regarding answer types for all KGQA datasets. Some datasets provide the answers for each question as part of the dataset. In this case, we analyzed the answer type according to the provided answers. For some datasets, the answers are not provided: both LC-QuAD datasets, the SDBQA datasets, and some test datasets of the QALD challenge. In this case, we used the SPARQL query to retrieve the answers the respective DBpedia version. If we could not retrieve the answers, we further analyzed the question: If none of these analysis steps results on a proper answer type, the type is set to unknown. This applies for many of the LC-QuAD questions, because no results could be retrieved. Table 16 shows the results of our analysis. The table contains the overall numbers of the occurrences of the answer types we pre-defined.

We provide the results of our answer type analysis as a separate dataset. The dataset contains the id from the original dataset, the name of the source dataset, the question string and the detected answer type as a JSON file for each dataset file²⁸