KGAnet: a knowledge graph attention network for enhancing natural language inference

A CNN [23] is a neural network with powerful functionality in computer vision and natural language processing, because it can extract the spatial features of Euclidean-structured data. However, many data are non-Euclidean in structure, such as social networks and molecular structures. A graph convolutional network is used to process such data. The function of the graph convolutional network is to fuse the nodes and the surrounding information to obtain richer and more accurate representations of the nodes. In recent years, many researchers have conducted studies in the field of graph convolution, and its methods fall into two main categories: spectral methods and non-spectral methods.

Non-spectral methods The non-spectral methods [24] directly define the convolution on a graph using an adjacency matrix to summarize the node features of all spatial neighboring matrices. The main challenge of non-spectral methods comes from the dynamic size of the neighborhood. Duvenaud et al. [25] proposed a CNN that runs directly on the original molecular graphs and learns a specific weighted matrix for each node. To enable traditional CNNs to be directly used with graph input, they select fixed-size neighbors and normalize these nodes [26]. Recently, Hamilton et al. [27] introduced the task of inductive node classification with the aim of classifying nodes that have not been seen during training. This method takes a fixed-size neighborhood of the node and then uses a specific aggregator (such as the mean operator or LSTM [28]) to learn the neighborhood features. It has achieved impressive results on several large-scale inductive benchmark tests.

Spectral methods Bruna et al. [29] first proposed the definition of a graph convolution network in the Fourier domain. Because the convolution definition contains a Laplacian feature decomposition of the graph, the calculation process is quite arduous. Subsequently, a smooth parametric spectral filter [30], Chebyshev polynomial [31], and Cayley polynomial [32] were introduced to improve the computational efficiency. Then, Kipf and Welling [33] greatly simplified the convolution operation using a first-order Chebyshev polynomial and achieved good results on node classification. Although the above methods significantly improve the computational efficiency, they depend on the structure of the graph itself and cannot be directly used for graphs of different structures. A GAT [21] introduces an attention mechanism based on graph convolutional networks and calculates the weights of different nodes in the central node domain according to node similarity, which allows the model to accept inputs of different sizes.

However, in a knowledge graph, to enrich the entity expressions, it is obviously not enough to consider the similarity of entities. In contrast to general graph structure data, the more important a relationship is to the central entity, the more important the connected neighbor entities are. It is reasonable for the property of the relationship to participate in the calculation of weights between entities.

Story generation In the field of story generation, Jian et al. [34] aligned the entities that appear in the story and ConceptNet [35] and used Graph Attention to generate entity representations that enhanced the model’s understanding of the entire story.

Dialogue system Hao et al. proposed a CCM [36] (commonsense knowledge-aware conversational model). Given a user post, the model retrieves the relevant knowledge graph from ConceptNet and then uses the static graph attention mechanism to increase the semantic information of the post. Then, during the word generation process, the model reads the retrieved knowledge graph and produces better generation through the dynamic graph attention mechanism.

CommonsenseQA Bill et al. proposed a knowledge-aware graph network module named KagNet [37], a text reasoning framework for commonsense answering questions, which effectively utilizes ConceptNet to perform explainable reasoning.

Recommendation Xiong et al. termed the hybrid structure of knowledge graph and user–item graph as collaborative knowledge graph (CKG). They developed a knowledge graph attention network (KGAT) [38], which achieved high-order relation modeling in an explicit and end-to-end manner under the graph neural network.

One common feature of their works is they all used graph neural networks with edge information to inject external knowledge for their professional fields. Inspired by these works, in this paper, we try to use a similar structure to design a general way to introduce external knowledge for the NLI domain.

In the NLI field, a lot of works have been devoted to studying the effectiveness of external knowledge and how to introduce external knowledge.

KIM [20] was the first NLI model to introduce external knowledge. Chen et al. [20] enhanced NLI models with external knowledge in co-attention, local inference collection, and inference composition components. They first explored how to introduce external knowledge and proved the effectiveness of external knowledge in the NLI field.

Wang et al. [39] proposed a ConSeqNet system combining text and graph models: This system uses ConceptNet as an external knowledge source to solve natural language inference (NLI) problems. At the same time, they also compared the effects of three external knowledge (wordnet, DBPedia, and ConceptNet) and tried on the diversity of external knowledge.

Annervaz et al. [40] used attention to automatically align entities and words and used convolutional neural network-based methods to extract external knowledge to reduce the scale of the attention space. They demonstrated that model training with the help of external knowledge can converge with fewer labeled samples.

These works of introducing external knowledge in NLI practice were performing well. Unlike them, our work verifies the effectiveness and flexibility of graph convolutional networks in the NLI field.

For each entity, our aim is to enable the NLI model to fully access external knowledge, so we use a KGAnet to generate representation vectors made up of the entity’s neighbors and relationships. In the knowledge graph, we call the first-order neighbor entity j of an entity i a neighbor entity and we call i the core entity. When processing a knowledge graph, one layer of the KGAnet combines the information of the first-order subgraph into the representation of the core entity. Furthermore, k layer of a KGAnet combines the input features of the k-order subgraph into the representation. In this study, we only consider first-order neighbors; the effect of the k-layer neighbors will be considered in future work. The KGAnet inputs are a set of entity features \(h=\{ {\varvec{h}}_{1}, {\varvec{h}}_{2}, \ldots , {\varvec{h}}_{N}\}\) and a set of relationship features \(r = \{ {\varvec{r}}_{11}, {\varvec{r}}_{12}, \ldots , {\varvec{r}}_{NN} \}\). Specifically, \({\varvec{h}}_{i} \in {\mathbb {R}}^{F}\) is a representation of entity i, and \({\varvec{r}}_{ij} \in {\mathbb {R}}^{F}\) is the relationship between entities i and j. The output is a new set of entity features \(h^{'} = \{ {\varvec{h}}_1^{'} ,{\varvec{h}}_2^{'} ,\ldots ,{\varvec{h}}_N^{'} \}\), \({\varvec{h}}_i^{'} \in {\mathbb {R}}^{F^{'}}\), where N is the number of entities of graph, F is the number of dimensions of the input entity features, and we denote the output node feature dimension as \(F^{'}\).

First, we calculate the degree of importance of the neighboring entity with respect to the core entity. The weight calculation involves two parts: the importance of the relationships and the similarity between entities. We, respectively, define a relationship’s importance and entities similarity as follows:In these formulas, \(W_{r} \in {\mathbb {R}}^{F}, W_{h} \in {\mathbb {R}}^{F^{'}\times F}\) and \(W_{a} \in {\mathbb {R}}^{F^{'}}\) are linear transformations. We use \(W_{h}\) to filter entity features, \(W_{r}\) to filter relationship features, and \(W_{a}\) to calculate the correlation of entity features. We use \(I_{ij}\) to denote a score function that determines the importance of the relationship \({\varvec{r}}_{ij}\) and use \(S_{ij}\) to denote a score function for the similarity between core entity i and entity j. When both \(I_{ij}\) and \(S_{ij}\) are small, we want to get a small value of \(E_{ij}\). When \(I_{ij}\) and \(S_{ij}\) are both large, we want the value of \(E_{ij}\) to be large. So we multiply the two scores to get the weights \(E_{ij}\) between the entities as follows:There is no I involved in the calculation in GAT [21]. The entity i and its neighbor entity j form a subgraph. Entity i and entity j in \(E_{ij}\) may not be associated with each other. We need the adjacency matrix A of the graph to act as a mask, erasing the relationships that do not exist in \(E_{ij}\). The specific operation is:\(A_{ij}\) is the adjacency matrix of the graph. If there is an relationship between entity i and entity j, the value of \(A_{ij}\) is 1; otherwise, the value is 0. When \(A_{ij}\) is 0, the value of \(E_{ij}\) is negative infinity to obtain a value close to 0 after normalization.

To make the weights of all neighbor entities of i easy to compare, we normalize \(E_{ij}\) as follows:After \(\alpha _{ij}\) obtained, we use it to sum over all neighbor entities according to importance as follows:where \({\mathcal {N}}_{i}\) is some neighborhood of node i in the graph and \(M_{j}\) is the neighbor feature after filtering. The new features \(\varvec{h_{i}^{'}}\) of entity i fuse the information about its surrounding entities and relationships. \(\sigma\) is a sigmoid function. During the training process, parameters \(W_{r}\), \(W_{a}\), and \(W_{h}\) are all trained: \(W_{r}\) and \(W_{a}\) learn how to provide a reasonable score for the relationship importance and entities similarity and \(W_{h}\) acquires the ability to select beneficial features from neighbor entities.

At present, for inter-sentence methods, cross-attention is widely used in NLI models and is regarded as the basic component of such models. Note that cross-attention can align words that are related between sentences. If the input word vector contains an association, then this association feature will be mined in the cross-attention. Below, we will briefly introduce the basic framework of all current inter-sentence models and mainly describe what the cross-attention is and its function.

Figure 2 shows a typical inter-sentence model structure. The premise \(a=\{{w_{1}}^{a},{w_{2}}^{a},\ldots ,{w_{l_{a}}}^{a}\}\) and hypothesis \(b = \{{w_{1}}^{b},{w_{2}}^{b},\ldots ,{w_{l_{b}}}^{b}\}\) are the input of the model. We denote the ith word in the premise as \({w_{i}}^{a}\) and the jth word in the hypothesis as \({w_{j}}^{b}\).

Cross-attention finds the degree of association of each word in two sentences. Using the degree of association, each sentence uses the other sentence’s word information to enhance its own word representation, completing a soft alignment of the two sentences. In this part, the relationships of the entities are extracted to enhance the interaction of the two sentences. In the cross-attention layer, we can calculate the similarity of words between sentences using the following equation:Then, we need to align the similar words in the two sentences as follows:Next, we can perform down-sampling on each sentence through max pooling and average pooling to obtain vectors \(V_{p}\) and \(V_{h}\) as follows:We concatenate \(V_{a}\), \(V_{b}\), their difference, and their element-wise product into a vector V, as shown in Eq. 16. Finally, we feed V into a multilayer perceptron classifier (MLP). MLP includes two layers of feedforward neural network, which have \(Hidden\_Size\) and 3 neurons, respectively. The activation function of the last layer is softmax, and the output is the relationship between the two sentences.

To obtain external knowledge, it is convenient to concatenate the output of KGAnet and the input of NLI and jointly train them. The combined method is shown in Fig. 3. The vector \(\varvec{Q_{k}} \in {\mathbb {R}}^Q\) of each word is composed of a pre-trained D-dimensional word vector \(\varvec{w_{k}}\) and a new entity feature \(\varvec{h_{i}^{'}}\) obtained by KGAnet, as shown in Eq. 17, where \(W_{q}\) is a linear transformation.Considering that not every word will find related entities in the graph, so some words in the sentence will correspond to empty entities. For these entities, we assign the following values:After \(\varvec{Q_k}\) is obtained, it is used as the new input data for the NLI model and is passed to various encoders. For instance, in DecAtt [13], encoder is a fully connected layer. In ESIM [16] and BiMPM [17], encoder is bidirectional LSTM [28].

In the experiment, all of our models use the Adam optimizer [41]. The word vector is a pre-trained GloVe [42]³ 300-dimensional vector. For words that are out-of-vocabulary, we assign them a 300-dimensional zero vector. Table 1 shows the hyper-parameters used for all experiments.

We design an experiment with three hyperparameters (\(Batch\_Size\), \(Hidden\_Size\), \(Learing\_Rate\)) of ESIM+KGAnet. The experimental results are shown in Fig 7 in “Appendix.” \(Hidden\_Size\) is the number of neurons in the first layer of the multilayer perceptron classifier (MLP) in Sect. 3.2. The experimental results show that: (1) the update step is too large when \(Learing\_Rate\) is greater than 0.001, so our model fail to converge; (2) when \(Batch\_Size\) is less than 32, the optimizer’s inaccurate estimate of the gradient results in poor model performance; and (3) \(Hidden\_Size\) has little effect on results. Generally speaking, when the learning rate is not large and the batch size is not small, the robustness of the model can be guaranteed.

In the experiment, the performances of all models were evaluated on the SNLI dataset [1] and the MultiNLI corpus [2]. In the SNLI dataset, there are three possible relationships between a premise and hypothesis: contradiction, neutral, and entailment. This dataset consists of a 549,367-sample training set, 9842-sample development set, and a 9842-sample test set. The MultiNLI dataset consists of a 392,703-sample training set. The test and development sets are split into 10,001-sample in-domain (matched) and 10,001-sample cross-domain (mismatched) sets.

To closely retain the scale of the knowledge graph used in the previous work [20], we chose WordNet [19] to provide the knowledge graph data, and we preprocessed the knowledge graph as follows: (1) we selected 14,216 entities including entities in wordnet that appear in the snli and mnli training sets and their neighbors. (2) We selected six relationships in WordNet: entailment, member meronymy, synonymy, antonymy, hypernymy, and similarity. TransE [43] is a graph embedding technology that we used to embed the 14,216 entities and their relationships as 300-dimensional vectors.

In view of the problems we attempt to address in Sect. 1, we evaluated two aspects of our approach: sufficiency and applicability. The sufficiency is demonstrated by comparing the testing accuracies and training processes of KGAnet and KIM [20], which is the previous method for acquiring external knowledge. The applicability is reflected by the improvements obtained by adding KGAnet to other NLI models without external knowledge.

In Sect. 3.1, we considered the importance of entity relationships based on GAT when we designed KGAnet.

Therefore, we individually added GAT and KGAnet to the same base model (ESIM) to verify whether the relationship is important. The comparison results are shown in Tables 4 and 5. The results show that KGAnet yields a substantially larger improvement than GAT.

Moreover, Fig. 6 shows the attention weights of the three models for an example from the SNLI test set. ESIM + KGAnet successfully predicts it, whereas ESIM + GAT and ESIM fail in this instance. The attention weights map shows the degree of association between the words from the premise and hypothesis in the NLI model. The word-pairs corresponding to a higher degree of association are indicated by darker cells. Whether it is necessary to consider the importance of the relationship can be determined by observing the difference between the attention maps. We hence have the following findings:

First In Fig. 6b, c, man is matched with male, but in Fig. 6a, they are not related. We found in WordNet that man and male have the common neighbor lover. This indicates that GAT and KGAnet capture not only direct relationships, but also the relationships between two indirectly connected nodes.

Second There are no direct or indirect relationships between \(\left\langle is, uses \right\rangle\), \(\left\langle is, laying \right\rangle\), \(\left\langle lying, laying \right\rangle\), \(\left\langle sideways, laying \right\rangle\), and \(\left\langle round, horizontal \right\rangle\) in WordNet, which indicates that in Fig. 6b, GAT produces a lot of noise. In contrast, in Fig. 6c, we can see that only the word-pair \(\left\langle lying, laying \right\rangle\) exists. We believe it is probable that the relationship importance function in KGAnet filters information about some unimportant neighbor entities in the subgraph.

Third The orange outline indicates the contribution \(\left\langle round, circular\right\rangle\), which has an association only in Fig. 6c. This demonstrates that in KGAnet, the relationship synonymy between \(\left\langle round, circular\right\rangle\) is indeed added to the calculation and has a certain influence on the word-pair relationships.

In summary, KGAnet reduces the noise, strengthens the association of words-pairs, and greatly improves the effect of NLI model because it considers the importance of the relationship.