Formulaic language identification model based on GCN fusing associated information

Feature selection

The most significant difference between formulaic language and general multi-word expression is that the structure of formulaic language is mostly fixed, which is often in the form of Verb-noun or subject–predicate–object. Therefore, part-of-speech features are regarded as one of the features to identify formulaic language. First, we use the Stanford part-of-speech tagger to analyze the text. The examples of part-of-speech analysis results are shown in Table 1. From the table, it can be found that multi-word units with fixed sentence patterns are more likely to be formulaic language. Then we assign a unique code to each result after part-of-speech tagging, so as to convert the text data into vectors. Finally, we input them into the embedding layer for training to generate word embedding vectors as part-of-speech features.

In addition, for text processing, characters, words, phrases more reflect the lexical information of the text rather than its semantic information, so they can not accurately express the content of the text. For formulaic language, it is a multi-word unit with high frequency and has complete structure, meaning and function, so semantic feature is also an essential feature of formulaic language. This article uses the feature vector trained by GloVe to represent the semantic features of formulaic language.

GloVe’s full name is Global Vectors for Word Representation. It is a word representation tool based on global word frequency statistics. It can express a word as a vector composed of real numbers. These vectors capture some semantic characteristics between words (Pennington, Socher & Manning, 2014). Its implementation is divided into the following three steps:

(1) A co-occurrence matrix X is constructed according to the corpus. Each element X_ij in the matrix represents the number of times that word i and context word j appear together in a context window of a specific size. Generally speaking, the minimum unit of this number is 1, but GloVe doesn’t think so: according to the distance d between two words in the context window, it proposes a decreasing weighting function: decay = 1/d to calculate the weight. That is, the farther the distance, the smaller the weight of the two words in the total count.

(2) Construct the approximate relationship between Word Vector and Co-occurrence Matrix, as shown in Eq. (1). (1)${\displaystyle w_i^T\widetilde{w_j}+b_i+\widetilde{b_j}=log\left(X_{ij}\right).}$

Among them, the $w_i^T$ and $\widetilde{w_j}$ in the above formula are the word vectors we finally need to solve; and b_i and $\widetilde{b_j}$ are the bias terms of the two-word vectors.

(3) Construct the loss function, as shown in Eq. (2). (2)${\displaystyle J=\sum _{i,j=1}^{V}f\left(X_{ij}\right){\left(w_i^T\widetilde{w_j}+b_i+\widetilde{b_j}-log\left(X_{ij}\right)\right)}^2.}$

Among them, $f\left(X_{ij}\right)$ is the weight function. Its calculation formula is shown in Eq. (3). (3)${\displaystyle f\left(x\right)=\left\{\begin{array}{c}{\left(x/x_{max}\right)}^{\alpha }ifx<x_{max}\hfill \\ 1otherwise\hfill \end{array}\right.}$

Among them, xrepresents the number of co-occurrences, and x_max represents the maximum number of co-occurrences.

Feature representation based on Bi-LSTM

Because LSTM is good at capturing the long-distance and long-term dependence of sentence context information, it can better avoid the problem of gradient disappearance and gradient explosion and has higher computational efficiency. Nevertheless, LSTM cannot capture the two-way information of sentences. For the task of formulaic language identification, if the forward information and backward information of sentences are added, the model can learn more semantic information when processing text (Hochreiter & Schmidhuber, 1997). Therefore, we use Bi-LSTM to learn the hidden layer representation of input sequences, hoping to obtain the features of sentences that can contain deeper semantic and syntactic information.

For sentence s_i = [x₁, x₂, …, x_t, …, x_n], input it into the Bi-LSTM network, we can get the hidden layer representation $\left\{h_1,h_2,\dots ,h_t,\dots ,h_n\right\}$ of sentence s_i. Each unit obtains the current hidden vector h_t according to the calculation of the previous hidden vector h_t−1 and the current input vector x_t, and its operation is defined as follows:

(4)${\displaystyle i_t=\sigma \left(W_{xi}{x}_t+W_{hi}{h}_{t-1}+W_{ci}{c}_{t-1}+b_i\right)}$ (5)${\displaystyle f_t=\sigma \left(W_{xf}{x}_t+W_{hf}{h}_{t-1}+W_{cf}{c}_{t-1}+b_f\right)}$ (6)${\displaystyle c_t=f_t{c}_{t-1}+i_{t}tanh\left(W_{xc}{x}_t+W_{hc}{h}_{t-1}+b_c\right)}$ (7)${\displaystyle o_t=\sigma \left(W_{x0}{x}_t+W_{ho}{h}_{t-1}+W_{co}{c}_t+b_o\right)}$ (8)${\displaystyle h_t=o_{t}tanh\left(c_t\right).}$

In the formula: i_t, f_t, c_t, o_t, h_t are the state of input gate, forget gate, cell state, output gate and hidden layer when the t-th text is input; W is the parameter of the model; b is the bias vector; σ is the Sigmoid function; tanh is the hyperbolic tangent function.

Bi-LSTM model is composed of forward LSTM and reverse LSTM models. The LSTM network of each layer outputs a hidden state information respectively, and the parameters of the model are updated by back propagation. The structure of Bi-LSTM model is shown in Fig. 2.

In Fig. 2, x_t represents the input of the network at time t, the LSTM in the box is the standard LSTM model, $\overrightarrow{{\mathrm{y}}_{\mathrm{t}}}$ is the output of the forward LSTM at time t, and ${\stackrel{⃖}{\mathrm{y}}}_{\mathrm{t}}$ is the output of the reverse LSTM at time t.⊕ indicates splicing operation. The output representation of Bi-LSTM at time t is defined as ${\mathrm{y}}_{\mathrm{t}}=\left[{\overrightarrow{\mathrm{y}}}_{\mathrm{t}}:{\stackrel{⃖}{\mathrm{y}}}_{\mathrm{t}}\right]$, that is, the output at time t is directly spliced by the forward output and the reverse output.

Feature fusion

Feature fusion includes early fusion and late fusion. Early fusion is to fuse multi-layer features first and then train the model on the fused features (only after complete fusion, unified training). In this article, early fusion is used as a comparative experiment. First, the part-of-speech features and semantic features are fused, and then the fused features are input into Bi-LSTM. The generated results are used as the basic feature vector. The structure diagram is shown in Fig. 3.

Compared with early fusion, late fusion first trains the model with a single feature and then fuses the training results of multiple models. The advantage of late fusion method is that it can flexibly select the model’s results and improve the fault tolerance of the system. The amount of calculation of fusion information is reduced and the real-time performance of the system is improved (Khaleghi, Khamis & Karray, 2013). In this article, the late fusion method is adopted. First, the part-of-speech features and semantic features are input into Bi-LSTM respectively, and then the results of the two models are spliced to form the basic feature vector. The structure diagram is shown in Fig. 4.

Associated information based on mutual information

Mutual information measures the correlation between two random variables, that is, the amount of information about another random variable contained in one random variable (Zhang & Wang, 2016). The mutual information of two discrete random variables X and Y is defined as: (9)${\displaystyle I\left(X;Y\right)=\sum _{x\varepsilon X}\sum _{y\varepsilon Y}p\left(x,y\right)log\frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}.}$

where p(x, y) is the joint probability distribution function of X and Y, p(x) and p(y) are the edge probability distribution functions of X and Y, respectively. If we want to measure the correlation degree of any two words x and y in a data set, we can calculate it as follows: (10)${\displaystyle I\left(x;y\right)=p\left(x,y\right)log\frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}.}$

where p(x) and p(y) are the probability of independent occurrence of x and y in the data set. It can be obtained by directly counting the word frequency and dividing it by the total number of words; p(x, y) is the probability that x and y appear in the data set at the same time. Directly count the number of times they appear at the same time, and then divide it by the number of all unordered pairs. Using mutual information to calculate the relationship between binary words, the higher the mutual information, the higher the correlation between x and y, and the greater the possibility of forming formulaic language.

Associated information based on dependency syntactic parsing

Dependency syntax reveals the dependency relationship and collocation relationship between words in a sentence. One dependency relationship connects two words, one is the core word and the other is the modifier. This relationship is related to the semantic relationship of the sentence. The dependency relationship between words in a sentence includes subject-predicate relationship, verb-object relationship, inter-object relationship, etc. The dependency syntactic relation of the sentence “evaluations play an invalid role in X.” is shown in Fig. 5. Among them, “play an invalid role in” is a formulaic language. From the figure, it can be seen that there are complex dependencies between these five words. Therefore, dependency syntactic parsing between words can express the dependency between two words. The closer the relationship is, the more likely it is to form a formulaic language.

Feature representation based on GCN

In the formulaic language identification using dependency syntactic relation, the existing studies mostly use the dependency syntax in the text to construct rules, extract features, and then input them into the classifier to recognize formulaic language through classification. Although this method can achieve certain results, the nonlinear semantic relationship between components in sentences has not been learned and utilized. Spatially, the relationship between words can be represented by a graph through mutual information and dependency parsing. The GCN is used to process the correlation information, so that the nonlinear semantic relationship between words in the sentence can be extracted.

In the feature representation based on GCN, the mutual information and dependency syntactic relation between words are used to determine the word connection relationship, and the basic features of words are represented as nodes. For the GCN with mutual information as input, firstly, the corpus is used as the data set to calculate the mutual information value between words, the word is used as the node, the mutual information value between nodes is used as the representation of the edge. The element a_ij in its adjacency matrix AɛR^N∗Nrepresents the mutual information value between the ith node and the jth node in the graph. For the GCN with dependency syntactic relation as input, firstly, the dependency syntax of sentences is analyzed, with words as nodes and the dependency between words as the representation of edges. The element a_ij in its adjacency matrix AɛR^N∗Nrepresents the dependency between the ith node and the jth node in the graph. If there is a dependency between the two nodes, the a_ij is 1, otherwise it is 0. For example, in the example of dependency syntactic parsing “evaluations play an invaluable role in X.”, the adjacency matrix A constructed based on dependency syntactic parsing is shown in Fig. 6.

In order to directly conduct deep learning modeling on graph data, the specific method adopts a variant model of convolution neural network-Graph Convolution Neural Network proposed by Kipf & Welling (2016), and the structure is shown in Fig. 7. The channels input by GCN is C, that is, the feature vector dimension of each node X_i is C, the channels output by GCN is F, that is, the feature vector dimension of each node Z_i is F, and finally the label Y_i of the node is predicted by the feature of the node.

Specifically, given a graph G = (V, E), V is a vertex set containing N nodes, E is an edge set including self-loop edges (that is, each vertex is connected to itself). The feature information of graph G = (V, E)can be represented by Laplace matrix, as shown in Eq. (11). (11)${\displaystyle L=D-A.}$

Or use the symmetric normalized Laplasse matrix, as shown in Eq. (12): (12)${\displaystyle L^{sys}=I_N-D^{-1/2}A{D}^{-1/2}.}$

where A is the adjacency matrix of the graph, I_N is the n-order identity matrix; D = diag(d) is the degree matrix of the vertex, as shown in Eq. (13). (13)${\displaystyle D^{ii}=\sum _j{A}_{ij}.}$

Based on the Fourier transform of graph, the graph convolution formula can be expressed as Eq. (14): (14)${\displaystyle g\mathrm{\ast }x=U\left[\left(U^{T}g\right)\cdot \left(U^{T}x\right)\right].}$

In the formula, x is the basic feature vector of the node; g is convolution kernel; U is the eigenvector matrix of Laplace matrix L.

In order to reduce the amount of calculation, scholars used Chebyshev polynomials to simplify the graph convolution formula in 2017. Finally, the layered propagation formula of graph convolution can be expressed as Eq. (15): (15)${\displaystyle X^{\left(l+1\right)}=\sigma \left[{\tilde{D}}^{-1/2}\tilde{A}{\tilde{D}}^{-1/2}{X}^{\left(l\right)}{W}^{\left(l\right)}\right].}$

In the formula, $\tilde{A}=A+I_N$, $\tilde{D}=\sum \tilde{A}$; σ is the activation function; W is the weight matrix to be trained.