A Sparse Self-Attention Enhanced Model for Aspect-Level Sentiment Classification

The given input sentence \({\text{S}}=\left\{{{\text{s}}}_{1},{{\text{s}}}_{2},{{\text{s}}}_{3},\dots ,{{\text{s}}}_{{\text{a}}},{{\text{s}}}_{{\text{a}}+1},{{\text{s}}}_{{\text{a}}+2},\dots {{\text{s}}}_{{\text{a}}+\left({\text{m}}-1\right)},\dots ,{{\text{s}}}_{{\text{N}}}\right\}\) of length \({\text{N}}\) with \({\text{m}}\) aspect words \(\left\{{{\text{s}}}_{{\text{a}}},{{\text{s}}}_{{\text{a}}+1},{{\text{s}}}_{{\text{a}}+2},\dots {{\text{s}}}_{{\text{a}}+({\text{m}}-1)}\right\}\) has to be recast into contextualized word embedding vectors by using \(pre-trained\) BERT model. The input format of the given sentence \({\text{S}}\) to the BERT is given by the split up of \("[{\text{CLS}}]+{\text{sentence}}+[{\text{SEP}}]+\mathrm{aspect term}+[{\text{SEP}}]"\). Thus this input format extracts the overt interactions between sentence and aspect term. The embedding information of sub-words generated by BERT is subjected to average pooling which produces ultimate embedding vector \({\text{X}}\in {{\text{R}}}^{{\text{N}}\times {\text{d}}}\), where \({\text{d}}\) is dimension of BERT output. The vector \({\text{X}}\) is then represented as vector sequence, \(\left\{{{\text{w}}}_{1},{{\text{w}}}_{2},\dots {{\text{w}}}_{{\text{n}}}\right\}\) for sentence and \(\left\{{{\text{v}}}_{{\text{a}}},{{\text{v}}}_{{\text{a}}+1},{{\text{v}}}_{{\text{a}}+2},\dots {{\text{v}}}_{{\text{a}}+({\text{m}}-1)}\right\}\) for aspect terms respectively.

The \(vanilla RNN\) suffers from vanishing or exploding \(gradient\) problem for long sequences due to replacement of entire sequence in hidden state during every time step \({\text{t}}\). The Gated Recurrent Unit (GRU) solves this issue by the inclusion of two additive gates in their architecture. The GRU contains two gates where one is \(update gate\) and other is \(reset gate\). The gates present in GRU removes the unimportant and retains only the important information. The reset gate combines current input with the important part of the previous \(hidden state\) to produce a new \(hidden state\). The update gate determines the amount of information from current hidden state to be included with final hidden state. This allows network to retain long-term dependencies. The workflow diagram of GRU has been illustrated in Fig. 1.Calculation process of GRU is given from Eq. (1) to (5):

The class labels represented in the proposed work \(\left\{{\text{positive}},\mathrm{ negative},\mathrm{ neutral}\right\}\).

The general GRU struggles to identify the key part of a sentence for the ASC task. To overcome this problem, this work proposes a novel SSA-GRU-AE. This work integrates sparse self-attention mechanism with GRU to capture a relevant part of sentence with respect to a given aspect. Aspect information play key role to classify polarity of given sentence. To utilize aspect information effectively this work proposes to generate embedding vector using BERT for each aspect. Here \({{\text{V}}}_{{{\text{a}}}_{{\text{i}}}}\in {\mathbb{R}}^{{{\text{d}}}_{{\text{a}}}}\) is embedding of aspect i, where \({{\text{d}}}_{{\text{a}}}\) is dimension of aspect embedding. \({\text{H}}\in {\mathbb{R}}^{{\text{d}}\times {\text{N}}}\) is matrix formed by hidden vectors \(\left[ {{\text{h}}_{1} ,{\text{h}}_{2} ,{\text{h}}_{3} , \ldots ,{\text{h}}_{{\text{N}}} } \right]\) that GRU generated. Here \({\text{d}}\) is size of the hidden layers and \({\text{N}}\) is the length of given sentence. \({\text{e}}_{{\text{N}}} \in {\mathbb{R}}^{{\text{N}}}\) is vector of ones. An attention weight vector \({\upalpha }\) and a hidden weight vector \({\text{ r}}\) is produced by self-attention mechanism, as specified in Eqs. (6) to (8).where \({\text{ M}} \in {\mathbb{R}}^{{\left( {{\text{d}} + {\text{d}}_{{\text{a}}} } \right) \times {\text{N}}}}\), \({\upalpha } \in {\mathbb{R}}^{{\text{N}}}\), \({\text{r}} \in {\mathbb{R}}^{{\text{d}}}\). \({\text{W}}_{{\text{h}}} \in {\mathbb{R}}^{{{\text{d}} \times {\text{d}}}}\), \({\text{W}}_{{\text{V}}} \in {\mathbb{R}}^{{{\text{d}}_{{\text{a}}} \times {\text{d}}_{{\text{a}}} }}\) and \({\text{W}} \in {\mathbb{R}}^{{{\text{d}} + {\text{d}}_{{\text{a}}} }}\) are weight parameters. \(\otimes\) is a concatenation operator, which repeatedly concatenates \({\text{V}}\) for \({\text{N}}\) times. \({\text{W}}_{{\text{V}}} {\text{V}}_{{\text{a}}} \otimes {\text{e}}_{{\text{N}}}\) repeats the linearly tranformed \({\text{V}}_{{\text{a}}}\) as many times till the last word in the given sentence and eventually represented in the Eq. (9),where \({\text{h}}^{*} \in {\mathbb{R}}^{{\text{d}}}\), \({\text{W}}_{{\text{p}}}\) and \({\text{W}}_{{\text{x}}}\) are weight parameters.

The self-attention mechanism captures significant part of sentence with respect to an aspect. Here \({\text{h}}^{*}\) represents feature of a particular sentence with respect to an aspect. Then a linear layer is added to transform sentence vector \({\text{e}}\), which is a vector of length which equals class number \(\left| {\text{C}} \right|\). Finally a softmax layer is applied to transform \({\text{e}}\) to conditional probability distribution as specified in Eq. (10).where \({\text{W}}_{{\text{s}}}\) and \({\text{b}}_{{\text{s}}}\) are the weight and bias parameter of \({\text{softmax}}\) layer. Further \({\text{L}}_{1}\) regularize is applied to make sure only minimal words of the sentence contributes to the semantic and sentiments of the sentence. The sparse self-attention mechanism is depicted in Eq. (11).

The proposed sparse self-attention mechanism effectively removes the words which are unimportant to predict the sentiment and calculates the key part of the sentence. After computation of the important part of the sentence, a weighted summation is performed to predict the sentiment polarity which is specified in Eq. (12).where \({\text{x}}_{{\text{i}}}\) denotes embedding of \({\text{i}}^{{{\text{th}}}}\) word in sentence, \({\text{n}}\) is length of sentence. Finally output of sparse self-attention layer is obtained using Eq. (13).where \({\hat{\text{y}}}\) is a predicted sentiment polarity which is a \(2 - {\text{D}}\) vector where \(\left( {1,0} \right)\) and \(\left( {0,1} \right)\) are \(positive\) and \(negative\) labels respectively.

The model is trained using back propagation where \(cross entropy\) loss function is used to minimize error between \({\text{y}}\) and \({\hat{\text{y}}}\) for all sentences which is specified in Eq. (14).where \({\text{i}}\) and \({\text{j}}\) is index of sentence and class respectively.

Then an Adagrad optimization [48] is adopted to train the model over mini batches. This optimizer improves the strength of SGD on learning process and also performs increase in updates to learning rate for unusual parameters and decrease in updates to learning rate for usual parameters. The workflow diagram of proposed SSA-GRU-AE is illustrated in Fig. 2.

5 datasets \(\left( {{\text{twitter}},{\text{ LAP}}14,{\text{ REST}}14,{\text{ REST}}15{\text{ and REST}}16} \right)\) are employed to prove that the proposed method mastered the existing baseline models.The 5 datasets contains reviews of users with a set of aspects with its polarities from which the sentences with contradictory polarities or inexplicit aspects are removed. The purpose of the proposed work is to ascertain aspect polarity of a sentence with respect to aspect. The dataset details are exhibited in Table 4.

Ten sentiment classification models are implemented as baseline models for comparison with proposed model is studied as:

The result analysis of proposed work with eighteen baseline models clearly proves that proposed work outperformed the existing models in terms of \(accuracy\) and \(F1 - score.\) The SSA-GRU-AE model consistently performed better than all the baseline models in all the five datasets \(\left( {twitter, LAP14, REST14, REST15 and REST16} \right)\) whereas SVM and LSTM performed poorly in all five datasets consistently due to the manual feature engineering of SVM and the lack of aspect information in LSTM for ASC.

The IAN and AOA model is implemented using attention mechanism to solve ASC task by attending all the aspect and context words that helps to outperform both the SVM and baseline LSTM models. The experimental results on all the five datasets proved that both the IAN and AOA consistently performed well. The issue with the attention mechanism is that if the dataset is noisy or it contains multiple aspects this mechanism falsely assigns high scores to irrelevant words and also these attention based models attend all the aspect words with different weights which incorrectly guides \(aspect term\) to focus on syntactically \(unrelated words\). These issues causes AOA and IAN models performs moderately on all five datasets, in terms of \(accuracy\) and \(F1-score\).

GCN based models effectively captures both syntactic word dependencies and long range word relations. These models worked well with datasets which are rich in syntax information with good grammatical structure. Especially all the GCN based models effectively worked on \(LAP14, REST15 and REST16\) datasets and also improved both the \(accuracy\) and \(F1-score\). These models failed to work well with the datasets which has less grammatical information and less sensitive syntax information. That is on both the Twitter and REST14 datasets these GCN models produced less \(accuracy\) and \(F1 - score\). The results shown in Table 5, 6, 7, 8, 9 and 10 clearly proved that proposed work outperformed existing GCN based models in terms of \(accuracy\) and \(F1-score\).

Memory network based models worked well on all five datasets continuously in terms of \(accuracy\) and \(F1 - score\). The base memory network model effectively captured aspect-sequence modeling well, but it failed to capture context and sequence information and hence decreased performance of the MemNet model in all the five datasets. But CMA-MemNet and DSMN captured both the information effectively, which helps them to perform better in all five datasets. The results on all five datasets showed that CMA-MemNet and DSMN performed well on all five datasets which outperformed all basic, \(attention\) and \(GCN\) models. Although these models performed well on all datasets, still it failed to recognize all aspects correctly, which led to performance loss.

The proposed novel sparse self-attention GRU with aspect embedding implementing BERT outperformed all the baseline models on all five datasets in terms of \({\text{accuracy}}\) and \({\text{F}}1 - {\text{score}}\). The BERT model used in our work effectively captured contextual information which helped to capture semantic information. Also sparse self-attention mechanism introduced in our work effectively removed unimportant words and captured only important words related to sentence. Also \({\text{L}}1\)-regularize applied on attentions helped to ensure that only a few words contributed to sentiment of sentences. This helped proposed model to perform better on datasets which are grammatically poor and noisy \(\left( {{\text{Twitter and REST}}14} \right)\). These advantages helped our model to outperform the existing baseline models on all five data sets where especially our model consistently performed well on \({\text{Twitter and REST}}14\) datasets too. Figures 3, 4, 5, 6, 7 shows the comparative results of all the baseline models with the proposed model on five different datasets.

An ablation research is carried out to examine the significance of each component in the proposed model. First pre-trained BERT model is replaced with GloVe model and then executed on five datasets. The proposed model without BERT performed poorly in terms \(accuracy\) and \(F1 - score\) compared to model with BERT. The model without BERT produced 2.9%, 2.7%, 1.6%, 1.4%, and 1.3% less \(accuracy\) and 2.6%, 2.4%, 1.3%, 1.1% and 1% less \(F1 - score\) than model with BERT on \(Twitter, REST14, LAP14, REST15 and REST16\) datasets respectively. The proposed model without BERT performed poor when compared to some of GCN and memory network models too. The BERT model effectively captured semantic relation between aspect and context. This helped to improve performance of proposed model.

To identify importance of \(sparse self - attention mechanism\), a model without \(sparse self - attention mechanism \) is designed. Then model without sparse self-attention mechanism was executed on five datasets. The proposed model without sparse self-attention mechanism performed poorly in terms \({\text{accuracy}}\) and \({\text{F}}1 - {\text{score}}\) compared to model with model with sparse self-attention mechanism. The model without sparse self-attention mechanism produced 2.6%, 2.3%, 1.4%, 1.2%, and 1.1% less \({\text{accuracy}}\) and 2.3%, 2.1%, 1.1%, 0.9% and 0.7% less \({\text{F}}1 - {\text{score}}\) than model with sparse self-attention mechanism on \({\text{Twitter}},{\text{ REST}}14,{\text{ LAP}}14,{\text{ REST}}15{\text{ and REST}}16\) datasets respectively.The experimental results on five datasets proved that proposed sparse self-attention mechanism improved performance of proposed model. The sparse self-attention mechanism effectively captured the importance of each word in the context with respect to \(aspect\). This helped to remove unimportant words from sentence and keep only important part of sentence. This helped to improve performance of proposed model over model without sparse self-attention mechanism.

For instance, consider the following phrase: " \(\user2{Even if it^{\prime}s a good day}\), \(\user2{I don^{\prime}t feel it}\). \(\user2{I^{\prime}m really miserable}\). " The terms "\(miserable\)," "\(feel\)," and "\(don^{\prime}t\)" are extremely significant in predicting the sentiment polarity of this sentence than the words "\(good\)" and "\(day\)". Many words, including "the," "in," "it," and "I'm," are also unimportant. Therefore, it's crucial to create a model that can accurately depict the significance of each word in the documents. Additionally, it must remain sparse enough that only a \(few words\) can accurately categorize the sentiment labels of the sentence. The self-attention layer of the proposed SSA-GRU-AE was used to determine the significance of each word in sentence. In order to make sure that only a handful of words are needed to identify the sentiment polarities of sentences, an L1 regularization is then used for these weights.

Consider the following sentence," The meal is tasty, but the restaurant is untidy. ", the proposed model predicts \(sentiment polarity\) for the aspect "meal" as \(positive\) and "restaurant" as \(negative\). The self-attention layer in the proposed model accurately identifies the \(aspect\) term "meal" and its \(context\) as "tasty" and also the \(aspect\) term "restaurant" and its \(context\) as "untidy". The sparse nature of the \(self - attention\) mechanism of the proposed model helps to retain only important words from the sentence such as "meal", "tasty", "restaurant" and "untidy" and also helps to avoid unimportant words such as "The", "is" and "but". The ability of sparsification in retaining only important words from the sentence helps to achieve ASC tasks efficiently. The self-attention mechanism serves as a sparsification mechanism in the proposed model, where it is regarded as a type of regularization that potentially enhances the quality of the model by efficiently diminishing noise within it. Due to the sparse nature of the proposed SSA-GRU-AE, neurons combine the output activations which are very similar, change their biases, and then rewire the network to reflect these changes. This sparsification enhances the efficiency of models by allowing them to function effectively in feature spaces with high dimensions. This also reduces the complexity of representation, wherein only a subset of dimensions is utilized at any given moment and decrease complexity by nullifying specific subsets of the model parameters. This resulted in reducing many useless words present in the sentence for \(sentiment polarity\) prediction with respect to an \(aspect\). Because of these advantages, the proposed model identifies the \(aspect\) phrases and its corresponding \(context\) words as a sequel to sentences, such as "meal" and "tasty" in the above statement.

Another example is, " The laptop's model is ok, and its performance is great. ". The proposed SSA-GRU-AE model identifies the aspect term "model" and its context as "ok" correctly and predicts its sentiment polarity as neutral. But for the aspect term "performance" the context is identified as "great" and predicts its sentiment polarity as positive. The sparse-self-attention mechanism assigns high weights to the terms "laptop", "model", "ok", " performance", and "great" and low weights to the terms "The", "is", "its" and "and". This example shows the importance of sparsification in self-attention mechanism for ASC tasks. The L1-regularize applied on attentions helped to ensure that only a few words contributed to the \(sentiment\) of sentences. This is due to the fact that when entire neurons or filters are eliminated, the principles of associativity and distributivity can be employed to convert a sparsified structure into a more compact, dense structure. Nevertheless, in the event that eliminating arbitrary components of a weight matrix, it becomes necessary to retain the indices corresponding to the non-zero items that remain. The process of model sparsification alters the model's characteristics, although it does not modify the sparsity pattern observed after successive inferences or forward passes. This helped proposed model to perform better on datasets which are grammatically poor and noisy.

Consider the following example, " The workers should do more work truly. " by considering "workers" as the aspect and "more work truly" as its context phrase to identify its sentiment polarity. The existing DNN models identify the aspect term as "workers" and its context term as "work truly" and predicts the sentiment polarity as positive. But actually the weight of the term "more" is important in this context. The sparse-self-attention in the proposed model helps to identify the context term "more work truly" to predicts its actual sentiment polarity "negative". This elucidates the importance of the sparse nature in self-attention mechanism. This also shows that the proposed model can able to capture the context words with implicit meaning (Fig. 8).