O\(^2\)-Bert: Two-Stage Target-Based Sentiment Analysis

In prior research, the conventional approach for the OTE task involved manually selecting features to build a model using frequency, rule-based, or machine learning techniques. Rana and Cheah [15] proposed TF-RBM, a two-layer rule-based model that leverages sequential patterns from customer reviews to define rules. It enhances opinion target extraction accuracy by integrating frequency- and similarity-based approaches. While rule-based methods are effective, their design poses challenges. Creating rules necessitates specialized linguistic expertise and knowledge, and capturing all rules for natural languages proves exceedingly difficult. Shams and Baraani-Dastjerdi [17] designed a Latent Dirichlet Allocation (LDA) topic model that leverages co-occurrence relations as prior domain knowledge to uncover more precise aspects. Jochim and Deleris [13] proposed a CRF-based technique for extracting named entities from medical literature. The study explored the impact of constraints to enhance accuracy. While the assumptions of CRF are generally well-defined, they may not always align with the problem at hand. CRF models face challenges in scalability and fail to effectively capture contextual information over long distances. As a result, achieving high performance in scenarios involving extensive data content can be challenging.

In recent years, researchers have primarily focused on integrating deep learning methods with CRF models to automatically extract opinion targets, as deep learning models excel at extracting high-level features. Guo et al. [5] employed a combination of CRF and recurrent neural network (RNN) to automatically extract opinion targets and opinion words, eliminating the need for manual feature selection. Wang et al. [9] addressed the issue of opinion target number and boundary errors in sequence tagger extraction by proposing a post-processing method. They aimed to control the number of extracted opinion targets and correct their boundaries. On the other hand, Lu and Liu [6] employed two bidirectional gated recurrent unit (BiGRU) networks to extract semantic features and incorporated a self-attention mechanism to capture global dependencies. Su et al. [12] proposed the XLNetCN model as a solution for the TBSA task, incorporating a capsule network with a dynamic routing algorithm. This approach effectively captures local and spatial hierarchical relations within the text sequence, leveraging its proficiency in multilabel text classifications. Pour and Jalili [10] proposed an innovative data preprocessing technique and a deep convolutional neural network that accurately classifies each word in a given sentence as either an aspect or non-aspect word.

Furthermore, researchers have also explored enhancing information encoding alongside feature extraction optimization. Zhang et al. [4] and Bravo-Marquez utilized BERT and Word2Vec in their embedding layer, while Li et al. [16] leveraged the position information of the opinion target during sentence encoding, resulting in improved performance. Kang et al. [7] proposed RABERT, a targeted opinion word extraction method, which encodes target information into BERT by incorporating target markers within the sentence. Additionally, a target-sentence relation network is integrated into RABERT to account for neighboring words. Moreover, various span-based models have been introduced. To handle cases where a token belongs to multiple entities, Gao et al. [20] utilizes a span-based tagging scheme as opposed to traditional sequence labeling models that can assign only one label per token. Hu et al. [21] proposed a span-based framework that identifies opinion targets based on their span boundaries and determines their polarities using span representations. They treated the task of extracting opinion targets and their sentiment polarities as a sequence tagging problem, addressing challenges related to the extensive search space and sentiment inconsistencies. Xu et al. [22] proposed a span-level model to extract opinion targets comprising multiple words, leveraging interactions between the spans of opinion targets and sentiment words. They also devised a dual-channel span pruning strategy that combines supervision from aspect term extraction and opinion term extraction tasks. In contrast, Yu Bai Jian et al. [23] proposed a novel paradigm called ASTE-RL, which first extracts sentiment words and then identifies their opinion targets, considering the mutual interactions among aspect terms, associated sentiments, and opinion terms.

For OSC, the most straightforward approach is based on a dictionary, which is dependent on sentimental words marked in the dictionary. Xu et al. [31] designed a dictionary-based method, whose dictionary can update the words newly joined and the compound words using machine learning algorithms. Dictionary-based approaches require human effort to gather opinion words. They necessitate manual removal or correction of words containing errors, and they lack the ability to tailor the thesaurus to a specific application scenario or context. Subsequently, machine learning techniques such as Support Vector Machine (SVM), Multilayer Perceptron (MLP), and others were employed for OSC tasks. Almaghrabi and Chetty [33] developed a Multilayer Perceptron model for sentiment analysis in Arabic and English languages. However, a major drawback of the aforementioned approaches was the manual selection and tuning of rules or features.

In recent research, attention mechanisms have emerged as a prominent method for deep learning. Attention mechanisms enable the learning of varying levels of importance for sequential words, enhancing information capture. Wang et al. [28] previously utilized the attention mechanism in conjunction with a long short-term memory (LSTM) network to acquire aspect embeddings and perform aspect-level sentiment classification. Similarly, Du incorporated SenticNet into an LSTM network, considering user expression patterns through the application of attention mechanisms. Although LSTM is adept at sequence modeling, it encounters challenges such as information loss over long distances and the inability to effectively capture constraints, such as the typical association of adjectives with nouns, thereby limiting its performance. Moreover, researchers have increasingly explored the integration of attention mechanisms with other deep learning techniques, yielding improved outcomes. Du et al. [30] developed a model that primarily leverages a capsule network and attention mechanism, employing a bidirectional recurrent neural network (BiRNN) to extract features. Li et al. [24] proposed a model incorporating a hierarchical multi-head attention mechanism and a graph convolutional network to effectively consider both syntactic dependencies and the relationship between opinion targets and their context. Miao et al. [32] proposed a contextual graph attention network that consists of two graph attention networks and a contextual attention network to capture aspect-sensitive text features and proposed a novel syntactic relative distance-based syntactic attention mechanism for enhanced attention towards opinion targets while reducing computational complexity. In contrast, Wei et al. [25] developed GP-GCN, which simplifies global features by addressing potential noise introduced by contextual information. They employed orthogonal projection and graph convolutional networks to reduce inter-node and node-global feature dependencies.

Tiwari et al. [39] conducted a comparative analysis of various BERT-based approaches for solving the ABSA task, including fine-tuned BERT models, adversarial training with BERT, and the integration of disentangled attention in BERT or DeBERTa. Chouikh et al. [38] focused on contrasting different versions of BERT specifically designed for the Arabic language in their model. Zhu et al. [40] developed the BERT-pair-ABSA model, which employed semantic expansion of auxiliary sentences and sentiment polarity calculation to gain insights into netizens’ changing concerns, emotional states, and evolving trends across different stages. Gutierrez et al. [14] examined the few-shot performance of GPT-3 in-context learning compared to fine-tuning smaller PLMs (similar to BERT) on two significant biomedical information extraction tasks: named entity recognition and relation extraction.

Typically, conventional approaches for named entity recognition (NER) in the context of opinion target extraction (OTE) rely on BIO-based or span-based models. However, as previously mentioned, a significant challenge is addressing the issue of “NULL” targets, where the target entity does not exist within the sentence, as illustrated in Example 1. In this example, the sentiment polarities of the target “food taste” and “waiting time” are respectively positive and negative. However, the mention of “waiting time” is absent in the sentence. Both BIO-based and span-based NER approaches fail to address this issue. To overcome this challenge, we introduce a dedicated entity number prediction module to enhance entity annotation. By utilizing this module, we can determine the count of entities, referred to as the “prediction number”. Alternatively, the number of annotated entity start positions, known as the “annotation number”, can also be used to infer the entity count. By comparing the difference between these two numbers, we can identify the presence of “NULL entities”. In this specific example, two entities are predicted, while only one entity start is annotated, indicating the presence of one “NULL entity”. Furthermore, in our experiments, the occurrence of the prediction count being lower than the annotation count is rare, primarily due to the training set’s distribution.

Given the three potential scenarios for the difference between the two numbers, it is feasible that the “prediction number” may be lower than the “annotation number.” In such cases, we can rely on the “prediction number” and generate the top N (equivalent to the “prediction number”) probable entities from the entity annotation. The entity number prediction module typically employs a classification model, utilizing the encoding information from the BERT model as input, coupled with a dense layer. Additionally, to address the imbalanced data distribution, the focal loss is employed.

Another challenge is to process the multiple entities as well as entities with multiple words. Traditional approaches, usually struggle to determine whether there is a multiple-word entity or several single-word entities. In Example 2, the number of entities cannot be determined. Moreover, in Example 3, there are nested entities. As a consequence, we propose to separate the entity annotation into two modules, in which, an entity starting annotation module marks the start positions of entities, while an entity length prediction module predicts the lengths of entities based on their start positions. In Example 2, “prix” is marked as the start position and the length of entity is predicted as 3, hence the annotated entity is “prix fixe menu”. Similarly, in Example 3, the annotated results are not influenced by nested entities.

Pretrained language models show great abilities to capture contextual semantics. In this work, BERT [41] is utilized as the encoders. Specifically, we use pretrained Roberta-base [42]¹ as the initial checkpoint and perform continue-pretraining on domain corpus. Each input sentence is firstly processed into a sequence like “[CLS] \(token_1\), \(token_2\), ..., \(token_n\), [SEP]”. Afterwards, these tokens are mapped to token embeddings, as well as segment embeddings and position embeddings, according to the basic usage of BERT. These embeddings are fed into the BERT transformer encoders to formulate the contextual embeddings of the tokens. The three modules share the contextual embeddings to make full use of pretraining. The following is the detailed descriptions of the three modules.

This section employs a CRF with entity start embeddings, utilizing the starting position and entity number as constraints to predict entity lengths. While CRF can extract entities independently, its accuracy is influenced by the selection of starting positions. Incorrect or incomplete selection of starting positions may result in a chain of errors. Furthermore, empirical evidence suggests that the recall of a standalone CRF is lower compared to incorporating an additional annotation module for labeling start positions. This module procedure is shown in Algorithm 1.

During CRF training, the loss function enables the learning of word relations. In contrast, traditional neural networks output results through softmax without considering associations. If there is a need to learn such relations, previous layers of the model must account for it. The models’ ability to establish connections is significantly impacted by the choice of loss function and tag settings.

CRF computes the probability of each tag at every position, taking into account the likelihood of a tag sequence. It evaluates the probabilities for each position and selects the method with the highest probability as the final outcome. For example, there is a sample with a length of eight, and every word may be one of the three categories, such as noun (n), adjective (a), and verb (v). In our CRF, we use the label system that includes ‘<start>’ (the beginning of a sentence), ‘<stop>’ (the ending of a sentence), ‘B’ (the beginning of entity), ‘E’ (the rest words of entity), ‘O’ (nonentity) and ‘<mask>’ (other objects).

CRF assumes the position of every word as n (noun), a (adjective) or v (verb) and adds all the probability of every position as the probability of a situation. Finally, it selects the maximum from all conditions, as in Example 4.

The loss computation in the OTE involves a weighted sum of the losses from three submodules, where the weights are determined as hyperparameters.

In ⑥ in Table 1, different “rolls” correspond to different polarities, while for most approaches result in similar polarities. Therefore, it is challenging for traditional sentiment analysis models to distinguish the contradictory polarities of multiple targets. The phenomenon is common in real-world applications, accounting for 21% on SemEval.

We utilize an attention mechanism to derive the sentiment polarity associated with entities. Moreover, the condition layer normalization algorithm [44] normalizes the input sentence into various normal distributions by incorporating an “input condition” within the range of (0,1) distribution from layer normalization.

The procedure of OSC is depicted in Algorithm 2. This model takes the entities as the conditions of condition layer normalization, which can normalize the different entities in the same sentence into different distributions. Finally, this model uses capsule networks to solve the “contradictory-polarity problem”. Capsule networks offer a more comprehensive determination by considering the relevance between outputs, in contrast to softmax-based methods.

To mitigate the impact of irrelevant paddings, we employ an attention pooling layer and a dense layer. Additionally, the use of attention helps prevent the capsule network from disregarding the contextual-opinion target relationship.

The capsule network utilizes a trained clustering algorithm to classify the sentiment polarity associated with each target. Detailed rule analysis ensures accurate target extraction and differentiation of various polarities. Ultimately, a category vector, encompassing polarity markings, is optimized and generated during network training.

The O\(^2\)-Bert model is suitable for short sentences (usually less than 128 tokens) in English corpuses, and is intended for English corpuses after tokenization. The model is not domain-specifically designed, therefore, both tweets and news texts are suitable. However, document-level journalistic news may not be suitable due to its length.

We use the SemEval2014-16 dataset in the experiment. This dataset comments contain the laptops and restaurants domain, presented in Table 3. The number of entities in each sentence is uneven. In the SemEval2014 dataset, none of the sentences include NULL, but the composition of words is more complex. The approximate distribution is presented in Table 4. In SemEval2015 & SemEval2016, most samples contain two entities, while the number of samples containing more entities, such as six or eight, is very limited, and some entities are null.

Ten cross-validations are used in the experiment. We divided the official training set into a training set and a validation set, respectively 70% and 30%, and kept the distribution of the samples in the training set. And we take the official test set as our test set. The datasets division is presented in Table 5.

We utilize F1-score to evaluate the OTE task, as it shares similarities with information retrieval. For assessing aspect-specific sentiment polarity, accuracy (Acc) and F1-score are commonly used to represent true results in evaluating the OSC task. Acc and F1-score are calculated as follows:where TP is true positive sample, FP is false positive sample, TN is true negative sample, and FN is false negative sample.

The O\(^2\)-Bert model designed in this paper uses Nadam as the optimizer, and the initial learning rate is 0.15. With the increase in epochs, the power index of the learning rate decreases, and it closes to 1E-5 after 500 epochs. The weight attenuation coefficient of the optimizer is 1E-2, and the momentum value is 0.9. To improve the robustness of the model parameters, Gaussian noise is randomly added to 1/30th of the data during the training of each epoch. At the same time, slight perturbation of the backpropagation gradient is performed with a probability of 0.001.

O\(^2\)-Bert effectively addresses challenges related to the “NULL” entity and overlapping entities, as previously stated. To verify that, we pick some sentences and list the results in Table 8. In Sentences 2 and 5, we can see that through the entity number prediction module, a “NULL” entity with its syntactic dependency word “worth” can be found. And the results of Sentences 3 and 4 indicate that the attention mechanism plays an important role in recognizing entities. Moreover, capturing the entity “beef hamburgers” in Sentence 5 shows that the entity starting annotation module and the entity length prediction module do work. However, there are still bad case. In Sentence 6, our model extracts the entity “fish” with the syntactic dependency words “not \(\dots\) came” , and thus thinks it negative. In Sentence 7, it’s too hard for our model to capture the description “Eight out of ten!”, so it simply finds some useless words like “comments”, “not”, and “true” our model fails to understand the statement “Eight out of ten!”, and make a judgement through “comments”, “not so true”..

To showcase the efficacy of the attention mechanism in the OSC task, we choose Sentences 1 and 4 as illustrative examples. We visualize the attention weights between entities and other words by heat map in Fig 5. The darker the color, the greater the attention weights. For Case 1, we can see that the word “incredible” has the highest attention weight among all words, and that’s because it has a syntactic dependency with “lava cake dessert”. Also, as an adjective, it expresses a strong sentiment. Similarly, in Case 2, our model’s heatmap demonstrates its heightened focus on the correlation between entities and their corresponding syntactic dependency words, aligning with our expectations.