Fake news detection based on a hybrid BERT and LightGBM models

Social media systems have attained extraordinary levels of achievement, opened unforeseen opportunities, and been changing the way news is disseminated, produced, and consumed, thereby becoming indispensable platforms used for a variety of applications. The dramatic development of social media characteristics in various platforms made the migration to social media platforms greatly desired even by reputable/well-known news organizations and agencies.

The usage of social media is accompanied by a dramatic increase in the threats of fake news and online misinformation. Fake news is fabricated stories that have similar characteristics to news media content but differs in organizational process or purpose in an effort to deceive readers [1]. Fake news is constantly growing through social media, online blogs, magazines, forums, and newspapers, making it difficult to find trustworthy news sources. Social media has evolved into an ideal platform for anybody to manufacture, distort, and propagate fake news. Because of the ease with which information can be created and distributed. For instance, according to Facebook [2], malicious entities contributed to less than one-tenth of 1% of civic content published on the network.

In recent years, fake news has been blamed for deepening political division and party strife, and also, it has a significant effect on topics, such as vaccination, nutrition, and stock values. According to a study by Ohio State University academics [3], false news most likely contributed to the decline in Hillary Clinton’s popularity on election day. The study implies that roughly 4% of Barack Obama’s 2012 supporters were discouraged from voting for Clinton in 2016 by their belief in false news articles. Another example [4], fake rumors about Tesla buying a lithium mining caused its shares to increase by nearly \(250\%\). According to [5], tweets about the COVID-19 pandemic contained inaccurate or unverified information, with \(24.8\%\) and \(17.4\%\), respectively. The falsity of news has a considerable (positive/negative) influence on readers. The development of effective analytical tools for online content is crucial to prevent having a negative impact on social, economic, and political life.

Manual fact-checking needs regular and manual updates by crowdsourced people or a small set of experts and also is unable to perform automated learning [6]. Machine-learning and deep-learning approaches have demonstrated accurate predictions and insights to handle a variety of complicated problems [7‐9]. Developing automatic, trustworthy, and accurate solutions for detecting fake news is a hot research area. The detection of fake news is a challenging natural language processing (NLP) problem that is concerned with text classification to distinguish between fake and real. NLP has advanced significantly over the past few years. Transformer-based pre-trained language models are now the state-of-the-art approach for many NLP problems [10‐12]. However, studying fake news detection using transformer-based models is still limited.

In this paper, we propose a hybrid model that combines a transformer-based architecture with a light gradient boosting machine (LightGBM) for fake news detection. Bidirectional encoder representations from transformers (BERT) are used to process the news articles and extract the text representation. BERT is one of the most effective language representation models, producing excellent results across a wide range of NLP applications. We add a LightGBM at the end of the BERT model to produce a hybrid classification that predicts “true” or “false” about news content. LightGBM is a high-efficiency gradient boosting framework that uses tree-based learning techniques. It supports parallel, distributed, and GPU learning and provides faster training speed. The proposed method is evaluated on three fake news datasets. We compare the performance of the proposed method with multinomial Naive Bayes (MNB), linear regression (LR), linear support vector machines (LSVM), and long short-term memory (LSTM) using different word embedding techniques. The proposed method achieves superior performance compared to the state-of-the-art. The main contributions of this work can be summarized as:The rest of the paper is organized as follows. In the section “Related work”, the related works on fake news detection are discussed. The section “Proposed method” presents the details of the proposed method. The experimental results, as well as a comparison to other methods, are discussed in the section “Experimental results and discussion”. Finally, the conclusion is drawn in the section “Conclusion”.

Various machine-learning-based techniques have been developed to detect fake news. These methods can be categorized into traditional approaches and deep-learning approaches.

The traditional machine-learning approaches, such as MNB, LR, LSVM, Decision Tree (DT), and extreme gradient boosting (XGBoost). In Ref. [13], Ahmed et al. used n-gram analysis with term frequency-inverted document frequency (TF-IDF) to extract features for detecting fake news. They studied and compared six different machine-learning techniques. The LSVM model achieved the highest accuracy score of 92% on ISOT dataset. However, it is unclear whether this approach can generalize well to other datasets. Similarly, in Ref. [14], the author studied five different machine-learning models with the same embedding technique. LSVM and XGBoost achieved the best results. In Ref. [15], Ozbay and Alatas applied the term frequency (TF) weighting method and document-term matrix to extract features from texts and then investigated 23 supervised models to identify fake news. DT is given the best results according to this study. Similarly, in Ref. [16], the same authors adapt salp swarm optimization (SSO) and grey wolf optimizer (GWO) algorithms instead of machine learning methods for the fake news detection problem. In Ref. [17], Kansal studied underlying writing style based on part-of-speech (POS) tags to detect fake news. The POS features are fed into the XGBoost to create the first model. An average of TF-IDF weights and Word2Vec word embeddings are fed into the multi-layer perception and then ensemble with the first model to get the final prediction. However, these methods require large amounts of labeled data and sophisticated models to achieve high accuracy. Moreover, these methods can struggle to adapt to new types of fake news, which are constantly evolving.

Deep learning is regarded as one of the hottest topics in the fields of machine learning, and artificial intelligence due to its learning capabilities from given data. Methods, such as convolutional neural networks (CNN) and LSTM, are becoming increasingly popular. In Ref. [18], Nasir et al. proposed a hybrid model of two deep-learning models: CNN and LSTM. CNN is used to extract features and then feed these features as input to LSTM to learn the long-term dependencies. Words in the text are represented as vectors using the global vectors for word representation (GloVe) pre-trained word embeddings. The method achieved better performance compared to seven traditional machine-learning methods. In Ref. [19], they introduced a deep-learning model called (FNDNet), such that the GloVe is used as a pre-trained word embedding. Three convolutional-pooling layers are used to extract features from the word embedding vectors, and then, these features are concatenated and fed into other convolutional-pooling layers followed by two dense layers for classification. In Ref. [20], Sastrawan et al. applied back-translation as a data augmentation to reduce imbalance classes and then carried out pre-processing step on the augmented data. They compared the performance of using pre-trained word embedding (Word2Vec, GloVe, and fastText) on pre-processed data. They tested three different deep-learning models, namely CNN, bidirectional LSTM, and ResNet to extract features from word embedding vectors and then detect fake news. In Ref. [21], an optimized deep-learning model called OPCNN-FAKE is proposed based on CNN. It consists of an embedding layer to create embedding vectors, a dropout layer to enhance regularization, a convolution-pooling layer to extract features and reduce the feature map, a flattened layer to produce a one-dimensional vector, and an output layer that takes the output of the previous layer and decides if the input text is fake or real. They compared their performance with Recurrent Neural Networks (RNN), LSTM, and six regular machine-learning techniques. In Ref. [22], Yang et al. introduced a CNN-based model (TI-CNN) that combines the explicit and latent features from both text and image information for detecting fake news. They collected a dataset for this purpose and compare their method to other models, such as LSTM, CNN, and GRU. Similarly, in Ref. [23], authors proposed a multi-modal coupled CNN model that fuses both text and image data to classify online news. However, these methods may not able to capture long-range contextual information. Moreover, the word embedding is not representing the context-specific information in the text.

Recently, the transformer model has emerged as one of the key highlights of deep-learning advances in NLP. In Ref. [24], authors compared five transformer models (XLNet, BERT, RoBERTa, DistilBERT, and ALBERT) to detect fake news. They run the experiments with various combinations of hyperparameters. The results showed comparable results of the transformer models. In Ref. [25], Kaliyar et al. proposed a deep-learning model (FakeBERT) based on BERT. The model is using BERT to get the word embeddings and feed it into three parallel layers of convolution with different kernel sizes. This BERT-based model showed better performance than other machine-learning-based methods. In Ref. [26], authors introduced a transformer-based deep learning model based on BART and RoBERTa to distinguish between different types of news articles. Each output of BART and RoBERTa embeddings is fed into a branch of LSTM and CNN-based architecture and then concatenated and passed through another LSTM and CNN-based architecture to get the results. In Ref. [27], Qazi et al. used an attention-based transformer model to detect fake and real news. They compared their performance with a hybrid CNN model that integrates text and meta-data. The transformer model showed better accuracy compared to the hybrid CNN model. However, many of these methods used deep-learning-based techniques combined with transformer models, which can be computationally expensive and require large amounts of training data. In contrast, the proposed method utilizes the LightGBM, which is much faster and requires less computational power.

Fake news is a growing problem in today’s society, with the potential to mislead and harm individuals, organizations, and even entire nations. Existing approaches to fake news detection often rely on manual fact-checking or rule-based systems, which can be time-consuming and limited in their coverage. In this paper, we propose a novel method based on fine-tuned BERT and LightGBM to improve the accuracy and efficiency of fake news detection. Our method leverages the power of BERT to capture complex linguistic patterns and the efficiency of LightGBM to optimize feature space and classification. By combining these two techniques, we aim to achieve better performance compared to other methods.

Figure 1 shows an overall view of the proposed system. We apply some pre-processing steps on the input text to get rid of unnecessary parts of the data. The input text is then tokenized to individual characters, subwords, and words that are good enough to represent the input data to be fed to “fine-tuned” BERT. We extract the text embedding from the special token [CLS] of the last three hidden layers of the BERT. We train the LightGBM model on the concatenated embedding vectors to get the final prediction.

In this paper, a novel hybrid model is proposed for automatically detecting false news. The proposed method combines BERT and LightGBM models to form an effective classification system. The LightGBM model is built on top of a pre-trained BERT word embedding model. The results show that the proposed method produces more accurate results, with an accuracy of 99.88%, 96.94%, and 99.06% on ISOT, TI-CNN, and FNC datasets, respectively. The proposed hybrid method outperforms different combinations of word embedding techniques and classification approaches, as well as current state-of-the-art methods on different real-world fake news datasets.