Skip to main content
Top
Published in:

01-12-2023 | Original Article

FNR: a similarity and transformer-based approach to detect multi-modal fake news in social media

Authors: Faeze Ghorbanpour, Maryam Ramezani, Mohammad Amin Fazli, Hamid R. Rabiee

Published in: Social Network Analysis and Mining | Issue 1/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Many people today get their news from social media. It is possible to propagate news using textual, visual, or multi-modal information. The popularity of social networks and their wide use by people make them attractive platforms for spreading fake news. Detecting fake news is essential to preventing its spread. Fake news can be a false article or a genuine article with misleading visual information. Adding an actual image to trustworthy unrelated news can also create a fake news story. In this paper, we propose a novel and efficient similarity and transformer-based detection algorithm called Fake News Revealer (FNR), which uses text and images of news to detect fake news. The algorithm uses contrastive loss to consider text and image relations and transformer models to extract contextual and semantic features. According to experiments on two public social media news data sets, the FNR algorithm competes with conventional methods and state-of-the-art fake news detection algorithms by adding a novel mechanism without adding extra parameters or weights.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literature
go back to reference Akiba T, Sano S, Yanase T et al (2019) Optuna: a next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. ACM, pp 2623–2631 Akiba T, Sano S, Yanase T et al (2019) Optuna: a next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. ACM, pp 2623–2631
go back to reference Antol S, Agrawal A, Lu J et al (2015) Vqa: visual question answering. In: Proceedings of the IEEE international conference on computer vision, pp 2425–2433 Antol S, Agrawal A, Lu J et al (2015) Vqa: visual question answering. In: Proceedings of the IEEE international conference on computer vision, pp 2425–2433
go back to reference Bai Y, Mei J, Yuille AL et al (2021) Are transformers more robust than CNNs? In: Advances in neural information processing systems, pp 26831–26843. arXiv:2111.05464 Bai Y, Mei J, Yuille AL et al (2021) Are transformers more robust than CNNs? In: Advances in neural information processing systems, pp 26831–26843. arXiv:​2111.​05464
go back to reference Boididou C, Andreadou K, Papadopoulos S et al (2015) Verifying multimedia use at medieval 2015. MediaEval 3:7 Boididou C, Andreadou K, Papadopoulos S et al (2015) Verifying multimedia use at medieval 2015. MediaEval 3:7
go back to reference Chen J, Jia C, Zheng H et al (2022) Is multi-modal necessarily better? Robustness evaluation of multi-modal fake news detection. arXiv:2206.08788 Chen J, Jia C, Zheng H et al (2022) Is multi-modal necessarily better? Robustness evaluation of multi-modal fake news detection. arXiv:​2206.​08788
go back to reference Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics, vol 1. Association for Computational Linguistics, pp 4171–4186 Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics, vol 1. Association for Computational Linguistics, pp 4171–4186
go back to reference Dosovitskiy A, Beyer L, Kolesnikov A et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. Preprint at arXiv:2010.11929 Dosovitskiy A, Beyer L, Kolesnikov A et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. Preprint at arXiv:​2010.​11929
go back to reference Farajtabar M, Yang J, Ye X et al (2017) Fake news mitigation via point process-based intervention. In: International conference on machine learning. PMLR, pp 1097–1106 Farajtabar M, Yang J, Ye X et al (2017) Fake news mitigation via point process-based intervention. In: International conference on machine learning. PMLR, pp 1097–1106
go back to reference Fenn E, Ramsay N, Kantner J et al (2019) Nonprobative photos increase truth, like, and share judgments in a simulated social media environment. JARMAC 8:131–138 Fenn E, Ramsay N, Kantner J et al (2019) Nonprobative photos increase truth, like, and share judgments in a simulated social media environment. JARMAC 8:131–138
go back to reference Fukui A, Park DH, Yang D et al (2016) Multimodal compact bilinear pooling for visual question answering and visual grounding. Preprint at arXiv:1606.01847 Fukui A, Park DH, Yang D et al (2016) Multimodal compact bilinear pooling for visual question answering and visual grounding. Preprint at arXiv:​1606.​01847
go back to reference Gupta M, Zhao P, Han J (2012) Evaluating event credibility on twitter. In: Proceedings of the 2012 SIAM international conference on data mining. SIAM, pp 153–164 Gupta M, Zhao P, Han J (2012) Evaluating event credibility on twitter. In: Proceedings of the 2012 SIAM international conference on data mining. SIAM, pp 153–164
go back to reference He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
go back to reference Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. In: International conference on artificial neural networks. Springer, pp 44–51 Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. In: International conference on artificial neural networks. Springer, pp 44–51
go back to reference Jain V, Kaliyar RK, Goswami A et al (2022) AENeT: an attention-enabled neural architecture for fake news detection using contextual features. Neural Comput Appl 34(1):771–782CrossRef Jain V, Kaliyar RK, Goswami A et al (2022) AENeT: an attention-enabled neural architecture for fake news detection using contextual features. Neural Comput Appl 34(1):771–782CrossRef
go back to reference Jin Z, Cao J, Guo H, et al (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia. ACM, MM \(\acute{1}\)7, pp 795–816 Jin Z, Cao J, Guo H, et al (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia. ACM, MM \(\acute{1}\)7, pp 795–816
go back to reference Jwa H, Oh D, Park K et al (2019) exBAKE: automatic fake news detection model based on bidirectional encoder representations from transformers (BERT). Appl Sci 9(19):4062CrossRef Jwa H, Oh D, Park K et al (2019) exBAKE: automatic fake news detection model based on bidirectional encoder representations from transformers (BERT). Appl Sci 9(19):4062CrossRef
go back to reference Kaliyar RK, Goswami A, Narang P et al (2020) FNDNet—a deep convolutional neural network for fake news detection. Cogn Syst Res 61:32–44CrossRef Kaliyar RK, Goswami A, Narang P et al (2020) FNDNet—a deep convolutional neural network for fake news detection. Cogn Syst Res 61:32–44CrossRef
go back to reference Kaliyar RK, Goswami A, Narang P (2021a) EchoFakeD: improving fake news detection in social media with an efficient deep neural network. Neural Comput Appl 33(14):8597–8613CrossRef Kaliyar RK, Goswami A, Narang P (2021a) EchoFakeD: improving fake news detection in social media with an efficient deep neural network. Neural Comput Appl 33(14):8597–8613CrossRef
go back to reference Kaliyar RK, Goswami A, Narang P (2021b) FakeBERT: fake news detection in social media with a BERT-based deep learning approach. Multimed Tools Appl 80(8):11765–11788CrossRef Kaliyar RK, Goswami A, Narang P (2021b) FakeBERT: fake news detection in social media with a BERT-based deep learning approach. Multimed Tools Appl 80(8):11765–11788CrossRef
go back to reference Khattar D, Goud JS, Gupta M et al (2019) Mvae: multimodal variational autoencoder for fake news detection. In: The world wide web conference. ACM, pp 2915–2921 Khattar D, Goud JS, Gupta M et al (2019) Mvae: multimodal variational autoencoder for fake news detection. In: The world wide web conference. ACM, pp 2915–2921
go back to reference Khosla P, Teterwak P, Wang C et al (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673 Khosla P, Teterwak P, Wang C et al (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673
go back to reference Kumari R, Ekbal A (2021) AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst Appl 184(115):412 Kumari R, Ekbal A (2021) AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst Appl 184(115):412
go back to reference Kwon S, Cha M, Jung K et al (2013) Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th international conference on data mining. IEEE, pp 1103–1108 Kwon S, Cha M, Jung K et al (2013) Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th international conference on data mining. IEEE, pp 1103–1108
go back to reference Liu Y, Wu YF (2018) Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence Liu Y, Wu YF (2018) Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence
go back to reference Newman EJ, Garry M, Bernstein DM et al (2012) Nonprobative photographs (or words) inflate truthiness. Psychon Bull Rev 19:969–974CrossRef Newman EJ, Garry M, Bernstein DM et al (2012) Nonprobative photographs (or words) inflate truthiness. Psychon Bull Rev 19:969–974CrossRef
go back to reference O’Brien N, Latessa S, Evangelopoulos G et al (2018) The language of fake news: opening the black-box of deep learning based detectors. In: Workshop on “AI for Social Good”. NIPS O’Brien N, Latessa S, Evangelopoulos G et al (2018) The language of fake news: opening the black-box of deep learning based detectors. In: Workshop on “AI for Social Good”. NIPS
go back to reference Palani B, Elango S, Viswanathan KV et al (2022) CB-Fake: a multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT. Multimed Tools Appl 81:5587–5620CrossRef Palani B, Elango S, Viswanathan KV et al (2022) CB-Fake: a multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT. Multimed Tools Appl 81:5587–5620CrossRef
go back to reference Peters M, Neumann M, Iyyer M, Zettlemoyer l et al (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics. Human Language Technologies, pp 2227–2237 Peters M, Neumann M, Iyyer M, Zettlemoyer l et al (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics. Human Language Technologies, pp 2227–2237
go back to reference Ping Tian D et al (2013) A review on image feature extraction and representation techniques. Int J Multimed Ubiquitous Eng 8(4):385–396 Ping Tian D et al (2013) A review on image feature extraction and representation techniques. Int J Multimed Ubiquitous Eng 8(4):385–396
go back to reference Qi P, Cao J, Yang T et al (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp 518–527 Qi P, Cao J, Yang T et al (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp 518–527
go back to reference Radford A, Kim JW, Hallacy C et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763 Radford A, Kim JW, Hallacy C et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763
go back to reference Ren S, He K, Girshick R, et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, p 28 Ren S, He K, Girshick R, et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, p 28
go back to reference Ruchansky N, Seo S, Liu Y (2017) Csi: a hybrid deep model for fake news detection. In: Proceedings of the 2017 ACM on conference on information and knowledge management, pp 797–806 Ruchansky N, Seo S, Liu Y (2017) Csi: a hybrid deep model for fake news detection. In: Proceedings of the 2017 ACM on conference on information and knowledge management, pp 797–806
go back to reference Shu K, Sliva A, Wang S et al (2017) Fake news detection on social media: a data mining perspective. SIGKDD Explor Newsl 19:22–36CrossRef Shu K, Sliva A, Wang S et al (2017) Fake news detection on social media: a data mining perspective. SIGKDD Explor Newsl 19:22–36CrossRef
go back to reference Shu K, Wang S, Lee D et al (2020) Disinformation, misinformation, and fake news in social media. SpringerCrossRef Shu K, Wang S, Lee D et al (2020) Disinformation, misinformation, and fake news in social media. SpringerCrossRef
go back to reference Singh B, De S, Zhang Y et al (2015) Layer-specific adaptive learning rates for deep networks. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, pp 364–368 Singh B, De S, Zhang Y et al (2015) Layer-specific adaptive learning rates for deep networks. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, pp 364–368
go back to reference Singhal S, Shah RR, Chakraborty T et al (2019) Spotfake: a multi-modal framework for fake news detection. In: 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, pp 39–47 Singhal S, Shah RR, Chakraborty T et al (2019) Spotfake: a multi-modal framework for fake news detection. In: 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, pp 39–47
go back to reference Singhal S, Pandey T, Mrig S et al (2022) Leveraging intra and inter modality relationship for multimodal fake news detection. In: Companion Proceedings of the Web Conference, pp 726–734 Singhal S, Pandey T, Mrig S et al (2022) Leveraging intra and inter modality relationship for multimodal fake news detection. In: Companion Proceedings of the Web Conference, pp 726–734
go back to reference Song C, Ning N, Zhang Y et al (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manag 58(102):437 Song C, Ning N, Zhang Y et al (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manag 58(102):437
go back to reference Wang S, Chen Y, Zhuo J et al (2018a) Joint global and co-attentive representation learning for image-sentence retrieval. In: Proceedings of the 26th ACM international conference on Multimedia, pp 1398–1406 Wang S, Chen Y, Zhuo J et al (2018a) Joint global and co-attentive representation learning for image-sentence retrieval. In: Proceedings of the 26th ACM international conference on Multimedia, pp 1398–1406
go back to reference Wang Y, Ma F, Jin Z et al (2018b) EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining. ACM, KDD \(\acute{1}\)8, pp 849–857 Wang Y, Ma F, Jin Z et al (2018b) EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining. ACM, KDD \(\acute{1}\)8, pp 849–857
go back to reference Wang Y, Ma F, Wang H et al (2021) Multimodal emergent fake news detection via meta neural process networks. In: Proceedings of the 27th ACM SIGKDD conference on Knowledge Discovery & Data Mining. ACM, pp 3708–3716 Wang Y, Ma F, Wang H et al (2021) Multimodal emergent fake news detection via meta neural process networks. In: Proceedings of the 27th ACM SIGKDD conference on Knowledge Discovery & Data Mining. ACM, pp 3708–3716
go back to reference Wang B, Feng Y, Xiong X et al (2022a) Multi-modal transformer using two-level visual features for fake news detection. Appl Intell 2022:1–15 Wang B, Feng Y, Xiong X et al (2022a) Multi-modal transformer using two-level visual features for fake news detection. Appl Intell 2022:1–15
go back to reference Wang J, Mao H, Li H (2022b) FMFN: fine-grained multimodal fusion networks for fake news detection. Appl Sci 12(3):1093CrossRef Wang J, Mao H, Li H (2022b) FMFN: fine-grained multimodal fusion networks for fake news detection. Appl Sci 12(3):1093CrossRef
go back to reference Wolf T, Debut L, Sanh V et al (2020) Huggingface’s transformers: state-of-the-art natural language processing. Preprint at arXiv:1910.03771 Wolf T, Debut L, Sanh V et al (2020) Huggingface’s transformers: state-of-the-art natural language processing. Preprint at arXiv:​1910.​03771
go back to reference Wu B, Xu C, Dai X et al (2020) Visual transformers: token-based image representation and processing for computer vision. Preprint at arXiv:2006.03677 Wu B, Xu C, Dai X et al (2020) Visual transformers: token-based image representation and processing for computer vision. Preprint at arXiv:​2006.​03677
go back to reference Yenter A, Verma A (2017) Deep CNN-LSTM with combined kernels from multiple branches. In: 2017 IEEE 8th annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), pp 540–546 Yenter A, Verma A (2017) Deep CNN-LSTM with combined kernels from multiple branches. In: 2017 IEEE 8th annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), pp 540–546
Metadata
Title
FNR: a similarity and transformer-based approach to detect multi-modal fake news in social media
Authors
Faeze Ghorbanpour
Maryam Ramezani
Mohammad Amin Fazli
Hamid R. Rabiee
Publication date
01-12-2023
Publisher
Springer Vienna
Published in
Social Network Analysis and Mining / Issue 1/2023
Print ISSN: 1869-5450
Electronic ISSN: 1869-5469
DOI
https://doi.org/10.1007/s13278-023-01065-0

Premium Partner