Top

Multimedia Systems

Published in:

31-03-2022 | Special Issue Paper

TMIF: transformer-based multi-modal interactive fusion for automatic rumor detection

Authors: Jiandong Lv, Xingang Wang, Cuiling Shao

Published in: Multimedia Systems | Issue 5/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The rapid development of social media platforms has made them one of the most important news sources. While it provides people with convenient real-time communication channels, fake news and rumors are also spread rapidly through social media platforms, misleading the public and even causing bad social impact. In view of the slow speed and poor consistency of artificial rumor detection, we propose an end-to-end automatic rumor detection model named TMIF, which is based on transformer to map multi-modal feature representations to the same data domain for fusion. It can capture the multi-level dependencies among multi-modal content while reducing the impact of multi-modal heterogeneity differences. We validated it on two multi-modal rumor detection datasets and proved the superior performance and early detection performance of the proposed model.

previous article Optimized multimedia data through computationally intelligent algorithms

next article A MADDPG-based multi-agent antagonistic algorithm for sea battlefield confrontation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Kapoor, K.K., Tamilmani, K., Rana, N.P., Patil, P., Dwivedi, Y.K., Nerur, S.: Advances in social media research: past, present and future. Inf. Syst. Front. 20(3), 531–558 (2018). https://doi.org/10.1007/s10796-017-9810-yCrossRef

Jurgens, M., Helsloot, I.: The effect of social media on the dynamics of (self) resilience during disasters: a literature review. J. Conting. Crisis Manag. 26(1), 79–88 (2018). https://doi.org/10.1111/1468-5973.12212CrossRef

Ghani, N.A., Hamid, S., Hashem, I.A.T., Ahmed, E.: Social media big data analytics: a survey. Comput. Hum. Behav. 101, 417–428 (2019). https://doi.org/10.1016/j.chb.2018.08.039CrossRef

Monti, F., Frasca, F., Eynard, D., Mannion, D., Bronstein, M.M.: Fake news detection on social media using geometric deep learning. arXiv:1902.06673 (2019)

Appel, G., Grewal, L., Hadi, R., Stephen, A.T.: The future of social media in marketing. J. Acad. Mark. Sci. 48(1), 79–95 (2020). https://doi.org/10.1007/s11747-019-00695-1CrossRef

Richard, K.: Predicting the future with social media. Int. J. Sci. Soc. 3(1), 33–39 (2021)CrossRef

Alalwan, A.A., Rana, N.P., Dwivedi, Y.K., Algharabat, R.: Social media in marketing: a review and analysis of the existing literature. Telemat. Inform. 34(7), 1177–1190 (2017). https://doi.org/10.1016/j.tele.2017.05.008CrossRef

Mheidly, N., Fares, J.: Leveraging media and health communication strategies to overcome the covid-19 infodemic. J. Public Health Policy 41(4), 410–420 (2020). https://doi.org/10.1057/s41271-020-00247-wCrossRef

Gao, J., Zheng, P., Jia, Y., Chen, H., Mao, Y., Chen, S., Wang, Y., Fu, H., Dai, J.: Mental health problems and social media exposure during covid-19 outbreak. Plos One (2020). https://doi.org/10.1371/journal.pone.0231924CrossRef

10.

Wu, D., Cui, Y.: Disaster early warning and damage assessment analysis using social media data and geo-location information. Decis. Support Syst. 111, 48–59 (2018). https://doi.org/10.1016/j.dss.2018.04.005CrossRef

11.

Lu, Y.-J., Li, C.-T.: GCAN: graph-aware co-attention networks for explainable fake news detection on social media. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 505–514. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.48

12.

Alkhodair, S.A., Ding, S.H., Fung, B.C., Liu, J.: Detecting breaking news rumors of emerging topics in social media. Inf. Process. Manag. 57(2), 102018 (2020). https://doi.org/10.1016/j.ipm.2019.02.016CrossRef

13.

Alzanin, S.M., Azmi, A.M.: Detecting rumors in social media: a survey. Procedia Comput. Sci. 142, 294–300 (2018). https://doi.org/10.1016/j.procs.2018.10.495CrossRef

14.

Choi, D., Chun, S., Oh, H., Han, J., Kwon, T., et al.: Rumor propagation is amplified by echo chambers in social media. Sci. Rep. 10(1), 1–10 (2020). https://doi.org/10.1038/s41598-019-57272-3CrossRef

15.

Shahsavari, S., Holur, P., Wang, T., Tangherlini, T.R., Roychowdhury, V.: Conspiracy in the time of corona: automatic detection of emerging covid-19 conspiracy theories in social media and the news. J. Comput. Soc. Sci. 3(2), 279–317 (2020). https://doi.org/10.1007/s42001-020-00086-5CrossRef

16.

Cao, J., Guo, J., Li, X., Jin, Z., Guo, H., Li, J.: Automatic rumor detection on microblogs: A survey. arXiv:1807.03505 (2018)

17.

Pathak, A.R., Mahajan, A., Singh, K., Patil, A., Nair, A.: Analysis of techniques for rumor detection in social media. Procedia Comput. Sci. 167, 2286–2296 (2020). https://doi.org/10.1016/j.procs.2020.03.281CrossRef

18.

Meel, P., Vishwakarma, D.K.: Fake news, rumor, information pollution in social media and web: a contemporary survey of state-of-the-arts, challenges and opportunities. Expert Syst. Appl. 153, 112986 (2020). https://doi.org/10.1016/j.eswa.2019.112986CrossRef

19.

Ma, J., Gao, W., Mitra, P., Kwon, S., Jansen, B.J., Wong, K.-F., Cha, M.: Detecting rumors from microblogs with recurrent neural networks. In: 25th International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 3818–3824 (2016)

20.

Lin, X., Liao, X., Xu, T., Pian, W., Wong, K.-F.: Rumor detection with hierarchical recurrent convolutional neural network. In: CCF International Conference on Natural Language Processing and Chinese Computing, pp. 338–348 (2019)

21.

Chen, T., Li, X., Yin, H., Zhang, J.: Call attention to rumors: deep attention based recurrent neural networks for early rumor detection. In: Ganji, M, Rashidi, L, Fung, BCM, Wang, C (eds.) Trends and Applications in Knowledge Discovery and Data Mining. Lecture Notes in Artificial Intelligence, vol. 11154, pp. 40–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04503-6_4

22.

Singh, J.P., Kumar, A., Rana, N.P., Dwivedi, Y.K.: Attention-based lstm network for rumor veracity estimation of tweets. Inf. Syst. Front. (2020). https://doi.org/10.1007/s10796-020-10040-5CrossRef

23.

Ma, J., Gao, W., Wong, K.-F.: Rumor detection on twitter with tree-structured recursive neural networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 1980–1989 (2018)

24.

Guo, H., Cao, J., Zhang, Y., Guo, J., Li, J.: Rumor detection with hierarchical social attention network. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM ’18, pp. 943–951. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3269206.3271709

25.

Li, Q., Zhang, Q., Si, L.: Rumor detection by exploiting user credibility information, attention and multi-task learning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1173–1179. Association for Computational Linguistics, Florence (2019). https://doi.org/10.18653/v1/P19-1113

26.

Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia. MM ’17, pp. 795–816. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3123266.3123454

27.

Khattar, D., Goud, J.S., Gupta, M., Varma, V.: Mvae: Multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference. WWW ’19, pp. 2915–2921. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3308558.3313552

28.

Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., Gao, J.: Eann: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’18, pp. 849–857. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3219819.3219903

29.

Zhang, H., Fang, Q., Qian, S., Xu, C.: Multi-modal knowledge-aware event memory network for social media rumor detection. In: Proceedings of the 27th ACM International Conference on Multimedia. MM ’19, pp. 1942–1951. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3343031.3350850

30.

Duc Tuan, N.M., Quang Nhat Minh, P.: Multimodal fusion with bert and attention mechanism for fake news detection. In: 2021 RIVF International Conference on Computing and Communication Technologies, pp. 1–6 (2021). https://doi.org/10.1109/RIVF51545.2021.9642125

31.

Chen, J., Wu, Z., Yang, Z., Xie, H., Wang, F.L., Liu, W.: Multimodal fusion network with latent topic memory for rumor detection. In: 2021 IEEE International Conference on Multimedia and Expo, pp. 1–6 (2021). https://doi.org/10.1109/ICME51207.2021.9428404

32.

Sharma, S., Sharma, R.: Identifying possible rumor spreaders on twitter: A weak supervised learning approach. In: 2021 International Joint Conference on Neural Networks, pp. 1–8 (2021). https://doi.org/10.1109/IJCNN52387.2021.9534185

33.

Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)

34.

Ilić, S., Marrese-Taylor, E., Balazs, J., Matsuo, Y.: Deep contextualized word representations for detecting sarcasm and irony. In: Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 2–7. Association for Computational Linguistics, Brussels (2018). https://doi.org/10.18653/v1/W18-6202

35.

Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., Hon, H.-W.: Unified language model pre-training for natural language understanding and generation. In: Wallach, H, Larochelle, H, Beygelzimer, A, d’Alche-Buc, F, Fox, E, Garnett, R (eds.) Advances in Neural Information Processing Systems, vol. 32 (2019)

36.

Lee, J.-S., Hsiang, J.: Patent classification by fine-tuning Bert language model. World Patent Inf. 61, 101965 (2020). https://doi.org/10.1016/j.wpi.2020.101965CrossRef

37.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I, Luxburg, UV, Bengio, S, Wallach, H, Fergus, R, Vishwanathan, S, Garnett, R (eds.) Advances in Neural Information Processing Systems, vol. 30 (2017)

38.

Dey, R., Salem, F.M.: Gate-variants of gated recurrent unit (GRU) neural networks. In: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems, pp. 1597–1600 (2017). https://doi.org/10.1109/MWSCAS.2017.8053243

39.

Hara, K., Saito, D., Shouno, H.: Analysis of function of rectified linear unit used in deep learning. In: 2015 International Joint Conference on Neural Networks, pp. 1–8 (2015). https://doi.org/10.1109/IJCNN.2015.7280578

40.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

41.

Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386CrossRef

42.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetMATH

43.

Boididou, C., Papadopoulos, S., Kompatsiaris, Y., Schifferes, S., Newman, N.: Challenges of computational verification in social multimedia. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 743–748 (2014). https://doi.org/10.1145/2567948.2579323

44.

Diederik, K., Jimmy, B., et al.: Adam: A method for stochastic optimization. arXiv:1412.6980, pp. 273–297 (2014)

Title: TMIF: transformer-based multi-modal interactive fusion for automatic rumor detection
Authors: Jiandong Lv
Xingang Wang
Cuiling Shao
Publication date: 31-03-2022
Publisher: Springer Berlin Heidelberg
Published in: Multimedia Systems / Issue 5/2023
Print ISSN: 0942-4962
Electronic ISSN: 1432-1882
DOI: https://doi.org/10.1007/s00530-022-00916-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 5/2023

Hardest and semi-hard negative pairs mining for text-based person search with visual–textual attention

Image and audio caps: automated captioning of background sounds and images using deep learning

Unsupervised single image dehazing with generative adversarial network

A MADDPG-based multi-agent antagonistic algorithm for sea battlefield confrontation

Context-guided coarse-to-fine detection model for bird nest detection on high-speed railway catenary

Spatial attention-guided deformable fusion network for salient object detection