nach oben

Erschienen in:

2025 | OriginalPaper | Buchkapitel

Did You Tell a Deadly Lie? Evaluating Large Language Models for Health Misinformation Identification

verfasst von : Surendrabikram Thapa, Kritesh Rauniyar, Hariram Veeramani, Aditya Shah, Imran Razzak, Usman Naseem

Erschienen in: Web Information Systems Engineering – WISE 2024

Verlag: Springer Nature Singapore

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The rapid spread of health misinformation online poses significant challenges to public health, potentially leading to confusion, undermining trust in health authorities, and hindering effective health interventions. Large Language Models (LLMs) have shown promise in various natural language processing tasks, including misinformation detection. However, their effectiveness in identifying health-specific misinformation has not been extensively benchmarked. This study evaluates the performance of seven state-of-the-art LLMs - GPT-3.5, GPT-4, Gemini, Flan-T5 XL, Gemma, LLaMA-2, and Mistral - on the task of health misinformation detection across four datasets (Monkeypox-V1, Monkeypox-V2, COVID-19, and CoAID). The models were tested under five different settings: zero-shot classification, 5-shot random examples, 10-shot random examples, 5-shot sampled examples, and 10-shot sampled examples. Performance was evaluated using macro F1-score, and inter-model agreement was assessed using Cohen’s Kappa and Fleiss’ Kappa scores. By comprehensively benchmarking these LLMs, this study aims to determine which models excel in particular scenarios and provide insights into their potential for combating health misinformation in online environments.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-agent LLMs

Nächstes Kapitel Native vs Non-native Language Prompting: A Comparative Analysis

Alenezi, M.N., Alqenaei, Z.M.: Machine learning in detecting covid-19 misinformation on twitter. Future Internet 13(10), 244 (2021)CrossRef

Baek, J., Aji, A.F., Saffari, A.: Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136 (2023)

Baktash, J.A., Dawodi, M.: GPT-4: a review on advancements and opportunities in natural language processing. arXiv preprint arXiv:2305.03195 (2023)

Beltagy, I., Peters, M.E., Cohan, A.: Longformer: the long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

Bhattacharjee, A., Moraffah, R., Garland, J., Liu, H.: Zero-shot LLM-guided Counterfactual Generation for Text. arXiv preprint arXiv:2405.04793 (2024)

Boissonneault, D., Hensen, E.: Fake news detection with large language models on the liar dataset (2024)

Bojjireddy, S., Chun, S.A., Geller, J.: Machine learning approach to detect fake news, misinformation in covid-19 pandemic. In: DG. O2021: The 22nd Annual International Conference on Digital Government Research, pp. 575–578 (2021)

Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

Cao, Y., et al.: Can large language models detect misinformation in scientific news reporting? arXiv preprint arXiv:2402.14268 (2024)

10.

Chae, Y., Davidson, T.: Large language models for text classification: from zero-shot learning to fine-tuning. Open Science Foundation (2023)

11.

Chang, Y., et al.: A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15(3), 1–45 (2024)CrossRef

12.

Chen, C., Shu, K.: Can LLM-generated misinformation be detected? arXiv preprint arXiv:2309.13788 (2023)

13.

Chen, X., et al.: How robust is GPT-3.5 to predecessors? A comprehensive study on language understanding tasks. arXiv preprint arXiv:2303.00293 (2023)

14.

Chung, H.W., et al.: Scaling instruction-finetuned language models. J. Mach. Learn. Res. 25(70), 1–53 (2024)

15.

Cui, L., Lee, D.: Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint arXiv:2006.00885 (2020)

16.

Di Sotto, S., Viviani, M.: Health misinformation detection in the social web: an overview and a data science approach. Int. J. Environ. Res. Public Health 19(4), 2173 (2022)CrossRef

17.

Eastin, M.S.: Credibility assessments of online health information: the effects of source expertise and knowledge of content. J. Comput.-Mediated Commun. 6(4), JCMC643 (2001)

18.

Freeman, K.S., Spyridakis, J.H.: An examination of factors that affect the credibility of online health information. Tech. Commun. 51(2), 239–263 (2004)

19.

Ghenai, A., Mejova, Y.: Catching zika fever: application of crowdsourcing and machine learning for tracking health misinformation on twitter. arXiv preprint arXiv:1707.03778 (2017)

20.

Goli, A., Singh, A.: Can LLMs capture human preferences? arXiv preprint arXiv:2305.02531 (2023)

21.

Gül, İ., Lebret, R., Aberer, K.: Stance detection on social media with fine-tuned large language models. arXiv preprint arXiv:2404.12171 (2024)

22.

Hayawi, K., Shahriar, S., Serhani, M.A., Taleb, I., Mathew, S.S.: Anti-vax: a novel twitter dataset for covid-19 vaccine misinformation detection. Public Health 203, 23–30 (2022)CrossRef

23.

Ilie, V.I., Truică, C.O., Apostol, E.S., Paschke, A.: Context-aware misinformation detection: a benchmark of deep learning architectures using word embeddings. IEEE Access 9, 162122–162146 (2021)CrossRef

24.

Jafri, F.A., Rauniyar, K., Thapa, S., Siddiqui, M.A., Khushi, M., Naseem, U.: Chunav: analyzing Hindi hate speech and targeted groups in Indian election discourse. ACM Trans. Asian Low-Resour. Lang. Inf. Process. (2024)

25.

Jafri, F.A., Siddiqui, M.A., Thapa, S., Rauniyar, K., Naseem, U., Razzak, I.: Uncovering political hate speech during Indian election campaign: a new low-resource dataset and baselines. arXiv preprint arXiv:2306.14764 (2023)

26.

Jiang, A.Q., et al.: Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

27.

Joshi, G., et al.: Explainable misinformation detection across multiple social media platforms. IEEE Access 11, 23634–23646 (2023)CrossRef

28.

Laban, P., et al.: LLMs as factual reasoners: insights from existing benchmarks and beyond. arXiv preprint arXiv:2305.14540 (2023)

29.

Lemire, M., Paré, G., Sicotte, C., Harvey, C.: Determinants of internet use as a preferred source of information on personal health. Int. J. Med. Informatics 77(11), 723–734 (2008)CrossRef

30.

Li, G., et al.: Re-search for the truth: multi-round retrieval-augmented large language models are strong fake news detectors. arXiv preprint arXiv:2403.09747 (2024)

31.

Li, Y., Wu, Y., Li, J., Liu, S.: Prompting large language models for zero-shot domain adaptation in speech recognition. In: 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–8. IEEE (2023)

32.

Ling, C., et al.: Beyond one-model-fits-all: a survey of domain specialization for large language models. arXiv preprint arXiv:2305.18703 (2023)

33.

Lippmann, P., Spaan, M., Yang, J.: Exploring LLMs as a source of targeted synthetic textual data to minimize high confidence misclassifications. arXiv preprint arXiv:2403.17860 (2024)

34.

Liu, Y., et al.: Understanding LLMs: a comprehensive overview from training to inference. arXiv preprint arXiv:2401.02038 (2024)

35.

Liu, Y., Yu, K., Wu, X., Qing, L., Peng, Y.: Analysis and detection of health-related misinformation on Chinese social media. IEEE Access 7, 154480–154489 (2019)CrossRef

36.

Liu, Z., Liu, B., Thompson, P., Yang, K., Jain, R., Ananiadou, S.: Conspemollm: conspiracy theory detection using an emotion-based large language model. arXiv preprint arXiv:2403.06765 (2024)

37.

Lu, Y., et al.: Collective human behavior in cascading system: discovery, modeling and applications. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 297–306. IEEE (2018)

38.

Naeem, S., Ali, A., Anam, S., Ahmed, M.M.: An unsupervised machine learning algorithms: comprehensive review. Int. J. Comput. Digit. Syst. (2023)

39.

Nasir, A., Sharma, A., Jaidka, K.: LLMs and finetuning: benchmarking cross-domain performance for hate speech detection. arXiv preprint arXiv:2310.18964 (2023)

40.

Pahune, S., Chandrasekharan, M.: Several categories of large language models (LLMs): a short survey. arXiv preprint arXiv:2307.10188 (2023)

41.

Pan, W., Liu, D., Fang, J.: An examination of factors contributing to the acceptance of online health misinformation. Front. Psychol. 12, 630268 (2021)CrossRef

42.

Parnami, A., Lee, M.: Learning from few examples: a summary of approaches to few-shot learning. arXiv preprint arXiv:2203.04291 (2022)

43.

Patwa, P., et al.: Fighting an infodemic: COVID-19 fake news dataset. In: Chakraborty, T., Shu, K., Bernard, H.R., Liu, H., Akhtar, M.S. (eds.) CONSTRAINT 2021. CCIS, vol. 1402, pp. 21–29. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73696-5_3CrossRef

44.

Paynter, J., et al.: Evaluation of a template for countering misinformation-real-world autism treatment myth debunking. PLoS ONE 14(1), e0210746 (2019)CrossRef

45.

Qaiser, S., Ali, R.: Text mining: use of TF-IDF to examine the relevance of words to documents. Int. J. Comput. Appl. 181(1), 25–29 (2018)

46.

Rathje, S., Mirea, D.M., Sucholutsky, I., Marjieh, R., Robertson, C., Van Bavel, J.J.: GPT is an effective tool for multilingual psychological text analysis (2023)

47.

Safarnejad, L., Xu, Q., Ge, Y., Chen, S.: A multiple feature category data mining and machine learning approach to characterize and detect health misinformation on social media. IEEE Internet Comput. 25(5), 43–51 (2021)CrossRef

48.

Santu, S.K.K., Feng, D.: Teler: a general taxonomy of LLM prompts for benchmarking complex tasks. arXiv preprint arXiv:2305.11430 (2023)

49.

Savelka, J., Ashley, K.D., Gray, M.A., Westermann, H., Xu, H.: Can GPT-4 support analysis of textual data in tasks requiring highly specialized domain expertise? arXiv preprint arXiv:2306.13906 (2023)

50.

Sbaffi, L., Rowley, J.: Trust and credibility in web-based health information: a review and agenda for future research. J. Med. Internet Res. 19(6), e218 (2017)CrossRef

51.

Sen, T., Das, A., Sen, M.: Hatetinyllm: hate speech detection using tiny large language models. arXiv preprint arXiv:2405.01577 (2024)

52.

Shah, S.B., et al.: Navigating the web of disinformation and misinformation: large language models as double-edged swords. IEEE Access (2024)

53.

Shiwakoti, S., Thapa, S., Rauniyar, K., Shah, A., Bhandari, A., Naseem, U.: Analyzing the dynamics of climate change discourse on twitter: a new annotated corpus and multi-aspect classification. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 984–994 (2024)

54.

Sinaga, K.P., Yang, M.S.: Unsupervised k-means clustering algorithm. IEEE Access 8, 80716–80727 (2020)CrossRef

55.

Søe, S.O.: A unified account of information, misinformation, and disinformation. Synthese 198(6), 5929–5949 (2021)MathSciNetCrossRef

56.

Swire-Thompson, B., Lazer, D., et al.: Public health and online misinformation: challenges and recommendations. Annu. Rev. Public Health 41(1), 433–451 (2020)CrossRef

57.

Team, G., et al.: Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

58.

Thapa, S., Rauniyar, K., Shiwakoti, S., Poudel, S., Naseem, U., Nasim, M.: Nehate: large-scale annotated data shedding light on hate speech in Nepali local election discourse. In: ECAI 2023, pp. 2346–2353. IOS Press (2023)

59.

Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

60.

Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

61.

Wang, Y., McKee, M., Torbica, A., Stuckler, D.: Systematic literature review on the spread of health-related misinformation on social media. Soc. Sci. Med. 240, 112552 (2019)CrossRef

62.

Wani, A., Joshi, I., Khandve, S., Wagh, V., Joshi, R.: Evaluating deep learning approaches for covid19 fake news detection. In: Chakraborty, T., Shu, K., Bernard, H.R., Liu, H., Akhtar, M.S. (eds.) CONSTRAINT 2021. CCIS, vol. 1402, pp. 153–163. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73696-5_15CrossRef

63.

Warner, E.L., Basen-Engquist, K.M., Badger, T.A., Crane, T.E., Raber-Ramsey, M.: The online cancer nutrition misinformation: a framework of behavior change based on exposure to cancer nutrition misinformation. Cancer 128(13), 2540–2548 (2022)CrossRef

64.

Wright, C., et al.: Effects of brief exposure to misinformation about e-cigarette harms on twitter: a randomised controlled experiment. BMJ Open 11(9), e045445 (2021)CrossRef

65.

Xing, F.: Designing heterogeneous LLM agents for financial sentiment analysis. arXiv preprint arXiv:2401.05799 (2024)

66.

Yang, J., et al.: Harnessing the power of LLMs in practice: a survey on chatgpt and beyond. ACM Trans. Knowl. Discov. Data 18(6), 1–32 (2024)CrossRef

67.

Zhao, Y., Da, J., Yan, J.: Detecting health misinformation in online health communities: incorporating behavioral features into machine learning based approaches. Inf. Process. Manag. 58(1), 102390 (2021)CrossRef

Titel: Did You Tell a Deadly Lie? Evaluating Large Language Models for Health Misinformation Identification
verfasst von: Surendrabikram Thapa
Kritesh Rauniyar
Hariram Veeramani
Aditya Shah
Imran Razzak
Usman Naseem
Verlag: Springer Nature Singapore
Buch: Web Information Systems Engineering – WISE 2024
Print ISBN: 978-981-9605-75-0

Electronic ISBN: 978-981-9605-76-7

Copyright-Jahr: 2025
DOI: https://doi.org/10.1007/978-981-96-0576-7_29

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"