Skip to main content

2020 | OriginalPaper | Buchkapitel

Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias

verfasst von : David La Barbera, Kevin Roitero, Gianluca Demartini, Stefano Mizzaro, Damiano Spina

Erschienen in: Advances in Information Retrieval

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

News content can sometimes be misleading and influence users’ decision making processes (e.g., voting decisions). Quantitatively assessing the truthfulness of content becomes key, but it is often challenging and thus done by experts. In this work we look at how experts and non-expert assess truthfulness of content by focusing on the effect of the adopted judgment scale and of assessors’ own bias on the judgments they perform. Our results indicate a clear effect of the assessors’ political background on their judgments where they tend to trust content which is aligned to their own belief, even if experts have marked it as false. Crowd assessors also seem to have a preference towards coarse-grained scales, as they tend to use a few extreme values rather than the full breadth of fine-grained scales.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Fußnoten
1
The setup was reviewed and approved by the Human Research Ethics Committee at the University of Queensland.
 
2
In the original dataset, Pants on Fire is used; we preferred Lie to facilitate workers.
 
Literatur
1.
Zurück zum Zitat Bittman, L., Godson, R.: The KGB and Soviet Disinformation: An Insider’s View. Pergamon-Brassey’s (1985) Bittman, L., Godson, R.: The KGB and Soviet Disinformation: An Insider’s View. Pergamon-Brassey’s (1985)
2.
Zurück zum Zitat Cuzzocrea, A., Bonchi, F., Gunopulos, D. (eds.): Proceedings of the CIKM 2018 Workshops co-located with 27th ACM International Conference on Information and Knowledge Management (CIKM 2018), Torino, Italy, 22 October 2018, CEUR Workshop Proceedings, vol. 2482. CEUR-WS.org (2019). http://ceur-ws.org/Vol-2482 Cuzzocrea, A., Bonchi, F., Gunopulos, D. (eds.): Proceedings of the CIKM 2018 Workshops co-located with 27th ACM International Conference on Information and Knowledge Management (CIKM 2018), Torino, Italy, 22 October 2018, CEUR Workshop Proceedings, vol. 2482. CEUR-WS.org (2019). http://​ceur-ws.​org/​Vol-2482
4.
Zurück zum Zitat Han, L., Roitero, K., Maddalena, E., Mizzaro, S., Demartini, G.: On transforming relevance scales. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM) (2019) Han, L., Roitero, K., Maddalena, E., Mizzaro, S., Demartini, G.: On transforming relevance scales. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM) (2019)
5.
Zurück zum Zitat Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decis. Support Syst. 43(2), 618–644 (2007)CrossRef Jøsang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decis. Support Syst. 43(2), 618–644 (2007)CrossRef
6.
Zurück zum Zitat Kriplean, T., Bonnar, C., Borning, A., Kinney, B., Gill, B.: Integrating on-demand fact-checking with public dialogue. In: Proceedings of CSCW, pp. 1188–1199 (2014) Kriplean, T., Bonnar, C., Borning, A., Kinney, B., Gill, B.: Integrating on-demand fact-checking with public dialogue. In: Proceedings of CSCW, pp. 1188–1199 (2014)
11.
Zurück zum Zitat Roitero, K., Maddalena, E., Demartini, G., Mizzaro, S.: On fine-grained relevance scales. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, pp. 675–684. ACM, New York (2018). https://doi.org/10.1145/3209978.3210052 Roitero, K., Maddalena, E., Demartini, G., Mizzaro, S.: On fine-grained relevance scales. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, pp. 675–684. ACM, New York (2018). https://​doi.​org/​10.​1145/​3209978.​3210052
12.
Zurück zum Zitat Self, C.C.: Credibility. In: An Integrated Approach to Communication Theory and Research, pp. 449–470. Routledge (2014) Self, C.C.: Credibility. In: An Integrated Approach to Communication Theory and Research, pp. 449–470. Routledge (2014)
13.
Zurück zum Zitat Shearer, E., Gottfried, J.: News use across social media platforms 2017. Pew Research Center 7 (2017) Shearer, E., Gottfried, J.: News use across social media platforms 2017. Pew Research Center 7 (2017)
14.
Zurück zum Zitat Starbird, K.: Disinformation’s spread: bots, trolls and all of us. Nature 571, 449 (2019)CrossRef Starbird, K.: Disinformation’s spread: bots, trolls and all of us. Nature 571, 449 (2019)CrossRef
15.
Zurück zum Zitat Viviani, M., Pasi, G.: Credibility in social media: opinions, news, and health information—a survey. Wiley Interdis. Rev.: Data Min. Knowl. Discov. 7(5), e1209 (2017) Viviani, M., Pasi, G.: Credibility in social media: opinions, news, and health information—a survey. Wiley Interdis. Rev.: Data Min. Knowl. Discov. 7(5), e1209 (2017)
16.
Zurück zum Zitat Wang, W.Y.: “Liar, Liar Pants on Fire”: a new benchmark dataset for fake news detection. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 422–426 (2017) Wang, W.Y.: “Liar, Liar Pants on Fire”: a new benchmark dataset for fake news detection. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 422–426 (2017)
17.
Zurück zum Zitat Zubiaga, A., Ji, H.: Tweet, but verify: epistemic study of information verification on Twitter. Soc. Netw. Anal. Min. 4(1), 163 (2014)CrossRef Zubiaga, A., Ji, H.: Tweet, but verify: epistemic study of information verification on Twitter. Soc. Netw. Anal. Min. 4(1), 163 (2014)CrossRef
Metadaten
Titel
Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias
verfasst von
David La Barbera
Kevin Roitero
Gianluca Demartini
Stefano Mizzaro
Damiano Spina
Copyright-Jahr
2020
DOI
https://doi.org/10.1007/978-3-030-45442-5_26

Neuer Inhalt