nach oben

Quality & Quantity

Erschienen in:

20.03.2023

A Bayesian index of association: comparison with other measures and performance

verfasst von: Anton Oleinik

Erschienen in: Quality & Quantity | Ausgabe 1/2024

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The article discusses a Bayesian measure of association, B-index, and compares it with the other existing measures of agreement, association, and similarity, both chance-corrected and non-corrected: Scott’s π, Krippendorff’s α, Cohen’s κ, Bennett, Alpert & Goldstein’s S, Cosine similarity, and the Jaccard similarity coefficient. PageRank adapted to particularities of annotation is also added to this list. Two versions of B-index are considered: with the informative and non-informative priors. An algorithm for calculating B-index written in pseudocode is provided. Particular attention is devoted to the uses of those measures in content analysis, communication studies, computational linguistics, psychology, computer science and network science. Real-world data gathered using an online platform for content analysis allowed comparing the behavior of all eight measures included in the scope of analysis. Three short texts (164 data points/sentences in total) were coded by 66 annotators. The behaviors of B-index with the non-informative prior and Bennett, Alpert & Goldstein’s S have some common patterns.

Vorheriger Artikel Correction: Farmers’ permanence in peripheral rural areas. place-based values as drivers of resistance beyond the decline

Nächster Artikel Do money and guilt primes affect the likelihood of theft by taking? Findings from a visualized scenario study

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

For example, users of QDA Miner have a choice between Krippendorff’s α and Scott’s π adjustment. The latter “yields results that are similar to those obtained using Cohen’s Kappa,” according to the developers of this computer program for content analysis.

available at https://www.thinkmate.org/en/. The platform has been created and run by the author with the support from Smolny College of St. Petersburg University (Russia) and Memorial University of Newfoundland and Labrador (Canada) at earlier stages of its development.

“If men define situations as real, they are real in their consequences” (Merton 1995).

If the number of other annotators is even, their split decisions (e.g., with respect to Fragments 5, 6, 7 and 9) are accounted for separately. Their number is uniformly distributed across four cells in Table 6d.

The project is available for viewing by registered users at https://thinkmate.org/en/. A registered user can consult texts that were content analyzed and calculate online all measures reported in this article.

In classical statistics it is acknowledged that the number of trials “does not have to be particularly large for the shape to be approximately normal, n ≥ 25 is sufficient” (Bolstad and Curran 2017). Indeed, the distribution of values of B-index calculated for a group of 66 annotators tends to be normal (n = 66), whereas that of PageRank scores (for λ = 0.1, 0.2 and 0.3) is skewed. The distribution of values of all measures included in the scope of comparison and calculated for pairs of annotators also tends to be normal, except for Cosine similarity and the Jaccard similarity coefficient (n = 2,145).

Annotator BA in Fig. 3, where the relevant dot lies close to the center too.

Annotator P in Fig. 3.

Annotators E, G, H, BH, BN and BO in Fig. 3 (they form a cluster).

Annotator Y in Fig. 3.

Annotator W in Fig. 3.

For two data points A = (a₁; a₂;:::; a_n) and B = (b₁; b₂;:::; b_n) the Euclidean distance is \({d}_{AB}^{E}=\sum_{k=1}^{n}\sqrt{{({a}_{k}-{b}_{k})}^{2}}\) (Fortunato 2010; Murphy 2012).

http://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection

http://disi.unitn.it/moschitti/corpora.htm

http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/

4 vCPU Cores, 8 GB RAM.

Amati, G.: Frequentist and Bayesian approach to information retrieval. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds.), Advances in Information Retrieval: 28th European Conference on IR Research (ECIR 2006), London, UK, April 10–12, 2006. Proceedings, pp. 13–24. Springer (2006)

Arrow, K.J.: A difficulty in the concept of social welfare. J Polit. Econ. 58(4), 328–346 (1950)CrossRef

Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistic. Comput. Linguist. 34(4), 555–596 (2008)CrossRef

Barabási, A.-L.: Linked. Perseus, Cambridge, MA (2002)

Barabási, A.-L.: Network Science. Cambridge University Press, Cambridge (2016)

Basu, S., Banerjee, M., Sen, A.: Bayesian inference for kappa from single and multiple studies. Biometrics 56(2), 577–582 (2000)CrossRef

Ben-Gal, I.: Bayesian networks. In: Ruggeri, F., Kenett, R.S., Faltin, F.W. (eds.) Encyclopedia of statistics in quality and reliability. Wiley (2008)

Bennett, E., Alpert, R., Goldstein, A.C.: Communications through limited-response questioning. Public Opin. q. 18(3), 303–308 (1954)CrossRef

Benoit, K., Conway, D., Lauderdale, B.E., Laver, M., Mikhaylov, S.: Crowd-sourced text analysis: reproducible and agile production of political data. Am. Polit. Sci. Rev. 110(2), 278–295 (2016)CrossRef

Berman, J.J.: Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann, Waltham, MA (2013)

Bernard, H.R.: Social Research Methods, 2nd edn. Sage, Thousand Oaks, CA (2013)

Bilić, P.: Search algorithms, hidden labour and information control. Big Data Soc 3(1) (2016)

Bolstad, W.M., Curran, J.M.: Introduction to Bayesian statistics, 3rd edn. Wiley, Hoboken, NJ (2017)

Brin, S., Motwani, R., Page, L., Winograd, T.: What can you do with a web in your pocket? Bull. IEEE Comput. Soc. Tech. Comm. Data Eng. 21, 37–47 (1998)

Broemeling, L.D.: Bayesian Methods for Measures of Agreement. Chapman & Hall/CRC, Boca Raton, FL (2009)CrossRef

Clarke, B., Sun, D.: Reference priors under the chi-squared distance. Sankhyā Indian J Stat Ser A 59(2), 215–231 (1997)

Cooil, B., Rust, R.T.: Reliability and expected loss: a unifying principle. Psychometrika 59(2), 203–216 (1994)CrossRef

Cooil, B., Rust, R.T.: General estimators for the reliability of qualitative data. Psychometrika 60(2), 199–220 (1995)CrossRef

Craggs, R., McGee Wood, M.: Evaluating discourse and dialogue coding schemes. Comput. Linguist. 31(3), 289–296 (2005)CrossRef

Dijkstra, L., Van Eijnatten, F.M.: Agreement and consensus in a Q-mode research design: an empirical comparison of measures, and an application. Qual. Quant. 43(5), 757–771 (2009)CrossRef

DiMaggio, P.: Adapting computational text analysis to social science (and vice versa). Big Data Soc. 2(2), 1–5 (2015)CrossRef

DiMaggio, P., Nag, M., Blei, D.: «Exploiting affinities between topic modeling and the sociological perspective on culture: application to newspaper coverage of U.S. government arts funding. Poetics 41(6), 570–606 (2013)CrossRef

Dourado, Í.C., Galante, R., Gonçalves, M.A., Torres, R.S.: Bag of textual graphs (BoTG): a general graph-based text representation model. J. Am. Soc. Inf. Sci. 70(8), 817–829 (2019)

Evangelopoulos, N., Zhang, X., Prybutok, V.R.: Latent semantic analysis: five methodological recommendations. Eur. J. Inf. Syst. 21(1), 70–86 (2012)CrossRef

Evans, M., McIntosh, W., Lin, J., Cates, C.: Recounting the courts? Applying automated content analysis to enhance empirical legal research. J. Empir. Leg. Stud. 4(4), 1007–1039 (2007)CrossRef

Fortunato, S.: Community detection in graphs. Phys. Rep. 486(3–5), 75–174 (2010)CrossRef

Gleich, D.F.: PageRank beyond the web. SIAM Rev. 57(3), 321–363 (2015)CrossRef

Goodman, L.A., Kruskal, W.H.: Measures of association for cross classifications. J. Am. Stat. Assoc. 49(268), 732–764 (1954)

Green, N.: A Bayesian network coding scheme for annotating biomedical information presented to genetic counseling clients. J. Biomed. Inform. 38(2), 130–144 (2005)CrossRef

Grimmer, J., Stewart, B.M.: Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit. Anal. 21(3), 267–297 (2013)CrossRef

Han, L., Zhang, G., Yong, B., He, Q., Feng, F., Zhou, Q.: Statistical study of characteristics of online reading behavior networks in university digital library. World Wide Web 22(3), 1175–1187 (2019)CrossRef

Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)CrossRef

Henry, T.R., Banks, D., Owens-Oas, D., Chai, C.: Modeling community structure and topics in dynamic text networks. J. Classif. 36(2), 322–349 (2019)CrossRef

Hopkins, D.J., King, G.: A method of automated nonparametric content analysis for social science. Am. J. Polit. Sci. 54(1), 229–247 (2010)CrossRef

Huang, L., Milne, D., Frank, E., Witten, I.H.: Learning a concept-based document similarity measure. J. Am. Soc. Inform. Sci. Technol. 63(8), 1593–1608 (2012)CrossRef

Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2(1), 193–218 (1985)CrossRef

Hutter, M., Lloyd, J.W., Ng, K.S., Uther, W.T.B.: Probabilities on sentences in an expressive logic. J. Appl. Log. 11(4), 386–420 (2013)CrossRef

Jaccard, P.: The distribution of the flora in the Alpine zone. New Phytol. 2(3), 205–219 (1912)

Jaynes, E.T.: Probability Theory: The logic of Science. Cambridge University Press, Cambridge (2003)CrossRef

Jimmy, J.L., Loe, K.F., Zhang, H.J.: Robust face detection in airports. EURASIP J. Appl. Signal Process. 4, 503–509 (2004)

Jurafsky, D., Martin, J.H.: Speech and Language Processing, draft of 3rd edition (forthcoming). https://web.stanford.edu/~jurafsky/slp3/

Ketler, R.: Analysis of type I and II error rates of Bayesian and frequentist parametric and nonparametric two-sample hypothesis tests under preliminary assessment of normality. Comput. Stat. (2020)

Krippendorff, K.: Content Analysis: An Introduction to Its Methodology, 2nd edn. Sage, Thousand Oaks, CA (2004a)

Krippendorff, K.: Measuring the reliability of qualitative text analysis data. Qual. Quant. 38(6), 787–800 (2004b)CrossRef

Krippendorff, K.: A quadrilogy for (big) data reliabilities. Commun. Methods Meas. 15(3), 165–189 (2021)CrossRef

Kruschke, J.K.: Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan, 2nd edn. Elsevier, London (2015)

Kruschke, J.K., Aguinis, H., Joo, H.: The time has come: Bayesian methods for data analysis in the organizational sciences. Organ. Res. Methods 15(4), 722–752 (2012)CrossRef

Labatut, V.: Generalized measures for the evaluation of community detection methods. Int. J. Soc. Netw. Anal. Min. SNAM 2(1), 44–63 (2015)

Le, T., Clarke, B.: On the interpretation of ensemble classifiers in terms of Bayes classifiers. J. Classif. 35(2), 198–229 (2018)CrossRef

Leiva, F.M., Ríos, F.J.M., Martínez, T.L.: Assessment of interjudge reliability in the open-ended questions coding process. Qual. Quant. 40(4), 519–537 (2006)CrossRef

Lemke, M., Niekler, A., Schaal, G.S., Wiedemann, G.: Content analysis between quality and quantity. Datenbank-Spektrum 15(1), 7–14 (2015)CrossRef

Ligtvoet, R.: Exact one-sided Bayes factors for 2 by 2 contingency tables. J. Classif. 34(3), 465–472 (2017)CrossRef

Lotman, Y.: Universe of the Mind: A Semiotic Theory of Culture. Indiana University Press, Bloomington (1990)

Lynch, S.M.: Introduction to Applied Bayesian Statistics and Estimation for Social Scientists. Springer, New York (2007)CrossRef

Mannens, E., Coppens, S., De Pessemier, T., Dacquin, H., Van Deursen, D., De Sutter, R., Van de Walle, R.: Automatic news recommendations via aggregated profiling. Multimed. Tools Appl. 63(2), 407–425 (2013)CrossRef

Mathet, Y.: The agreement measure γ_cat a complement to γ focused on categorization of a continuum. Comput. Linguist. 43(3), 661–681 (2017)CrossRef

Mathet, Y., Widlöcher, A., Métivier, J.-P.: The unified and holistic method gamma (γ) for inter-annotator agreement measure and alignment. Comput. Linguist. 41(3), 437–479 (2015)CrossRef

Merton, R.K.: The thomas theorem and the Matthew effect. Soc. Forces 74(2), 379–424 (1995)CrossRef

Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)

Oleinik, A.: Mixing quantitative and qualitative content analysis: triangulation at work. Qual. Quant. 45(4), 859–873 (2011)

Oleinik, A.: Detection of opinion communities with the help of chance-corrected measures of agreement. SN Comput. Sci. 1, 136 (2020)

Oleinik, A.: Relevance in Web search: between content, authority and popularity. Qual. Quant. 56, 173–194 (2022)

Oleinik, A., Popova, I., Kirdina, S., Shatalova, T.: On the choice of measures of reliability and validity in the content-analysis of texts. Qual. Quant. 48(5), 2703–2718 (2014)

Perrault, W.D., Leigh, L.E.: Reliability of nominal data based on qualitative judgments. J. Mark. Res. 26(2), 135–148 (1989)CrossRef

Rand, W.M.: Objective criteria for the evaluation of clustering methods. J. Am. Stat. Assoc. 66(336), 846–850 (1971)CrossRef

Savoy, J.: Text representation strategies: an example with the state of the union addresses. J. Am. Soc. Inf. Sci. 67(8), 1858–1870 (2016)

Scharkow, M.: Thematic content analysis using supervised machine learning: an empirical evaluation using German online news. Qual. Quant. 47(2), 761–773 (2013)CrossRef

Scott, W.A.: Reliability of content analysis: the case of nominal scale coding. Public Opin. q. 19(3), 321–325 (1955)CrossRef

Siegel, S., Castellan, N.J.: Nonparametric Statistics for the Behavioural Sciences, 2nd edn. McGraw Hill, New York (1988)

Simon, H.A.: Rationality as process and as product of thought. Am. Econ. Rev. 68(2), 2–16 (1978)

Sprenger, J.: Statistics between inductive logic and empirical science. J. Appl. Log. 7(2), 239–250 (2009)CrossRef

Su, L.Y.-F., Cacciatore, M.A., Liang, X., Brossard, D., Scheufele, D.A., Xenos, M.A.: Analyzing public sentiments online: combining human- and computer-based content analysis. Inf. Commun. Soc. 20(3), 406–427 (2017)CrossRef

Tang, L., Liu, H.: Community Detection and Mining in Social Media. Morgan & Claypool, San Rafael, CA (2010)

Thelwall, M., Kousha, K.: Goodreads: a social network site for book readers. J. Am. Soc. Inf. Sci. 68(4), 972–983 (2017)

Van der Linden, W., Lewis, C.: Bayesian checks on cheating on tests. Psychometrika 80(3), 689–706 (2015)CrossRef

Van Rooij, I., Kwisthout, J., Blokpoel, M., Szymanik, J., Wareham, T., Toni, I.: Intentional communication: computationally easy or difficult? Front. Neurosci. 5, art.52 (2011)

Vellino, A., Alberts, I.: Assisting the appraisal of e-mail records with automatic classification. Rec. Manag. J. 26(3), 293–313 (2016)

Vinh, N.X., Epps, J., Bailey, J.: Information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. J. Mach. Learn. Res. 11(95), 2837–2854 (2010)

Wang, X., Tao, T., Sun, J.-T., Shakery, A., Zhai, C.: DirichletRank: solving the zero-one gap problem of pagerank. ACM Transact. Inform. Syst. 26(2), art10 (2008)

Wang, G., He, X., Ishuga, C.I.: HAR-SI: A novel hybrid article recommendation approach integrating with social information in scientific social network. Knowl.-Based Syst. 148, 85–99 (2018)CrossRef

Warner, R.M.: Applied Statistics: From Bivariate Through Multivariate Techniques, 2nd edn. Sage, Thousand Oaks, CA (2013)

Warrens, M.J.: On similarity coefficients for 2×2 tables and correction for chance. Psychometrika 73(3), 487–502 (2008)CrossRef

Weller, S.C.: Cultural consensus theory: applications and frequently asked questions. Field Methods 19(4), 339–368 (2007)CrossRef

Yang, Q.: A novel recommendation system based on semantics and context awareness. Computing 100(8), 809–823 (2018)CrossRef

Youness, G., Saporta, G.: Comparing partitions of two sets of units based on the same variables. Adv. Data Anal. Classif. 4(1), 53–64 (2010)CrossRef

Zhai, C., Massung, S.: Text Data Management and Analysis: A Practical Introduction to Information Retrieval and Text Mining. ACM Books and Morgan & Claypool Publishers, New York (2016)

Zhang, M., Wang, W., Li, X.: A Paper recommender for scientific literatures based on semantic concept similarity. In: Buchanan, G., Masoodian, M., Cunningham S.J. (eds.), Digital Libraries: Universal and Ubiquitous Access to Information. 11th International Conference on Asian Digital Libraries, ICADL 2008, Bali, Indonesia, December 2–5, 2008. Proceedings, 359–cx362 (2008)

Zhao, X., Feng, G.C., Ao, S.H., Liu, P.L.: Interrater reliability estimators tested against true interrater reliabilities. BMC Med. Res. Methodol. 22(1), 232 (2022)CrossRef

Titel: A Bayesian index of association: comparison with other measures and performance
verfasst von: Anton Oleinik
Publikationsdatum: 20.03.2023
Verlag: Springer Netherlands
Erschienen in: Quality & Quantity / Ausgabe 1/2024
Print ISSN: 0033-5177
Elektronische ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-023-01639-2

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 1/2024

Charting fields and spaces quantitatively: from multiple correspondence analysis to categorical principal components analysis

Competitiveness, corruption, and income inequalities: approaching the ‘Janus’ face of development with simultaneous equation modelling

Global mapping of seawater desalination research: A bibliometric analysis of research trends from 1980–2022

Farmers’ permanence in peripheral rural areas. Place-based values as drivers of resistance beyond the decline

Revisiting policy combinations under IS–LM–EE framework introducing capacity utilization

What are teacher relational skills? A defining study using a bottom-up modified Delphi method

Premium Partner