Top

Quality & Quantity

Published in:

01-03-2015

Fleiss’ kappa statistic without paradoxes

Authors: Rosa Falotico, Piero Quatto

Published in: Quality & Quantity | Issue 2/2015

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes.

previous article A control chart for an exponential distribution using multiple dependent state sampling

next article Development of a polar extreme method for use in partial least squares SEM

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Cicchetti, D.V., Feinstein, A.R.: High agreement but low kappa: II. Resolving the paradoxes. J. Clin. Epidemiol. 43(6), 551–558 (1990). doi:10.1016/0895-4356(90)90158-L CrossRef

Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960). doi:10.1177/001316446002000104 CrossRef

Dijkstra, L., van Eijnatten, F.M.: Agreement and consensus in a q-mode research design: an empirical comparison of measures, and an application. Qual. Quant. 43(5), 757–771 (2009). doi:10.1007/s11135-009-9249-4 CrossRef

Feinstein, A.R., Cicchetti, D.V.: High agreement but low kappa: I. the problems of two paradoxes. J. Clin. Epidemiol. 43(6), 543–549 (1990). doi:10.1016/0895-4356(90)90159-M CrossRef

Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378 (1971). doi:10.1037/h0031619 CrossRef

Fleiss, J.L., Levin, B., Paik, M.C.: Statistical methods for rates and proportions. Wiley, Hoboken (2003)CrossRef

Gwet, K.L.: Computing inter-rater reliability and its variance in the presence of high agreement. Br. J. Math. Stat. Psychol. 61(1), 29–48 (2008). doi:10.1348/000711006X126600 CrossRef

Harvey, A.G., Tang, N.K.: (Mis)perception of sleep in insomnia: a puzzle and a resolution. Psychol. Bull. 138(1), 77 (2012). doi:10.1037/a0025730 CrossRef

Lantz, C.A., Nebenzahl, E.: Behavior and interpretation of the k statistic: resolution of the two paradoxes. J. Clin. Epidemiol. 49(4), 431–434 (1996). doi:10.1016/0895-4356(95)00571-4 CrossRef

Markon, K.E., Chmielewski, M., Miller, C.J.: The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review. Psychol. Bull. 137(5), 856 (2011). doi:10.1037/a0023678 CrossRef

Mielke, P.J.W., Berry, K.J.: Permutation methods: a distance function approach. Springer, New York (2007)

Östlin, P., Wärneryd, B., Thorslund, M.: Should occupational codes be obtained from census data or from retrospective survey data in studies on occupational health? Soc. Indic. Res. 23(3), 231–246 (1990)CrossRef

Scott, W.A.: Reliability of content analysis: the case of nominal scale coding. Pub. Opin. Q. (1955). doi:10.1086/266577

Shao, J., Tu, D.: The jackknife and bootstrap. Springer, New York (1995)CrossRef

Shoukri, M.M.: Measures of interobserver agreement and reliability. Chapman & Hall, Boca Raton (2004)

Uttal, D.H., Meadow, N.G., Tipton, E., Hand, L.L., Alden, A.R., Warren, C., Newcombe, N.S.: The malleability of spatial skills: a meta-analysis of training studies. Psychol. Bull. 139(2), 352 (2013). doi:10.1037/a0028446 CrossRef

Title: Fleiss’ kappa statistic without paradoxes
Authors: Rosa Falotico
Piero Quatto
Publication date: 01-03-2015
Publisher: Springer Netherlands
Published in: Quality & Quantity / Issue 2/2015
Print ISSN: 0033-5177
Electronic ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-014-0003-1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 2/2015

Overly ambitious critics and the Medici Effect: a reply to Kampen and Tamás

A critical review of service excellence models: towards developing an integrated framework

Revisiting linkages between financial development, trade openness and economic growth in South Africa: fresh evidence from combined cointegration test

A webometric approach to policy analysis and management using exponential random graph models

Simulating interventions in graphical chain models for longitudinal data

A fuzzy set theory solution to combining Likert items into a single overall scale (or subscales)

Premium Partner