It frequently occurs in psychological research
that an investigator is interested in assessing the extent
of interrater agreement when the data are
measured on an ordinal scale. This monte carlo
study demonstrates that the appropriate statistic to
apply is weighted kappa with its revised standard
error. The study also demonstrates that the minimal
number of cases required for the valid application
of weighted kappa varies between 20 and 100,
depending upon the size of the ordinal scale. This
contrasts with a previously cited large sample estimate
of 200. Given the difficulty of obtaining sample
sizes this large, the latter finding should be of
some comfort to investigators who use weighted
kappa to measure interrater consensus.
Cicchetti, Domenic V & Fleiss, Joseph L. (1977). Comparison of the null distributions of weighted kappa and the C ordinal statistic. Applied Psychological Measurement, 1, 195-201. doi:10.1177/014662167700100206
Cicchetti, Domenic V.; Fleiss, Joseph L..
Comparison of the null distributions of weighted kappa and the C ordinal statistic.
Retrieved from the University of Minnesota Digital Conservancy,
Content distributed via the University of Minnesota's Digital Conservancy may be subject to additional license and use restrictions applied by the depositor.