Comparison of the null distributions of weighted kappa and the C ordinal statistic

Abstract

It frequently occurs in psychological research that an investigator is interested in assessing the extent of interrater agreement when the data are measured on an ordinal scale. This monte carlo study demonstrates that the appropriate statistic to apply is weighted kappa with its revised standard error. The study also demonstrates that the minimal number of cases required for the valid application of weighted kappa varies between 20 and 100, depending upon the size of the ordinal scale. This contrasts with a previously cited large sample estimate of 200. Given the difficulty of obtaining sample sizes this large, the latter finding should be of some comfort to investigators who use weighted kappa to measure interrater consensus.