Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large

Abstract

The results of this computer simulation study indicate that the weighted kappa statistic, employing a standard error developed by Fleiss, Cohen, and Everitt (1969), holds for a large number of k categories of classification (e.g., 8 ≤ k ≤ 10). These data are entirely consistent with an earlier study (Cicchetti & Fleiss, 1977), which showed the same results for 3 ≤ k ≤ 7. The two studies also indicate that the minimal N required for the valid application of weighted kappa can be easily approximated by the simple formula 2k². This produces sample sizes that vary between a low of about 20 (when k = 3) to a high of about 200 (when k = 10). Finally, the range 3 ≤ k ≤ 10 should encompass most extant clinical scales of classification.