Group dependence of some reliability indices for mastery tests

Abstract

Reliability indices for mastery tests depend not only on true-score variance but also on mean and cutoff scores. This dependence was examined in the case of three decision-theoretic indices: (1) the coefficient of agreement; (2) kappa; and (3) the proportion of correct decisions. The binomial error model was assumed, with a two-parameter beta distribution for true scores. The reliability indices were computed at five values of the mean, four values of KR-21, and four cutoff scores. Results show that the dependence of kappa on mean and cutoff scores is opposite to that of the proportion of correct decisions, which is linearly related to average threshold loss. Moreover, kappa can be very small when most examinees are classified correctly. Thus, objections against the classical reliability coefficient apply even more strongly to kappa.