The effect of guessing on item reliability under answer-until-correct scoring

Abstract

The answer-until-correct (AUC) procedure requires that examinees respond to a multiple-choice item until they answer it correctly. The examinee’s score on the item is then based on the number of responses required for the item. It was expected that the additional responses obtained under the AUC procedure would improve reliability by providing additional information on those examinees who fail to choose the correct alternative on their first attempt. However, when compared to the zero-one (ZO) scoring procedure, the AUC procedure has failed to yield consistent improvements in reliability. Using a modified version of Horst’s model for examinee behavior, this paper compares the effect of guessing on item reliability for the AUC procedure and the ZO procedure. The analysis shows that the relative efficiency of the two procedures depends strongly on the nature of the item alternatives and implies that the appropriate criteria for item selection are different for each procedure. Conflicting results reported for empirical comparisons of the reliabilities of the two procedures may result from a failure to control for the characteristics of the items.