The effect of numbers of experts and common items on cutting score equivalents based on expert judgment

Abstract

The effect of different numbers of experts and common items on the scaling of cutting scores derived by experts’ judgments was investigated. Four test forms were created from each of two examinations; each form from the first examination shared a block of items with one form from the second examination. Small groups of experts set standards on each using a modification of Angoff’s (1971) method. Cutting score equivalents were estimated for the matched forms using different group sizes and numbers of common items; they were compared with cutting score equivalents based on score equating. Results showed that a reduction in error is associated with using more experts or having more items in common between the two forms. For 25 or more common items and five or more judges, the error was about one item on a 100-item test. More than five experts or 25 common items made only a very small difference in error. Index terms: cutting scores, equating, expert judgment, standard setting.