The effect of different numbers of experts and
common items on the scaling of cutting scores
derived by experts’ judgments was investigated.
Four test forms were created from each of two
examinations; each form from the first examination
shared a block of items with one form from
the second examination. Small groups of experts
set standards on each using a modification of
Angoff’s (1971) method. Cutting score equivalents
were estimated for the matched forms using different
group sizes and numbers of common items;
they were compared with cutting score equivalents
based on score equating. Results showed that a
reduction in error is associated with using more
experts or having more items in common between
the two forms. For 25 or more common items and
five or more judges, the error was about one item
on a 100-item test. More than five experts or 25
common items made only a very small difference
in error. Index terms: cutting scores, equating,
expert judgment, standard setting.
Norcini, John, Shea, Judy & Grosso, Louis. (1991). The effect of numbers of experts and common items on cutting score equivalents based on expert judgment. Applied Psychological Measurement, 15, 241-246. doi:10.1177/014662169101500303
Norcini, John; Shea, Judy; Grosso, Louis.
The effect of numbers of experts and common items on cutting score equivalents based on expert judgment.
Retrieved from the University of Minnesota Digital Conservancy,
Content distributed via the University of Minnesota's Digital Conservancy may be subject to additional license and use restrictions applied by the depositor.