Previous research has studied the effects of different
methods of item-option weighting on the reliability
and the concurrent and predictive validity
of achievement tests. Generally, increases in reliability
are found, but with mixed results for validity.
This research attempted to interrelate several
methods of producing option weights (i.e., Guttman
internal and external weights and judges’
weights) and examined their effects on reliability
and on concurrent, predictive, and face validity.
Option weights to maximize reliability produced
cross-validated (N = 974) increases in Hoyt reliability
over rights-only scoring (.82 versus .58, respectively)
; decreases in correlations with other
achievement tests; few changes in predictive validity
; and a loss in face validity (i.e., some correct options
had lower weights than incorrect options).
Weights to maximize validity did not cross-validate
and led to a reduction in reliability and to mixed
validity results. Judges’ weights produced increases
in reliability and mixed results with validity. The
size of Guttman weights were shown to interact
with item-option and test characteristics. It was
concluded that option weighting offered limited, if
any, improvement over unit weighting.