In applied measurement, test scores are usually
transformed to decisions. Analogous to classical
test theory, the reliability of decisions has been defined
as the consistency of decisions on a test and a
retest or on two parallel tests. Coefficient kappa
(Cohen, 1960) is used for assessing the consistency
of decisions. This coefficient has been developed for
assessing agreement between nominal scales. It is
argued that the coefficient is not suited for assessing
consistency of decisions. Moreover, it is argued
that the concept consistency of decisions is not
appropriate for assessing the quality of a decision
procedure. It is proposed that the concept consistency
of decisions be replaced by the concept
optimality of the decision procedure. Two types of
optimality are distinguished. The internal optimality
is the risk of the decision procedure with respect
to the true score the test is measuring. The external
optimality is the risk of the decision procedure with
respect to an external criterion. For assessing the
optimality of a decision procedure, coefficient delta
(van der Linden & Mellenbergh, 1978), which can
be considered a standardization of the Bayes risk or
expected loss, can be used. Two loss functions are
dealt with: the threshold and the linear loss functions.
Assuming psychometric theory, coefficient
delta for internal optimality can be computed from
empirical data for both the threshold and the linear
loss functions. The computation of coefficient delta
for external optimality needs no assumption of psychometric
theory. For six tests coefficient delta as
an index for internal optimality is computed for
both loss functions; the results are compared with
coefficient kappa for assessing the consistency of
decisions with the same tests.
Mellenbergh, Gideon J & Van der Linden, Wim J. (1979). The internal and external optimality of decisions based on tests. Applied Psychological Measurement, 3, 257-273. doi:10.1177/014662167900300214
Mellenbergh, Gideon J.; Van der Linden, Wim J..
The internal and external optimality of decisions based on tests.
Retrieved from the University of Minnesota Digital Conservancy,
Content distributed via the University of Minnesota's Digital Conservancy may be subject to additional license and use restrictions applied by the depositor.