Reliability of test scores and decisions

Abstract

A criterion-referenced test can be viewed as testing either a continuous or a binary variable, and the scores on a test can be used as measurements of the variable or to make decisions (e.g., pass or fail). Recent work on the reliability of criterion-referenced tests has focused on the use of scores from tests of continuous variables for decision-making purposes. This work can be categorized according to type of loss function-threshold, linear, or quadratic. It is the loss function that is used either explicitly or implicitly to evaluate the goodness of the decisions that are made on the basis of the test scores. The literature in which a threshold loss function is employed can be further subdivided according to whether the goodness of decisions is assessed as the probability of making an erroneous decision or as a measure of the consistency of decisions over repeated testing occasions. This review points to the need for simple procedures by which to estimate the probability of decision errors.