Estimating reliabilities of computerized adaptive tests

Abstract

This paper presents two methods for estimating the reliability of a computerized adaptive test (CAT) without using item response theory. The required data consist of CAT and paper-pencil (PP) scores from identical or equivalent samples, and scores for all examinees on one or more covariates. Multiple R's and communalities are used to compute the ratio of CAT and PP reliabilities. When combined with the PP reliability calculated by a conventional procedure, these ratios yield estimates of CAT reliability. Index terms: computerized adaptive testing, item response theory, predictive validity, reliability, tailored testing.