Computerized mastery testing with nonequivalent testlets

Abstract

A procedure for determining the effect of testlet nonequivalence on the operating characteristics of a testlet-based computerized mastery test (CMT) is introduced. The procedure involves estimating the CMT decision rule twice-once with testlets treated as equivalent and once with testlets treated as nonequivalent. In the equivalent testlet mode, the likelihood functions (LFS) estimated for specific number-correct (NC) scores are assumed to be constant across testlets and a single set of cutscores is estimated for all testlets. In the nonequivalent testlet mode, the LFS estimated for specific NC scores are allowed to vary from one testlet to another and a different set of cutscores is estimated for each permutation of testlet presentation order. Small differences between the estimated operating characteristics of the equivalent testlet decision rule and the nonequivalent testlet decision rule indicate that the assumption of equivalent testlets was warranted. This procedure is demonstrated with data from the Architect Registration Examination. Index terms: Bayesian methods, computerized mastery testing, decision theory, item response theory, test equivalence, testlets.