A procedure for determining the effect of testlet
nonequivalence on the operating characteristics of
a testlet-based computerized mastery test (CMT) is
introduced. The procedure involves estimating the
CMT decision rule twice-once with testlets treated
as equivalent and once with testlets treated as
nonequivalent. In the equivalent testlet mode, the
likelihood functions (LFS) estimated for specific
number-correct (NC) scores are assumed to be constant
across testlets and a single set of cutscores is
estimated for all testlets. In the nonequivalent
testlet mode, the LFS estimated for specific NC
scores are allowed to vary from one testlet to another
and a different set of cutscores is estimated for
each permutation of testlet presentation order.
Small differences between the estimated operating
characteristics of the equivalent testlet decision rule
and the nonequivalent testlet decision rule indicate
that the assumption of equivalent testlets was warranted.
This procedure is demonstrated with data
from the Architect Registration Examination.
Index terms: Bayesian methods, computerized mastery
testing, decision theory, item response theory, test