Multidimensional mastery testing with CAT

Abstract

Computerized mastery testing (CMT) is a subset of computerized adaptive testing (CAT) with the intent of assigning examinees to one of two, mutually exclusive, categories. Most mastery testing algorithms have been designed to classify examinees on either side of a cut-point in one dimension, but many psychological attributes are inherently multidimensional. Little psychometric work has generalized these unidimensional algorithms to multidimensional traits. When classifying examinees in multidimensional space, practitioners must choose a cut-point function that separates a mastery region from a non-mastery region. The possible cut-point functions include one in which a linear combination of ability across dimensions must exceed a threshold and one in which each ability must exceed a threshold irrespective of any other ability. Moreover, two major components of every classification test are choosing successive questions and determining when a classification decision should be made. One frequently used stopping rule in unidimensional mastery testing is the Sequential Probability Ratio Test (SPRT), in which a classification is made either when the log-likelihood test statistic is sufficiently large or when the maximum number of items has been reached. Due to inefficiencies in the SPRT, alternative algorithms have been proposed, such as the Generalized Likelihood Ratio (GLR), and the SPRT with Stochastic Curtailment (SCSPRT). The current study explores properties of unidimensional classification testing algorithms, generalizes unidimensional methods to multidimensional mastery tests, and then tests many of the multidimensional procedures. Most of the multidimensional algorithms yield relatively efficient and accurate multidimensional classifications. However, some multidimensional classification problems, such as classifying examinees with respect to a linear classification bound function, are more robust to poor choices in the item bank or adaptive testing algorithms. Based on results from the main study in this thesis, a follow-up study is proposed to better combine sequential classification methods with those based on directly quantifying incorrect classifications. I conclude by discussing consequences of the results for practitioners in realistic mastery testing situations.