The Impact of Local Item Dependence on Computer Adaptive Testing given Between and Within Testlet Adaptivity

Abstract

In educational tests, a group of questions related to a shared stimulus is called a testlet (e.g., a reading passage with multiple related questions). Use of testlets is very common in educational tests. Additionally, computerized adaptive testing (CAT) is a mode of testing where the test forms are created in real time tailoring to the test taker's previous responses. CATs have gained popularity because they produce a more accurate test score estimate within a shorter time with fewer questions. In this dissertation, the performance of CAT with testlets was studied. Conditional independence is an essential assumption of Item Response Models. However, when testlets are used, item responses to the items in the same testlet may not be independent since these items are associated with a common stimulus. In this case, the conditional independence assumption is violated, and this violation can affect the entire performance of CAT and lead to inaccurate test scores. Common practice for testlet based CATs is to select a testlet adaptively and administer all the items bundled in the testlet. However, the accuracy and efficiency aspects of CAT may not be fully realized with this approach of adaptivity. To increase the accuracy and efficiency in testlet-based CATs, between and within testlet adaptivity was studied. Other manipulated variables were the level of adaptivity, different testlet/item selection approaches, latent trait estimation methods, test length/number of testlets, and magnitude of testlet dependence. The performances of testlet based CATs with various scenarios were evaluated descriptively by running a series of simulations. One of the important findings of the study is that not only between testlets but also within testlets adaptation improved the performance of the CAT for more accurate and precise estimation of the latent traits. The results will serve as a baseline for future research and practice.