Item location effects and their implications for IRT equating and adaptive testing

Abstract

A context effect occurs when examinees’ item responding behavior is affected by the location of an item within a test. Recent advances in testing practice, most notably adaptive testing and certain innovative equating schemes, require items to be more invariant across intended usages than earlier methods. In this paper, location effects are identified as a form of multidimensionality, and examples of testing situations where location effects are important are described. Then, the susceptibility to item location effects of 10 item types from the Graduate Record Examination General Test is investigated by comparing the item difficulty parameters of sets of items across intended usages. Results are replicated using a second form of the test. Two of the 10 item types, analysis of explanations and logical diagrams, are clearly affected by item location in the population tested. One common item type, reading comprehension, appears to be affected somewhat by item context in this population. It is strongly advised that these item types not be used in an adaptive testing program without first assessing their susceptibility to location effects within the population (and subpopulations) of interest.