A comparison of item calibration media in computerized adaptive testing

Abstract

A concern in computerized adaptive testing is whether data for calibrating items can be collected from either a paper-and-pencil (P&P) or a computer administration of the items. Fixed blocks of power test items were administered by computer to one group of examinees and by P&P to a second group. These data were used to obtain computer-based and P&P-based three-parameter logistic model parameters of the items. Then each set of parameters was used to estimate item response theory pseudo-adaptive scores for a third group of examinees who had received all of the items by computer. The effect of medium of administration of the calibration items was assessed by comparative analyses of the adaptive scores using structural modeling. The results support the use of item parameters calibrated from either P&P or computer administrations for use in computerized adaptive power tests. The calibration medium did not appear to alter the constructs measured by the adaptive test or the reliability of the adaptive test scores Irndex. terms: computerized adaptive testing, item calibration, item parameter estimation, item response theory, medium of administration, trait level estimation.