Full-metric concurrent calibration for the development of CAT item banks.

Abstract

The development of an item bank to be used in a CAT frequently requires the administration of subsets of different test items to different groups of examinees, along with a set of common items, which requires linking to place parameters of a bank onto a common scale. Whereas previous linking research has mainly focused on methods of linking, the present study looked at the resulting metric of the linking process. Study 1 investigated the performance of full-metric concurrent calibration by varying the number of groups to be equated, the equivalence in θ among examinees, and the number of common items. The statistical characteristics of anchor items in full-metric calibration were examined in Study 2. Results indicated that as more groups were linked, full-metric concurrent calibration tended to have poorer recovery (i.e., larger bias, RMSE), especially for non-equivalent groups. The number of common items seems not important in full-metric concurrent calibration. Better linking was obtained when the average of anchor item discriminations was higher than that of the total test discrimination and the average of anchor item difficulties was the same as that of the total test difficulties. Larger standard deviation of anchor item discrimination and anchor item difficulty parameters, on the other hand, decreased parameter recovery in full-metric concurrent calibration. Limitations and implications of the current study are also discussed.