Improving efficiency of cognitive diagnosis by using diagnostic items and adaptive testing.

Abstract

Diagnosing student use of problem-solving strategies (cognitive rules) is one of the major concerns of cognitive diagnostic assessment (CDA). With such information, subsequent instruction can be effectively tailored to meet individual student learning needs. For this purpose, student responses to assessment items should be interpretable in terms of cognitive rules. However, most of the current implementations of CDA still treat student responses dichotomously (correct versus incorrect), leading to substantial loss of diagnostic information. Diagnostic items are multiple choice items in which response options are explicitly associated with certain cognitive rules, and considered as a potentially useful tool for more efficient CDA. These considerations lead to the following research questions: (a) to what extent does the use of diagnostic items improve efficiency of cognitive diagnosis from treating student responses dichotomously, and (b) what characteristics of diagnostic items are more responsible for efficiency improvement than others? The present research comprised two studies and approached these questions quantitatively, using (a) a latent class model as a psychometric model for diagnostic items, (b) the number of items administered to reach a diagnosis as a measure of efficiency of cognitive diagnosis, and (c) adaptive testing simulations in which optimal items are sequentially selected for each examinee. Study 1 examined efficiency improvement using simulated responses to hypothetical diagnostic items whose characteristics were varied systematically. Study 2 examined efficiency improvement found in existing real response data on Siegler's Balance Scale Task. Both studies supported the use of diagnostic items in that they substantially improved efficiency of cognitive diagnosis, although several item characteristics had differential effects on the efficiency. Limitations to these studies included validity of the latent class model and simulation settings. Based on these limitations, future research should be directed to further understanding of student response behaviors relevant to cognitive rule usage and the corresponding extension of the current latent class model for diagnostic items.