Cognitive diagnosis for multiple-choice responses: nonparametric classification method, Q-matrix theory, and computerized adaptive testing

Abstract

The multiple-choice (MC) item format has been widely used in educational assessments across diverse content domains. MC items purportedly allow for collecting richer diagnostic information. The effectiveness and economy of administering MC items may have further contributed to their popularity not just in educational assessment. The MC item format has also been adapted to the cognitive diagnosis (CD) framework. Early approaches simply dichotomized the responses and analyzed them with a CD model for binary responses. Obviously, this strategy cannot exploit the additional diagnostic information provided by MC items. De la Torre’s MC Deterministic Inputs, Noisy “And” Gate (MC-DINA) model was the first for the explicit analysis of items having MC response format. However, relying on the expectation-maximization algorithm for estimation, the MC-DINA model cannot guarantee stable and reliable estimates in small-sample settings —despite their effectiveness and efficiency when samples are large — due to computational feasibility issues caused by insufficient sample sizes. To address this issue, the nonparametric classification method for multiple-choice Items (MC-NPC) was proposed in Chapter 2. The mathematical justification supporting the legitimacy of this method was provided, and the classification performance was evaluated by simulation studies and a real-world application. Computerized adaptive testing for cognitive diagnosis (CD-CAT) achieves remarkable estimation efficiency and accuracy by adaptively selecting and then administering items tailored to each examinee. The process of item selection stands as a pivotal component of a CD- CAT algorithm, with various methods having been developed for binary responses. However, the Jensen–Shannon divergence (JSD) index introduced by Yigit et al. is the only item selection method exclusively designed for MC items. As a parametric item selection method, the JSD index requires a large sample to calibrate item parameters by the MC-DINA model, which may be infeasible when there is only a small or no calibration sample. To bridge this gap, a nonparametric item selection method for MC items (MC-NPS) was proposed in Chapter 3. The MC-NPS method implements novel discrimination power that measures an item’s ability to effectively distinguish among different attribute profiles, as well as a Q-optimal procedure for MC items to select items during the initial phase of a CD- CAT algorithm. The effectiveness and efficiency of the MC-NPS method were confirmed by simulation studies. Like most CD methods, the Q-matrix serves as the foundation for the CD methods for MC items. A crucial property of the Q-matrix is completeness, which refers to whether a Q-matrix can identify all possible attribute profiles. An incomplete Q-matrix may result in parameter identifiability issues and the misclassification of examinees, potentially distorting the interpretation of results and subsequent pedagogical interventions. Chapter 4 explored the sufficient and necessary conditions for a complete Qmatrix for MC items. Additionally, it addressed the closely related topic, the Q-optimal procedure for CD-CAT under the MC-DINA model, with formal establishment of its sufficient and necessary conditions. The proposed theorems, lemmas, and corollaries were rigorously proven through mathematical proofs, and simulation studies provided additional support for the completeness conditions of the Q-matrix for MC items and the Q-optimal procedure for CD-CAT under the MC-DINA model.