An exploratory technique for finding the Q-matrix for the DINA model in cognitive diagnostic assessment: combining theory with data.

Abstract

Cognitive diagnostic assessment (CDA) is used to measure the specific knowledge structures and the processing skills that examinees possess. One of the components of CDA is the Q-matrix, a J x K matrix indicating whether an item j requires skill k for correct execution. Although the Q-matrix is usually considered known, emerging evidence indicate otherwise. As such, the purpose of this thesis was to investigate a potential exploratory technique that could be used to supplement theory in finding the Q-matrix of a cognitive diagnostic test in data that satisfy the DINA (Deterministic Input Noisy "And") model. The proposed method is based on principal components analysis. The components model is a reparameterization of the DINA model relating examinee responses to cognitive diagnostic tasks. Understanding the relationship between the components and DINA skills can provide information for Q-matrix development. This relationship was investigated by answering four questions, the first two being analytical and the other two being empirical: 1) When does a skill (dimension) in the Q-matrix correspond to a component in the components analysis? 2) When does a skill (dimension) in the Q-matrix fail to correspond to a component in the components analysis? 3) When does the proposed methodology (components analysis) yield a plausible and useful solution for Q-matrix development? And 4) When does the methodology result in a Q-matrix that improves parameter estimation and examinee classification accuracy? The results indicated that components analysis which is akin to item analysis on a block of items can indeed augment theory in developing Q-matrices especially for items that are designed to be diagnostic and that measure a narrow content domain. Some limitations are discussed and potential areas for further work highlighted.