Browsing by Subject "Multidimensional item response theory"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Assessing Dimensionality of Latent Structures Underlying Dichotomous Item Response Data with Imperfect Models
    (2013-07) Zopluoglu, Cengiz
    The purpose of this study was to investigate the effect of model misspecification due to minor latent factors on a variety of dimensionality assessment methods proposed in the literature by using both real and simulated data. Several dimensionality assessment procedures based on eigenvalue examination (i.e., parallel analysis), conditional covariances (i.e., DETECT), and model selection approach (e.g., NOHARM and Mplus based chi-square statistics, RMSEA, GFI, AIC) were considered in the study. Two studies were conducted. In Study 1, the average, standard deviation, and range of the number of dimensions suggested by different approaches were investigated using sample datasets drawn from a very large real item response dataset treated as the population. In Study 2, a comprehensive simulation study was run, and the performances of the analytical methods were evaluated using the number of major dimensions in the true generating model as a reference. The current study provides some interesting and provoking results regarding the performances of some well-known and most commonly used practices under certain conditions. The results of the current study suggest that most of the methods proposed in the literature and available for practitioners are not necessarily useful tools in dimensionality assessment, particularly if the goal of dimensionality assessment is to identify the latent traits with major influences, when the underlying factor structure is complex and minor factors are present. The current study provides some insight for the performance of different dimensionality assessment approaches with misspecified models when the underlying latent structure was factorially complex.
  • Thumbnail Image
    Item
    Reliability and Validity Evidence of Diagnostic Methods: Comparison of Diagnostic Classification Models and Item Response Theory-Based Methods
    (2022-05) JANG, YOO JEONG
    Despite the increasing demand for diagnostic information, observed subscores have been often reported to lack adequate psychometric qualities such as reliability, distinctiveness, and validity. Therefore, several statistical techniques based on CTT and IRT frameworks have been proposed to improve the quality of subscores. More recently, DCM has also attracted increasing attention as a powerful diagnostic tool that can provide fine-tuned diagnostic feedback. Despite its potential, there has been a dearth of research evaluating the psychometric quality of DCM, especially in comparison with diagnostic methods from other psychometric frameworks. Therefore, in this simulation study, DCM was compared with two IRT-based subscore estimation methods in terms of classification accuracy, distinctiveness, and incremental criterion-related validity evidence of subscores. Manipulated factors included diagnostic methods, subscale length, item difficulty distribution, intercorrelations of subscores, and criterion validity coefficients. For classification accuracy, all diagnostic methods yielded comparable results when the center of item difficulty coincided with mean examinee ability and cut-scores. However, when average item difficulty was mismatched with mean examinee ability and cut-scores, DCM yielded substantially higher/lower classification accuracy than IRT-based methods with direction and magnitude of discrepancy depending on the type of agreement measures employed. For subscore distinctiveness, compared to IRT-based methods, DCM yielded subscores more distinct from each other and overall scores when continuous rather than discrete subscores were utilized. Lastly, regarding incremental criterion-related validity evidence, the contribution of DCM estimates over and above overall scores tended to be comparable to but slightly smaller than that of IRT-based methods. Additionally, higher classification accuracy was associated with longer subscales, item difficulty distribution more aligned with examinee ability distribution and cut-scores, and higher intercorrelations of subscores. The same conditions except for higher intercorrelations of subscores also tended to be associated with higher subscore distinctiveness. In contrast, incremental criterion-related validity evidence of subscores was largely a function of intercorrelations of subscores and magnitude of criterion validity coefficients: it increased with lower intercorrelations of subscores and higher criterion validity coefficients. In general, the results of this study suggested that IRT-based methods would be preferable over DCM as diagnostic means when item responses are obtained from IRT-based assessment forms.