Assessing Dimensionality of Latent Structures Underlying Dichotomous Item Response Data with Imperfect Models

Abstract

The purpose of this study was to investigate the effect of model misspecification due to minor latent factors on a variety of dimensionality assessment methods proposed in the literature by using both real and simulated data. Several dimensionality assessment procedures based on eigenvalue examination (i.e., parallel analysis), conditional covariances (i.e., DETECT), and model selection approach (e.g., NOHARM and Mplus based chi-square statistics, RMSEA, GFI, AIC) were considered in the study. Two studies were conducted. In Study 1, the average, standard deviation, and range of the number of dimensions suggested by different approaches were investigated using sample datasets drawn from a very large real item response dataset treated as the population. In Study 2, a comprehensive simulation study was run, and the performances of the analytical methods were evaluated using the number of major dimensions in the true generating model as a reference. The current study provides some interesting and provoking results regarding the performances of some well-known and most commonly used practices under certain conditions. The results of the current study suggest that most of the methods proposed in the literature and available for practitioners are not necessarily useful tools in dimensionality assessment, particularly if the goal of dimensionality assessment is to identify the latent traits with major influences, when the underlying factor structure is complex and minor factors are present. The current study provides some insight for the performance of different dimensionality assessment approaches with misspecified models when the underlying latent structure was factorially complex.