Comparing the robustness of dimensionality assessment methods for polytomous data in the presence of careless responding

Abstract

Assessing test dimensionality is a critical psychometric procedure conducted before estimating score reliability and fitting item response theory models, among other applications. It has been suggested, however, that the dimensional structure of a measure is influenced not only by the test items but also by test-takers’ interactions with these items and their responding behaviors. One such behavior, careless responding (CR; i.e., responding to assessment items without considering their content due to factors such as low motivation and test length) has recently received considerable attention, particularly in low-stakes assessments. CR has been shown to attenuate correlations, underestimate score reliability, and distort dimensionality assessment, among other negative effects. Despite these issues, no prior study has systematically compared the robustness of dimensionality assessment methods in the presence of CR. This study addresses this gap by examining the robustness of three dimensionality assessment methods: (a) parallel analysis (PA), (b) polytomous dimensionality evaluation to enumerate contributing traits (PolyDETECT), and (c) item factor analysis (IFA) when applied to polytomous data contaminated by CR. Both an applied analysis and a simulation study were conducted. In the applied analysis, a random sample of 50,000 participants from the Big-Five Personality Test dataset was used, where five CR detection methods were utilized to flag careless participants. These methods flagged between 0.50% and 11.35% of participants individually, with a combined total of 29% flagged as careless. Upon CR detection, three subsets of 2,000 participants each were sampled: (a) a careful dataset (100% careful participants), (b) a careless dataset (100% careless participants), and (c) a contaminated dataset (70% careful and 30% careless participants). PA, PolyDETECT, and IFA (testing five- and six-factor models) were applied to all three datasets. Results showed that PolyDETECT was highly robust to CR, consistently identifying five dimensions across all datasets. In contrast, PA and IFA overestimated the number of factors, regardless of the dataset type. CR was found to underestimate eigenvalues in PA, thus affecting variance-explained inferences, while it inflated IFA-based model comparison criteria (e.g., AIC).In the simulation study, the effect of CR on unidimensional and multidimensional test structure was further explored, by manipulating CR type (random, midpoint, and fixed responding), CR prevalence (.10, .20, and .30), and CR severity (.25, .50, and .75). Results indicated that no method was entirely robust to CR, but the IFA-based BIC performed best for unidimensional data (.79 correct decisions), while PolyDETECT excelled with multidimensional data (.97 correct decisions). Fixed responding, high prevalence (.30), and moderate severity (.50) most distorted dimensionality assessment. Logistic regression showed CR factors explained 36% and 25% of the variance in incorrect decisions for unidimensional and multidimensional data, respectively. CR effects were more pronounced in unidimensional data, likely due to shorter tests and smaller samples. Taken together, results of the applied analysis and simulation study underscore the importance of accounting for CR in dimensionality assessment, especially in low-stakes settings.