Browsing by Author "Deng, Jiayi"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Linking Errors Introduced By Rapid Guessing Responses When Employing Multigroup Concurrent Irt Scaling
    (2024) Deng, Jiayi
    Test score comparability in international large-scale assessments (LSA) is of utmost importance in measuring the effectiveness of education systems and understanding the impact of education on economic growth. To effectively compare test scores on an international scale, score linking is widely used to convert raw scores from different linguistic version of test forms into a common score scale. An example is the multigroup concurrent IRT calibration method, which is used for estimating item and ability parameters across multiple linguistic groups of test-takers. The method uses common item parameters to most items and groups, with a select few items allowed to have group-specific parameters. Although prior researchers used empirical data from international LSAs to demonstrate the effectiveness of multigroup concurrent IRT calibration in offering greater global comparability in score scales, it is important to note that they assumed comparable test-taking efforts across cultural and linguistic populations. This assumption may not hold true due to differential rapid guessing (RG) rates, potentially biasing item parameter estimation. To address this gap, I proposed a real data analysis and simulation to explore this area. The objective of the current study is to investigate the linking errors introduced by RG responses when employing multigroup concurrent IRT calibration.In the real data analysis, data from the Arabic and Chinese groups in the PISA 2018 Form 18 science module were linked, with RG responses flagged using response time information. Despite observed differential RG, the linking procedure proved robust to anchor identification and ability estimation. In the simulation, data was generated for two groups with varying motivation levels. These groups were administered two linguistic versions of a test form comprising multiple-choice items. Factors such as differential RG rate, association between ability and RG propensity, group impact, sample size, and model fit criteria were considered. The assessment focused on anchor identification accuracy, item parameter estimation accuracy, and ability parameter estimation accuracy and precision. The findings showed that multigroup concurrent IRT calibration was robust against differential RG, with sample size and group impact being primary factors influencing errors. However, differential RG could affect ability estimation precision and item parameter estimation accuracy.
  • Thumbnail Image
    Item
    Quantifying the Biasing Effect of Rapid Guessing on Estimates of Coefficient Alpha
    (2021) Deng, Jiayi; Rios, Joseph
    An underlying assumption of coefficient alpha is that random error is uncorrelated; however, this assumption is violated when examinees engage in similar construct-irrelevant behaviors across items. One construct-irrelevant behavior that has gained increased attention in the literature is rapid guessing (RG), which occurs when examinees answer quickly with intentional disregard for item content. To examine the extent that estimates of coefficient alpha are biased due to RG, a simulation study was conducted in which the ability characteristics of rapid responders and the percentage and pattern of RG were manipulated. After controlling for test length and difficulty, results indicated that RG characteristics had a practically negligible impact on estimates of coefficient alpha, with the average degree of bias found to range from -.05 to .02 for upwards of 30% of RG responses in the data. This negligible effect was supported in a meta-analytic investigation, which observed a difference in coefficient alpha of .07 when comparing filtered (i.e., RG responses removed) and unfiltered (i.e., RG responses left in the data) datasets across five studies and 12 effect sizes. These findings suggest that estimates of coefficient alpha are largely robust to violations of the assumption that random error is uncorrelated due to construct-irrelevant behaviors such as RG.