Examining Power and Type 1 Error for Step and Item Level Tests of Invariance: Investigating the Effect of the Number of Item Score Levels

Abstract

Within polytomous items, differential item functioning (DIF) can take on various forms due to the number of response categories. The lack of invariance at this level is referred to as differential step functioning (DSF). The most common DSF methods in the literature are the adjacent category log odds ratio (AC-LOR) estimator and cumulative category log odds ratio estimator (CU-LOR). Although the study of DSF may be helpful when opposing DIF effects within an item can go undetected or for informing what part of a multi-step item may need improvement, research regarding DSF procedures is limited. The effect of number of item score levels has not been investigated with regard to the relationship between DSF and traditional DIF methods, including differences in statistical behavior. This study investigates the effect of the number of item score levels on power and Type I error of the following DSF methods: AC-LOR, CU-LOR as well as DIF methods: Mantel (chi-square) Test, Liu Agresti, Generalized Mantel-Haenszel, and Simultaneous Step Level test (SSL). This study also examined which statistical procedures are most effective for adjusting per comparison Type I errors for the SSL method: Dunn-Bonferroni, Benjamini and Hochberg, or Holm’s. Conditions varied included (a) sample size ratio, (b) number of item score levels, (c) generating model, (d) impact, and (e) DSF pattern. Results suggest that altering the number of score levels did not have an effect on the DSF/DIF detection methods. When considering both statistical and practical significance of factors affecting power, the pattern of DSF was the most important effect. Additionally, the Dunn-Bonferroni adjustment was adequate when using the SSL method. The SSL method performed well compared to the other DIF methods and should be considered for simultaneously detecting both DSF and DIF. The significance of these results as well as limitations and future directions are discussed.