Evaluation of small-sample statistics that test whether variables measure the same trait

Abstract

This study investigated the performance of five small-sample statistics (Lord’s, Kristof’s, McNemar’s, Forsyth and Feldt’s, and Braden’s) that test whether two variables measure the same trait except for measurement error. The conservative Type I error rates of the Lord and Kristof procedures and the liberal error rates of the McNemar, Forsyth and Feldt, and Braden procedures were corrected by determining appropriate critical values. Power comparisons were then made at the fixed α levels. In general, the McNemar statistic was shown to be the most powerful. Finally, the effects of non-normality were investigated, and it was demonstrated that the Braden technique became very liberal, whereas the other statistics tended to be somewhat liberal at the .01 significance level and reasonably robust at the .05 level. Index terms: Disattenuated measures, Measurement error, Monte carlo simulation, Non-normality, Small-sample statistics, Type I error and power rates.