Systematic Errors in Approximations to the Standard Error of Measurement and Reliability

Abstract

Lord’s approximation to the standard error of measurement of a test uses only n, the number of items. Millman’s is based on n and p̄, the mean difficulty. Saupe has used Lord’s approximation to derive an approximation to the reliability. Through an empirical demonstration involving 200 classroom tests, all three approximations are shown to be biased. The Lord and Millman approximations overestimate s[subscript x]√(1-KR20), and thus Saupe’s underestimates r[subscript x, subscript x prime] for these tests. The unweighted mean of the tests’ mean item difficulties was .68, supporting Lord’s original warning that his approximation be used cautiously with tests that are either very difficult or very easy. Still, the approximations did correlate very highly with their criteria, supporting their continued limited use.