Simulating the Effects of Test Score Reliability and Test Dimensionality on Teacher Value-Added Scores and Inferences

Abstract

A teacher evaluation system that has gained in popularity is value-added assessment (VAA). In VAA, student test scores are used in an attempt to isolate the effect of individual teachers on student learning. Teacher value-added scores are interpreted as measures of teacher effectiveness, and are now widely used to evaluate teachers, and in pay-for-performance programs. Despite a body of literature on a variety of VAA issues, very little attention has been given to the psychometric properties of the student achievement tests used to derive teacher value-added scores, and the implications of violations to model assumptions on using value-added scores to make decisions about teachers. The purpose of the present study was to examine the potential bias produced in estimating teacher effectiveness (i.e., in teacher value-added scores) when there are violations to (a) the assumption of unidimensionality in the measurement model used to estimate student achievement scores, and (b) the assumption that control variables are free of measurement error in the statistical models used to estimate teacher value-added scores. An additional purpose was to examine the effect(s) of model assumption violations on the use of teacher value-added scores to make high-stakes decisions about teachers’ relative effectiveness. The results showed that teacher value-added scores were differentially biased by variations in the properties of the student achievement tests used to derive the value-added scores. Both bias in the estimation of teacher value-added scores, and the consistency of teachers’ estimated relative effectiveness were most sensitive to variations in the specification of test dimensionality. The results strongly caution against the use of value-added scores to make high-stakes decisions about teachers.