Exploring the Accuracy of School Rankings for Accountability Decisions

Abstract

In the U.S., the school accountability system in education uses testing as a major part of its decision-making process. Part of this decision making comes from the result of student test scores, which are aggregated to estimate school scores (typically in the metric of percent proficient). Regarding precision of such school metrics, measurement error is assumed to have a net value of zero across the population of students within a school, but students are not randomly assigned to schools. The aggregation of student scores to obtain school scores retains the measurement error from the students within their respective schools. This study used available data to emulate how one state uses test scores to rank schools and simulated various conditions to examine how the accuracy of the school rankings might be affected.The conditions examined multiple measurement error models to evaluate changes in accuracy based on the various measurement error models. If measurement error has a net value of zero, accuracy should not be affected across this condition. Another condition examined the different methods of aggregating student scores to obtain school scores. If student measurement error is aggregated along with the student score to estimate school scores, some schools may have more error than other schools, and this information may be helpful when making interpretations about school rankings. The last condition was the reliability of the test. Real data summary statistics were used in the study, so the minimum and maximum test reliability was examined regarding how accuracy was affected across the other conditions. Several major takeaways were found. When there was more error variance, the accuracy of school rankings dropped, but changes in the specification of the error distribution did not change the accuracy of school rankings. The biggest association with accuracy of school rankings was how student test scores were aggregated to obtain school scores. The aggregation of student scores condition had the largest difference in accuracy across the conditions, on average. In this study, I hoped to inspire more research to provide validity evidence to support the decision-making process of the accountability system, especially studies that attempt to minimize error within these processes.