Quantifying Quality: The Effects of Score Transformation Method and School Demographics on School Rankings Under the Elementary and Secondary Education Act

Abstract

Reasons for quantifying and ordering relative school achievements as a measure of school quality are numerous. They range from informing parents about where to enroll their children to complying with federal reporting and accountability requirements. Even after accepting the premise that results on state tests designed to measure student mastery of subject standards can serve as a proxy for the measure of a school's quality, questions remain about how individual student results should be transformed into a school-level measure in a way that is more reflective of how a school is serving its students than of what type of students a school serves. This study examines the effects of using different score transformations from the same test results to rank schools by investigating three questions: (1) What effect does the method of transforming student scores on Minnesota state exams have on relative Minnesota school performance rankings over time?; (2) What effect do school demographics have on relative Minnesota school performance rankings over time?; and (3) What effect do the interactions of method of transforming student scores on Minnesota state exams and school demographics have on relative Minnesota school performance rankings over time? A unique opportunity for robust analysis of a complete set of statewide individual level testing and enrollment records was available though a special agreement with the Minnesota Deaprtment of Education. Comparison of multilevel models shows that simpler score transformations lead to quantification of quality and relative school rankings that are the least related to the demographic characteristics of the students a school serves.