Building Evaluation Capacity in Educational Organizations: A Longitudinal Case Study of One Metropolitan School District

Abstract

Although evaluation capacity building (ECB) is relatively prevalent in the literature, there is not an extensive empirical body of work related to the topic. The purpose of this study was to add to the existing literature through a 30-year historical longitudinal case study of a large metropolitan school district from the perspective of its leadership. This research examined the longitudinal evidence of the context, processes, and activities that impact ECB within a K-12 educational setting. In addition, it traced the development of an internal evaluation department and the steps to build supports for data use and evaluation in the midst of accountability, using process use as the primary strategy toward ECB. The study consisted of interviews with 14 district leaders, along with document reviews over the years 1985 to 2015, encompassing significant legislative enactments such as No Child Left Behind (NCLB) and the public accountability era in K-12 education. The district leadership was found to value data use across time, along with community involvement and input, and demonstrated a strong legislative awareness and presence. The leaders who were interviewed also viewed evaluation as mission and vision critical across time. Substantial changes were made over the 30-year time frame in the number and types of staff members in the department, the internal process and activities of the department, how the department collaborated with others across the organization, and the organizational reporting structure. In addition, organizational factors such as the shift from evaluation for mandatory reasons (such as grants and legislative requirements) to evaluation for internal decision making and inquiry occurred over the 30-year span, along with important changes in technology and data integrity, accessibility, and understandability. Eight key recommendations to propel ECB emerged from the research: 1) attend to data integrity, accessibility, and understandability – including technology; 2) capitalize on one-time and special funding; 3) recognize that leadership matters and change is disruptive; 4) hire the right people; 5) remember that disposition/personality matters; 6) make ECB an intentional, ongoing focus; 7) work toward consistency, not the new next best thing; and 8) build on previous work instead of completely starting over.