Validity and Reliability Evidence for an Experimental Performance Evaluation Instrument for Educational Speech-Language Pathologists

Abstract

Performance evaluation for educators is intended to measure, develop, and support professional practices, and, in turn, improve student outcomes. To date, however, very little research exists to support the performance evaluation practices for non-classroom educators (Holdheide, Goe, Croft, & Reschly, 2010), such as educational speech-language pathologists (SLPs). Validity and reliability evidence for an experimental performance evaluation instrument specifically designed for SLPs was examined in this study. Study data were from 111 SLPs in a mid-size urban district who were evaluated one during an academic school year. The performance of the 111 SLPs was also described, so that any potential bias in the instrument could be examined. Results showed a restricted range of performance in which most SLPs were rated as proficient or exemplary on performance evaluation items. Some preliminary indications of bias were present, such that SLPs serving birth-five students, students with combined communication disorders, or students with moderate-severe disabilities were rated 6-10 points lower in total score (out of a maximum score of 108) compared to their colleagues who did not serve those populations. Construct validity analyses showed that the instrument's items were only loosely related to each other, although exploratory factor analyses did suggest an underlying structure of four domains. Face validity was gauged through optional perception surveys of the 111 SLPs in which the majority of survey participants felt the instrument items represented effective SLP practices. Finally, performance evaluation items demonstrating the highest evidence of reliability were related to an SLP’s planning of intervention; items with the lowest evidence of reliability were related to an SLP’s management of the session and rapport with students. Despite some limitations, it was concluded the SLP performance evaluation instrument showed initial evidence of being able to evaluate SLPs fairly, accurately, and with perceived credibility from the district SLPs.