Development and Validation of a Survey to Measure Perceived Team Communication Skills in Middle and High School STEM Out-of-School Time Programs

Abstract

Twenty-first century skills are vital for preparing youth for careers in science, technology, engineering, and math (STEM) fields. STEM out-of-school time (OST) programs play an important role in helping youth develop these skills, particularly the teamwork skills necessary for the growing collaborative nature of STEM jobs. However, there is a lack of appropriate measures to evaluate this key programmatic outcome in STEM OST settings. This dissertation research addresses the lack of measures through the development of an instrument to assess team communication skills in middle and high school STEM OST programs. The instrument was developed and validity evidence was gathered though a rigorous four-phase process. Phase 1 focused on identifying and operationalizing the teamwork skill area to be measured by the instrument. The skill area of team communication was most common among STEM OST programs and was defined as information exchange, closed-loop communication, and listening. In Phase 2, the survey scenario and items were developed and then reviewed by experts in STEM OST, youth development evaluation, teamwork, and measurement. Phase 3 involved think-aloud interviews and a national pilot test. Revisions to the survey occurred throughout each phase, leading to the final phase: a national field test of the instrument with 959 youth from 40 STEM OST programs across the country. Through confirmatory factor analysis, a five-factor model of team communication skills was found to be a good fit. The model included two factors for closed-loop communication, two factors for information exchange, and a listening factor. Responses for each of the five factors were reliable with coefficient alphas ranging from  = .70 to  = .79. The final instrument is a 28-item scenario-based, self-report measure of middle and high school youths’ perceptions of their team communication skills. The survey instrument and operationalization of team communication skills in STEM OST programs will be valuable for both the evaluation and STEM OST fields.