Assessment Of Cognitive Transfer Outcomes For Students Of Introductory Statistics

Abstract

This study chronicles the creation of an assessment tool that quantifies cognitive transfer outcomes for introductory statistics students. Literature suggested that outcomes associated with cognitive transfer are closely aligned with statistical thinking and are indicative of students’ ability to apply learning to novel scenarios beyond the classroom. No assessment tool had been developed and published for the purpose of measuring cognitive transfer outcomes among statistics students. The results of this study suggest that the Introductory Statistics Understanding and Discernment Outcomes (I-STUDIO) assessment tool may effectively serve this purpose. The assessment tool was developed according to a rigorous protocol of expert feedback and iterative piloting. Data were collected and analyzed from a nationwide sample of nearly 2,000 students attending a wide variety of post-secondary institutions, and the I-STUDIO instrument was found to measure both forward-reaching and backward-reaching high road transfer outcomes with good psychometric properties. Data analysis indicated high reliability and diverse validity evidence. This evidence included confirmatory factor analysis models with compelling alignment to the theoretical model and analysis of qualitative themes among expert feedback. Analysis of scoring consistency also showed strong inter-rater agreement. Although the sample size of the scored responses is somewhat small by convention for item response theory, a graded response model generally showed good item functioning. Furthermore, the data suggested that the I-STUDIO assessment estimated student ability with consistent precision across a wide range of above-average and below-average students. Teachers and researchers can use I-STUDIO for comparing outcomes of alternative curricula. Additionally, the I-STUDIO instrument can be used to measure the effect of curriculum changes designed to improve transfer outcomes. Furthermore, the instrument and scoring rubric were designed to accommodate diverse curricula for the purpose of refining course outcomes.