Psychometric Evaluation of Social and Emotional Learning Measures Using Multidimensional Item Response Theory Models

Abstract

Social and emotional learning (SEL) has become essential in educational settings and human development. Federal policy has begun to incorporate SEL factors into education accountability metrics. Schools, institutes, and scholars have started implementing SEL in K-12 educational contexts. Social interactions and emotions are a daily part of students’ lives, both in and out of school, which is why many educators are willing to teach and measure SEL competencies. Moreover, in previous research, it has been found that students’ higher SEL competencies are positively associated with higher academic performance and lower risky behaviors. As interest in SEL grows, measurement tools continue to evolve. An interesting finding from the overview of existing SEL measures was that not only a general SEL scale is measured but in practice, multiple dimensions, or subscales of SEL. This finding implies the multidimensional nature of SEL measures. How- ever, researchers have not attempted to use the multidimensional item response theory to identify the multidimensionality of SEL measures and evaluate their psychometric properties. This dissertation aimed to apply MIRT models to the MSS-SEL data, identify the best-fitting model, and conduct a psychometric evaluation of the MSS-SEL items and dimensions. To mediate person-score calibration and evaluation, psychometric properties such as internal structure and item parameter estimates of the MSS-SEL items were evaluated under MIRT framework. The results showed that among the competing correlated models, MGRM fitted better than the MG- PCM to the MSS-SEL data. Additionally, among the competing bifactor models, bifactor-GRM fitted better than the bifactor-GPCM. The results indicated that for both the correlated MIRT model and bifactor model applications, the graded response modeling approach functioned better for the MSS-SEL response data than the generalized partial credit modeling approach. The results of the MGRM application showed that some MSS-SEL factors were relatively more highly correlated than others. One of the bifactor-GRM results showed that the general factor (i.e., SEL) accounted for by 66% of the common variance among items. Each specific factor was accounted almost equally well for the rest of the variance. Overall, the 37 SEL items reflected the general factor reasonably well, and most of the items were moderately/strongly explained by the general factor.This dissertation demonstrated what could have been missed if just a single model (i.e., UIRT, correlated MIRT model, or bifactor model) was fitted to the data. Furthermore, validity evidence was provided for the potential use and interpretation of the MSS-SEL scores. Several implications can be made especially for researchers and practitioners who frequently get involved in all different phases of SEL measurement. First, it is important to plan ahead for an item and scale development and to search for theoretically suitable IRT models. Second, it is crucial to examine the internal structure and use MIRT models when there are multiple SEL constructs to be measured. Third, the types of scores that are meaningful for the report and to be used should be clearly declared by the SEL measurement developers and users. Psychometric evaluation of the existing SEL measures using relevant measurement models will contribute to the validation of the SEL measures in support of the intended interpretation and use of such measures by educators and practitioners. This, of course, is a core component of validity evidence to interpret and use SEL scores at multiple levels. Moreover, this is a part of a larger effort to understand and articulate a set of psychometric principles for SEL measurement.