Using citation analysis methods to assess the influence of STEM education evaluation

Abstract

This study explores the validity of using citation analysis methods as a way of assessing the influence of program evaluations conducted within the areas of science, technology, engineering, and mathematics (STEM). Interest in the broad influence of evaluations has caught the attention of evaluation theorists, practitioners, and funders recently. However, methods for measuring the influence of evaluations have yet to be developed and validated. Citation analysis is widely used within scientific research communities to measure the relative influence of scientific research and/or specific scientists. This study explores the applicability of citation analysis for understanding the broad impact of STEM education program evaluations.

Nine assumptions regarding the validity of using citation analysis methods to assess STEM education evaluation product influence are examined using data from four sources: (1) citation analysis data, (2) the opinions of an expert panel, (3) data from a survey of primary investigators and evaluators from local projects connected with four national program evaluations, and (4) a review of relevant literatures. The data collected for the validation study suggest that citation analysis methods provide data to help understand, to a limited extent, the influence of large-scale program evaluations on the fields of STEM education and evaluation. In particular, citation data can be used to understand and compare patterns of influence of multi-site STEM program evaluations.

Citations, however, are only one among many possible measures of one limited type of influence arising from the dissemination of evaluation products. Additionally, citation data do not appear to be useful for precisely quantifying the actual level of influence of any one evaluation. Moreover, the examination of the content of citations is critical. Without understanding the content of the citations, judgments cannot be made about whether citations are actually measuring influence. Consequently, it is important to stress that citations are only one measure of one possible influence arising from an evaluation and are limited and should be interpreted as such.