Beyond Exit Surveys: A Critical Examination of Current Evaluation Practices for Diversity Certificate Training Programs in Higher Education

Abstract

In contemporary American society, evidence of bias within educational institutions abounds (Douglas & Halas, 2013; Hardie & Tyson, 2013; Martin, 2009; Milner, 2013; Ropers-Huilman, Winters, & Enke, 2013; Rothstein, 2013; Wildhagen, 2012). Educational institutions and their leaders articulate a commitment to fostering an inclusive, equitable environment. As a result, diversity and training departments develop programs to help individuals gain the skills and knowledge to change their behavior in the workplace and beyond. This study highlights a problem facing diversity training staff and evaluators. Current practices based on practitioner reports indicated that diversity program evaluations were not measuring the outcomes of their program on individuals or institutions. They reported changes such as enactment of new policies or the numbers of participants in the program, implicitly suggesting that these measures translate to changes in the organization’s culture, without offering any evidence to support such a claim. Scholarly human resource literature posited generic training evaluation models, such as Kirkpatrick’s (2006) four-level model created more than fifty years ago. After analyzing and contrasting additional training evaluation models (Guskey, 2014a; Hamblin, 1974; Phillips & Stone, 2002), it was noted that little scholarly research exists addressing how, if at all, these models may fit for a diversity certificate training program evaluation considering the nuances and politics related to diversity programming in higher education. The purpose of this study was to understand how managers evaluated the outcomes of their diversity certificate training programs in higher education. It is important to note that the purpose of the study was not to assess the outcomes of the programs themselves or delve into which practices in the various programs may or may not be promising. The study employed a grounded theory approach to develop an evaluation framework. Nine higher educational institutions throughout the United States participated in this study. DCT program managers focused primarily on gathering formative feedback for program improvement. Study participants relied on informal ways of knowing their programs were having an impact. They used their intuition and made implicit assumptions, such as when a program is full, that equates to the quality and reputation of the program. Furthermore, they described informal conversations with past and current program participants that reassured the training staff that their program was having an impact. Study participants described lack of knowledge and skills as well lack of staff time as key perceived barriers to conducting evaluations. The result of this study is a framework depicting its findings including context, supports, barriers and opportunities for DCT evaluation. Opportunities to incorporate varied DCT evaluation methods are discussed. As little has been published about this model of diversity training, suggestions for future research include conducing and publishing results from DCT program evaluations. Using multiple methods to conduct the evaluation and publishing the results would contribute to the evaluation knowledge base as well.