The Effect of Incorporating Active Learning In Calibration Exercises On Intra and Interrater Reliability Among Dental Hygiene Faculty

Abstract

ABSTRACT Purpose/Objective: The purpose of this study was to investigate if incorporating active learning into calibration exercises increased the level of intra- and interrater reliability among dental hygiene faculty. Methods: The study used a two-group randomized experimental design with a convenience sample consisting of ten dental hygiene faculty members from the division of dental hygiene at the University of Minnesota (n=10). Baseline training in motivational interviewing (MI) was provided to all faculty at a day-long continuing education course. One month later, all faculty viewed three videos of students performing MI during an OSCE and graded their performance using a standardized grading rubric. The treatment group then engaged in the identical motivational interviewing OSCE required of the students. One month later, both study groups viewed the same three videos and graded the students’ MI performance using the identical standardized grading rubric. (See Appendix C). Results: The overall intra-rater reliability was calculated using Cohen’s Kappa statistic, pre-and post-intervention for both the control and treatment groups. Results revealed moderate to weak intra-rater reliability for both groups (.494). Fleiss’ kappa statistic was used to assess interrater reliability. The treatment group achieved higher levels of agreement verses the control group on six of the ten questions. Only one question (See Figure 1: R06) had perfect or near perfect agreement in both study groups. Conclusion: There was no statistically significant difference found in intra- and interrater reliability scores between the control and treatment groups following an active learning intervention. Even though statistical significance was not achieved, individual faculty data suggests active learning did have an effect on the faculty in the treatment group. Post intervention, faculty in the treatment group had greater variations in scores indicating the experience had challenged their frame of knowledge and may have become more empathetic to the challenges of motivational interviewing having conducted an MI session themselves.