Multiple Regression in Industrial Organizational Psychology: Relative Importance and Model Sensitivity

Abstract

When evaluating research findings, it is important to examine what statistical methods were used to reach and support the stated conclusions. Regression is a common analysis in the Industrial/Organizational psychology literature and researchers have debated how to interpret the standardized optimal weights produced in ordinary least squares (OLS) regression. Multiple methods for determining the relative importance of predictors in a regression model have been proposed, along with a variety definitions of what is meant by predictor importance. Conversely, it has been shown that by slightly decreasing the model R2 that is obtained through OLS multiple regression an infinite number of alternative weight vectors can be produced, calling into question the meaning of OLS weights when the alternative weights diverge from the OLS weights. Articles published from 2003-2014 in the Journal of Applied Psychology, Academy of Management Journal, and Psychological Science that used OLS regression were reviewed. It was found that regression is used to answer questions on a wide variety of topics and interpreted in a multitude of ways in the I/O psychology and general psychology literature. The study found that different relative importance analyses can result in different conclusions about what predictors are most important. Examining alternative weight vectors further brings into question conclusions drawn based on optimal weights. For the majority of studies examined alternative weight vectors were found that provided a different rank ordering of predictors with only a small loss in model fit. The findings in this paper highlight and reinforce the need for Industrial/Organizational psychologists to turn a critical eye on the interpretation of regression analyses, especially regression weights, in reaching substantive conclusions.