Fully Bayesian Penalized Regression With A Generalized Bridge Prior

Abstract

We consider penalized regression models under a unified framework. The particular method is determined by the form of the penalty term, which is typically chosen by cross validation. We introduce a fully Bayesian approach that incorporates both sparse and dense settings and show how to use a type of model averaging approach to eliminate the nuisance penalty parameters and perform inference through the marginal posterior distribution of the regression coefficients. We establish tail robustness of the resulting estimator as well as conditional and marginal posterior consistency for the Bayesian model. We develop a component-wise Markov chain Monte Carlo algorithm for sampling. Numerical results show that the method tends to select the optimal penalty and performs well in both variable selection and prediction and is comparable to, and often better than alternative methods. Both simulated and real data examples are provided.