Adaptive regularization using the entire solution surface

Abstract

Regularization is essential for obtaining high predictive accuracy and selecting relevant variables in high-dimensional data. Within the framework regularization, several sparseness penalties have been suggested for delivery of good predictive performance in automatic variable selection. All assume that the true model is sparse. In this dissertation, we propose a penalty, a convex combination of the L 1 - and L ∞ -norms, that adapts to a variety of situations including sparseness and nonsparseness, grouping and nongrouping. The proposed penalty performs grouping and adaptive regularization.

In regularization, especially for high-dimensional data analysis, efficient computation of the solutions for all values of tuning parameters is critical for adaptive tuning. In the literature, there exist several algorithms computing an entire solution path for one single tuning parameter. All use the Kuhn-Tucker conditions for constructing the solution path. However, there does not seem to exist such an algorithm for multiple tuning parameters. This is partly because of the difficulty of applying the Kuhn-Tucker conditions when many slack variables are involved.

In this dissertation, we introduce a homotopy algorithm utilizing the subdifferential, a systematic method of handling nonsmooth functions, for developing regularization solution surfaces involving multiple tuning parameters. This algorithm is applied to the proposed L 1 L ∞ penalty and permits efficient computation. Numerical experiments are conducted via simulation. In simulated and real examples, the proposed penalty compares well against popular alternatives.