Methodologies in Dimension Reduction and Error Variance Estimation of High-dimensional Regressions

Abstract

This thesis discusses several methodologies in dimension reduction and error variance estimation of high-dimensional regression models. The models of interest mainly include the principal components regression, high-dimensional additive regression, and high-dimensional varying-coefficients regression. For dimension reduction, we address the ``blind selection" issue in the conventional principal components regression and provide a simple regression method to solve this issue. The conventional principal components regression retains the principal components with large variances and discards those with small variances. This principal components selection process can easily lead to a poor prediction when the response variable is highly correlated to principal components with small variances. We propose a simple remedy named response-guided principal components regression (RgPCR) that selects principal components for regression based on both the variance of principal components and the goodness of fit to the response. RgPCR is easy to implement without using any optimization algorithms and works naturally for both low dimensional and high dimensional data. We derive a Cp type statistic for selecting the tuning parameter in RgPCR. For high-dimensional regressions, we focus on the error variance estimation of high-dimensional additive regressions and high-dimensional varying-coefficient regressions. The error variance estimation serves a crucial role in the statistical inference of regression models, and it is challenging under high-dimensional settings. We propose a novel error variance estimation procedure for high-dimensional additive regressions and high-dimensional varying-coefficient regressions. As part of the estimation procedure, we propose the scaled group-Lasso regression, which accomplishes the variable selection, coefficients estimation, and error variance estimation jointly in one stage. We provide a fast computing algorithm of the scaled group-Lasso regression through groupwise coordinate descent. We also discuss other candidate error variance estimators of high-dimensional additive regressions and high-dimensional varying-coefficient regressions. We present extensive simulations and real data studies to evaluate the performance of error variance estimators under various model settings.