Nonlinear Regression Based on Neural Networks: Prediction, Sufficient Dimension Reduction, and Model Compression

Abstract

In the modern era, neural networks have achieved great success in various fields of applications. Since neural networks allow the modeling of nonlinear relationships, they have become popular and useful tools for tackling different problems such as prediction, classification, clustering, dimension reduction, visualization, computer vision, and many others. This thesis focuses on three interrelated topics in the study of neural networks under the regression setting with two primary goals. One is to incorporate traditional statistical learning methods and recent machine learning techniques, the other is to utilize statistical theory to provide theoretical justification and understanding of some neural network methods. With the rapid growth and development in modern information technology, high-dimensional data are ubiquitous in various scientific fields. Specifically, recent machine learning techniques on prediction performances under high-dimensional data frameworks have supported many applications for gene expression inferencing. We propose a deep neural network learning approach that incorporates deep learning techniques and statistical methods to effectively enhance the predictive accuracy to address a multi-task learning problem in gene expression profiling. Unlike the conventional complicated multi-task learning methods, our proposed method can effectively learn the interrelations from larger-scale tasks. As the data structure of high dimensions may raise challenges in terms of both computation and inferencing, it is often of interest to find a reduced dimensionality of the data from linear combinations of the predictors that can retain as much original information as possible in the relationship between the response and the original predictors. We propose a sufficient dimension reduction method that is able to not only obtain the structural dimension effectively without prior knowledge, but also estimate the central space. In addition, the framework is extended to the approximated case, which may be more applicable in practice. With the development of powerful extensions of classical works in deep learning, many state-of-the-art architectures today have achieved spectacular results and increasing attention in various applications. However, these works generally rely on complex architectures with tens and even hundreds of layers, which result in the need for tremendous storage space, time allocation, computation costs, and a copious amount of data. These concerns motivate research in model compression. Many empirical results have shown that when a certain model is more complex than what might be needed, different techniques can be applied to reduce model complexity and still retain the generalization power in the original model. The theoretical work, however, is limited. We examine neural network pruning from the perspective of sparse linear approximation and provide a theoretical understanding of how to characterize a network that is compressible and by how much it can be properly pruned with little prediction accuracy degradation.