A parametricness index and consistency with complexity penalty for model selection.

Abstract

In model selection literature two classes of criteria perform well asymptotically in different situations: Bayesian information criterion (BIC) (as a representative) is consistent in selection when the true model is finite dimensional (parametric scenario); Akaike's information criterion (AIC) performs well when the true model is infinite dimensional (nonparametric scenario). But there is little work that addresses if it is possible and how to detect the situation that a specific model selection problem is in. In this work, we differentiate the two scenarios theoretically. We develop a measure, parametricness index (PI), to assess whether a model selected by a consistent procedure can be practically treated as the true model, which also hints on AIC or BIC is better suited for the data. A consequence is that by switching between AIC and BIC based on the PI, the resulting regression estimator is simultaneously asymptotically efficient for both parametric and nonparametric scenarios. In addition, we systematically investigate the behaviors of PI in simulation and real data and show its usefulness.

Traditionally, the consistency property of BIC type of criteria for model selection is derived with a fixed number of predictors. A natural question is: does the consistency property still hold in high dimensional setting? The answer is in the positive direction [18, 69]; however, there are serious limitations of the assumptions in [18, 69]. Specifically, in [18], the size of the true model is assumed to be bounded, which may exclude many applications. In [69], the conditions 2 assumes that the smallest eigenvalue of the covariance matrix of all the predictors is always positive, which could be a little unrealistic due to the correlation among all the predictors, especially when the number of predictors is large. And the condition 4 in [69] assumes that the smallest coefficient in the true model is higher than a certain order, which is reasonable, but the order could be improved. We provide sufficient conditions on consistency for BIC and similar types of criteria in high dimensional settings and show that these conditions are also necessary in a sense by giving counterexamples. We demonstrate that the results in [18, 69] are special cases of ours. Moreover, our results eliminate the the restriction in [18] on the size of the true model and relax the assumptions in [69] on the true model. We also generalize the concept of consistency and provide similar results to this new concept. A statistical risk bound for the model selected by the BIC type of criterion is also derived.