Topics in Uncertainty Quantification for Model Selection

<p dir="ltr">Model selection procedures have become quite popular in scientific ap plications; however, study of their variance has not kept pace. In response, researchers have begun developing model confidence set methods, where the goal is to select a set of models likely to contain the best model with high probability, rather than identifying a single model. Due to the popularity of model selection in applied research, we believe that further introducing and formalizing model confidence set procedures could be especially beneficial to the applied sciences, leading to greater flexibility and reliability in model selection tasks. </p><p dir="ltr">While model confidence sets have become a valuable tool for assessing model selection variability, their practical use can be hindered by significant computational demands. Many of the first methods involve all pairwise comparisons among a set of candidate models, which can be quite costly. To address this issue, we develop a computationally efficient procedure, called model path selection (MPS), which draws inspiration from greedy search algorithms. Specif ically, we create a branching forward selection procedure that allows for a faster search over the candidate model space. With this, we provide both a set of well-performing models of identical complexity and characterize the variability of forward selection. </p><p dir="ltr">Although MPS is a great tool for generating multiple models of a particular size and class, researchers often want to compare models of differing size or function class. Among the most popular methods for doing so is the comparison of V-fold cross-validated errors. To better formalize this task, we derive high dimensional Gaussian comparison results for standard V-fold cross-validated risk estimates. As a result, we are able to construct asymptotically valid model confidence sets and uniform confidence bands for cross-validated risks, which enable the simultaneous comparison of models with different complexities. These cross-validation model confidence sets allow us to generate a stopping rule for MPS, which otherwise requires users to specify a number of desired features. Lastly, we apply this new version of MPS to synthetic genetic data.</p>