On Akaike and likelihood cross-validation criteria for model selection

Abstract:

The talk discusses Akaike and likelihood crossvalidation criteria for model/estimator choice. After a presentation of the main concept on model selection, we will focus on the choice of estimators in non-standard cases. First, we study two examples arising when we wish to assess the quality of estimators on a particular set of information, while the estimators may use a larger set of information. The first example occurs when we construct a model for an event which happens if a continuous variable is above a certain threshold. We can compare estimators based on the  observation of only the event or on the whole continuous variable. The other example is that of predicting survival

based on survival information only, or using in addition information on patient's disease. We develop modified AIC and LCV criteria to compare estimators in this non-standard situation. Second, we study the choice of estimators in prognostic studies. Estimators for a clinical event may use repeated measurements of markers in addition

to fixed covariates. These measurements can be linked to the clinical event by joint modelling involving latent structures. When the objective is to choose between different estimators based on joint models for prediction, the conventional Akaike information criterion (AIC) is not well adapted and decision should be based on predictive accuracy. We define an adapted risk function called expected prognostic cross entropy (EPCE) and further modify it for right-censored observations. The risk functions can be estimated by leave-one-out cross validation, for which we give approximate formulas and asymptotic distributions.