Model Averaging

Model averaging allows you to compute the average of a parameter from the models in the Results Browser. By doing so, you include model selection uncertainty in the estimate of precision of the parameter, and thus produce unconditional estimates of variances and standard errors. Parameters that are to be model averaged are selected from the possible parameters in the parameter index matrices (PIMs). Note that you should generally only model average across one data type when multiple data types exist in the Results Browser because you have changed the data type. Other traps that you can get caught with concern individual covariate values and changing the definition of PIMs. These issues are discussed in more detail below.

The precision of an estimator should ideally have 2 variance components: (1) the conditional sampling variance, given a model Mi with sampling variance var-hat(theta-hat | Mi) and (2) variation associated with model selection uncertainty. Buckland et al. (1997) provide an effective method to estimate an estimate of precision that is not conditional upon a particular model. Assume that the scalar parameter is in common to all models considered (e.g., phi, or N, or S). This will often be the case for our full set of a priori models, and is always the case if the objective is prediction. [If our focus is on a model structural parameter that appears only in a subset of our full set of models, then we must restrict ourselves to that subset in order to make the sort of inferences considered here about the parameter of interest.]

From Buckland et al. (1997) we will take the estimated unconditional var(theta-hat) as

var-hat(theta-hat) = {sum from i=1 to R of w(i) sqrt[var-hat(theta-hat | Mi) plus (theta-hat(i) - ave(theta-hat))^2] }^2,

where,

ave(theta-hat) = sum from i = 1 to R of w(i) theta-hat(i)

and the w(i) are the Akaike weights (delta(i)) scaled to sum to 1. The subscript i refers to the ith model. ave(theta-hat) is a weighted average of the estimated parameter over R models (i = 1, 2, ..., R). This estimator of the unconditional variance is clearly the sum of 2 components: the conditional sampling variance var-hat(theta-hat | Mi) and a term for the variation in the estimates across the R models (theta-hat(i) - ave(theta-hat))^2. The square root of these terms is then merely weighted by the w(i). This approach is very useful. Obviously, the estimated conditional

SE-hat(theta-hat) = sqrt[var-hat(theta-hat)] .

We expect even better approaches to be discovered over the next few years.

Model Averaging

Sometimes there are several models that seem plausible, based on the AIC or QAIC (or BIC or QBIC) values. In this case, there is a formal way to base inference on more than a single model. This entails a weighted average of the estimates of a parameter for R models. Akaike weights are a natural to use (alternative weights can come from estimates of model selection frequencies, based on the bootstrap). Again, we assume that the parameter is the same across the models (or that only a subset of models containing the parameter of interest is considered). Again, define the estimator

ave(theta-hat) = sum from i=1 to R of w(i) theta-hat(i)

and in this case, theta-hat is the parameter of interest. The estimator of the unconditional variance is the same as that given above.

Unconditional Confidence Intervals

The matter of a (1 - alpha)100% unconditional confidence interval is now considered. The simplest such interval is given by the end points

theta-hat +/- z(1 - alpha/2) SE(theta-hat)

where SE(theta-hat) = sqrt[ var-hat(theta-hat) ] .

Here, the confidence interval is set around a single theta-hat, or a model averaged estimate ave(theta-hat).

Output from the model averaging procedure is placed in a NotePad Window for viewing.

Unconditional Variance-Covariance Matrix of Model Averaged Estimates

Two check boxes are available at the bottom of the tabbed screen to obtain the variance-covariance of the model averaged estimates. The first check box puts the variance-covariance matrix in a NotePad Window. The second check box puts the variance-covariance matrix into a DBF file that you can retrieve into a spreadsheet.

Multiple Data Types, Changed PIM definitions, and Individual Covariates: Traps to Watch when Model Averaging

Because you may have used the Change Data Type option, the Results Browser may have models computed from more than one data type. Because each data type has its own distinctive arrangement of parameters in PIMs, mixing data types will generally cause problems. A check box at the bottom of the model averaging window is normally checked to indicate that only models for the current data type are used. If this box is unchecked, then the user runs the considerable risk of mixing parameters that will result in nonsense in the model averaging results.

Likewise, you may have changed the definition of some of the PIMs with the Change PIM Definition menu choice, and so some models with PIMs defined differently from the rest may result in incorrect results. For example, if the first transition probability from strata A is defined in some models as psi A to A, and in other models as psi A to B, and these real parameters are model averaged, the results may be incorrect. Thus, be sure to check the model averaging results to verify that correct parameters were selected.

A final trap concerns individual covariates. The user can specify the values of individual covariates to be used to compute the real and derived parameter values. If different values of the individual covariate are specified for different models to be model averaged, the results will be nonsense. Thus, be sure to use the same individual covariate values in all models to be model averaged, e.g., the mean value.

More details on model averaging procedures are given in the WWW file http://www.cnr.colostate.edu/class_info/fw663/average.pdf.

Some problems are too large for the interactive interface for specifying the parameters to be model averaged. An option is available in the Set Preferences window to change the interface to a less interactive mode. Also, push buttons have been added to the model averaging parameter selection windows to select all the parameters in the top row, or all the parameters in the first diagonal to speed up the process of selecting parameters.