8 Comparing the AIC to the BIC AIC BIC .BIC tends to have models that are "too ●AIC tends to have models that are“too big",good for prediction (perhaps),but small",but still have optimal predictive properties.Will not exaggerate impor- not so good for understanding whether tance of covariates (sometimes too con- specific covariates are important or not. servative). ●Tends to overfit. Tends to underfit (but see next point!). Not consistent:Even with an infinite Consistent,asymptotically will return sample size,will not necessarily converge to correct model,tends to remain too big. the correct model. Performs better for simple models, Performs better for complex models "Truth is low dimensional". "Truth is high dimensional" Usable for non-nested models. Usable for non-nested models. Not related to any significant tests. Equivalent to a stepwise procedure with a critical value of 15.7%. Asymptotically equivalent to a Bayes Factor,so easily leads to model probabil- Not equivalent to a Bayes Factor,so no natural way to get model probabilities,so ities,so easy to perform model averaging no model averaging. (done automatically in BIC software for R). Example of using the R BMA package for Bayesian model selection Here is an example of using the BMA model selection program First,we will generate some data: x <-rnorm(1000,mean=0,sd=4) z <-rnorm(1000,mean=5,sd=20) w<-rnorm(1000,mean=-3,sd=4) y <rnorm(1000,mean 3*x 2*z,sd=25) Y will be our independent variable,and we will create a matrix for the other variables as follows: data.matrix <-matrix(c(x,w,z),byrow=F,nrow=1000)8 Comparing the AIC to the BIC AIC BIC • AIC tends to have models that are “too big”, good for prediction (perhaps), but not so good for understanding whether specific covariates are important or not. • Tends to overfit. • Not consistent: Even with an infinite sample size, will not necessarily converge to correct model, tends to remain too big. • Performs better for complex models “Truth is high dimensional”. • Usable for non-nested models. • Equivalent to a stepwise procedure with a critical value of 15.7%. • Not equivalent to a Bayes Factor, so no natural way to get model probabilities, so no model averaging. • BIC tends to have models that are “too small”, but still have optimal predictive properties. Will not exaggerate impor￾tance of covariates (sometimes too con￾servative). • Tends to underfit (but see next point!). • Consistent, asymptotically will return the correct model. • Performs better for simple models, “Truth is low dimensional”. • Usable for non-nested models. • Not related to any significant tests. • Asymptotically equivalent to a Bayes Factor, so easily leads to model probabil￾ities, so easy to perform model averaging (done automatically in BIC software for R). Example of using the R BMA package for Bayesian model selection # Here is an example of using the BMA model selection program # First, we will generate some data: x <- rnorm(1000, mean=0, sd=4) z <- rnorm(1000, mean=5, sd=20) w <- rnorm(1000, mean=-3, sd=4) y <- rnorm(1000, mean = 3*x + 2*z, sd=25) # Y will be our independent variable, and we will create a matrix for # the other variables as follows: data.matrix <- matrix(c(x,w,z), byrow=F, nrow=1000)