JMP 16- Confidence Intervals getting wider during backwards elimination of fixed effects in a mixed model DoE?

[#1]

Hello,

Apologies if this has been asked before. I used the search function before posting and couldn't find anything on this forum.

I'm exploring mixed models for DoE in an effort to gain deeper understanding on how fixed and random effects contribute to a process output. However, when I try to manually use backwards elimination to simplify my fixed effect regression parameters, I notice that the confidence intervals for my parameter estimates (ex: my intercept, which is important for my purposes) get really wide, which is also reflected by wide and uneven confidence intervals in the profiler. 

I would have thought that simpler models would free up more degrees of freedom for better variance / standard error measurements and thus make the confidence intervals smaller, but the opposite appears true. It's worth noting these effects are more pronounced when I increase the complexity of my random effects (ex: concatenating parameters as a random block vs performing variance component analysis on each random factor individually).

See below of an example DoE I have been trying to build and explore using simulated results (not shown):

X1 X2 and X3 would be fixed factors, while the rest are my example random factors.

Any help would be appreciated! So far it just seems like my best course of action would be to retain the full response surface model and not explore any stepwise elimination options at this point.

Best,

Matt

[#2 · Accepted solution]

I would not use the length of the confidence interval on the estimates to select a model. They reflect the uncertainty in the estimates given the selected model.

They intervals lengthen because removing a term adds the associated sum of squares to the error term. The error term is used to estimate the standard error of each estimate.

[#3]

First, welcome to the community.  I'm not sure I can answer your questions as I don't have enough context (like what is your goal?), but here are my thoughts.  As Mark already answered, when you remove terms from the model, those SS's and DF's are pooled in the MSE.  I would be using other techniques to reduce the model, for example

1. First and foremost, SME (subject matter experts) and their related hypotheses.  Do the parameter effects and coefficients make sense?

2. Check for collinearity (Analyze>Multivariate Methods>Multivariate (and test for outliers))

3. As you remove or add terms to the model, watch the delta between RSquare and RSquare Adjusted.  The larger the delta, the more over specified the model

4. RMSE, the model with the smaller RMSE the better

5. p-values

By the way, confidence intervals are related only to the data in hand.  The ability to extrapolate those into the future is completely dependent on how representative your study is of the future.

[#4]

Thank you both for the fast replies. I was reducing my model via manual backwards elimination by minimizing AICc and was curious as to why my CIs were becoming wider. Your explanations make sense.