Choose Language Hide Translation Bar
Highlighted
frankderuyck
Level IV

Re: Analysis of a custom design with blocks

OK thanks for inputs. Pity that stewise can't handle random effects, would be great; backward analysis is cumbersome when many factors are involved. Is the AICc criterion not OK for judging the models; the first model has 17 significant effect R² = 0,98 but AICc is much higher than the 2nd lower #effect model with R² = 0,9 ?

Highlighted
cwillden
Super User

Re: Analysis of a custom design with blocks

This response was created in duplicate by accident. Deleted content
-- Cameron Willden
Highlighted
cwillden
Super User

Re: Analysis of a custom design with blocks

Ultimately, you can’t know if your overfit until you do some confirmation runs. You really have a screening design, so I wouldn’t be overly concerned with overfitting at this stage as long as a good chunk of model terms drop out. I wouldn’t be expecting to make accurate predictions inside your design space yet either.
There’s not a wrong or right way here. If you go stepwise, then yes you’ll have to treat blocks as fixed. This will suck up a lot of degrees of freedom, making all effect tests less powerful. You should be able to enforce whole effects only so that some of terms for individual blocks don’t drop out.
After stepwise selects a model, you should manually change block back to a
a random effect in a regular least squares fit with REML.
Stepwise is usually looked down upon these days in favor of methods with regularization. You don’t get that in base JMP, so stepwise is really the only answer for automated model selection.
I typically do model selection manually. There’s also the Screening platform for screening design analysis. I haven’t used it in a couple years, but that’s worth a look.
-- Cameron Willden
Highlighted
frankderuyck
Level IV

Re: Analysis of a custom design with blocks

Thanks Cameron for this useful comment; I will have a look at this more or less forgotten DOE screening analysis platform. I agree that the fixed block approach consumes lot of degrees of freedom, on the other hand this will make sure that finally only strong effects will be screened out so I am happy with my lower but stonger effect R² = 0,9 model; I have tried the cumbersome standard least squares backward selection and sometimes it is hard to judge when to remove an effect or not; would you reject an interaction effect with p = 0,075? Using backward selection, nearly every time I find another model.. judgement of p is critical!! I prefer fixed effect/stepwise and indeed by creating the final model block must be transformed to a random effect. Regards, Frank

Highlighted

Re: Analysis of a custom design with blocks

The screening platform is for two-level factors. It will model any factors with more than two levels (such as 9 blocks) as powers of fixed effects, up to the 8th power in this case, and not as random effects. You should not use the Screening platform in this case.

Also, this simple platform is for screening factors, not effects. It ignores your specified model to create contrasts based on the key principles of screening.

Learn it once, use it forever!
Article Labels

    There are no labels assigned to this post.