Analysis of a custom design with blocks

[#1]

Using the custom designer a 36 run DOE was created for 7 factors: 5 continuous and 2 categorical. Each day 4 runs are performed so a blocking factor with 9 blocks is added to the input parameter list. DOE evaluation shows a nice parameter power, acceptable low correlation among the factors and the blocks nearly orhogonal to the factors. Analysis of the results by fitting a response surface can be done in several ways; which one of below is correct/recommended? I consider block as a random effect.

1. Transforming block to random effect and using Standard least squares + REML: after removing non effective fparameters --> 17 active effects, lot of interactions & quadratic effects: R² = 0,97 R²adj = 0,94 AICc = 327 This looks to overfitting to me..? However R²adjusted is still close to R²...

2. Stepwise & block = fixed effect (stepwise does not accept random effects?):10 fixed effects: 3 fixed block effects and 7 parameters --> 5 main effects, 1 interaction and 1 quadratic effect. Consideringt block as a random effect, before making the model I transformed the 3 fixed block effects to 3 random effects, is this correct? Making the REML model I get R² = 0,90 R²adj = 0,87 AICc = 256

The two models are clearly different! I would prefer the second model with lower #effects & AICc and still R² = 0,9. There are too few runs to create a testset so what is your opinion?

Remark: in the 2nd, stepwise procedure, instead or assigning the 3 fixed block effects as 3 random effects I also can create a REML model by taking up the 9 level block as one random effect, is this the right way?

Thanks for input! Frank

[#2]

OK thanks for inputs. Pity that stewise can't handle random effects, would be great; backward analysis is cumbersome when many factors are involved. Is the AICc criterion not OK for judging the models; the first model has 17 significant effect R² = 0,98 but AICc is much higher than the 2nd lower #effect model with R² = 0,9 ?

[#3]

This response was created in duplicate by accident. Deleted content

[#4]

Ultimately, you can’t know if your overfit until you do some confirmation runs. You really have a screening design, so I wouldn’t be overly concerned with overfitting at this stage as long as a good chunk of model terms drop out. I wouldn’t be expecting to make accurate predictions inside your design space yet either.
There’s not a wrong or right way here. If you go stepwise, then yes you’ll have to treat blocks as fixed. This will suck up a lot of degrees of freedom, making all effect tests less powerful. You should be able to enforce whole effects only so that some of terms for individual blocks don’t drop out.
After stepwise selects a model, you should manually change block back to a
a random effect in a regular least squares fit with REML.
Stepwise is usually looked down upon these days in favor of methods with regularization. You don’t get that in base JMP, so stepwise is really the only answer for automated model selection.
I typically do model selection manually. There’s also the Screening platform for screening design analysis. I haven’t used it in a couple years, but that’s worth a look.

[#5]

Thanks Cameron for this useful comment; I will have a look at this more or less forgotten DOE screening analysis platform. I agree that the fixed block approach consumes lot of degrees of freedom, on the other hand this will make sure that finally only strong effects will be screened out so I am happy with my lower but stonger effect R² = 0,9 model; I have tried the cumbersome standard least squares backward selection and sometimes it is hard to judge when to remove an effect or not; would you reject an interaction effect with p = 0,075? Using backward selection, nearly every time I find another model.. judgement of p is critical!! I prefer fixed effect/stepwise and indeed by creating the final model block must be transformed to a random effect. Regards, Frank

[#6]

The screening platform is for two-level factors. It will model any factors with more than two levels (such as 9 blocks) as powers of fixed effects, up to the 8th power in this case, and not as random effects. You should not use the Screening platform in this case.

Also, this simple platform is for screening factors, not effects. It ignores your specified model to create contrasts based on the key principles of screening.