Analysis of a custom design with blocks

[#1]

Using the custom designer a 36 run DOE was created for 7 factors: 5 continuous and 2 categorical. Each day 4 runs are performed so a blocking factor with 9 blocks is added to the input parameter list. DOE evaluation shows a nice parameter power, acceptable low correlation among the factors and the blocks nearly orhogonal to the factors. Analysis of the results by fitting a response surface can be done in several ways; which one of below is correct/recommended? I consider block as a random effect.

1. Transforming block to random effect and using Standard least squares + REML: after removing non effective fparameters --> 17 active effects, lot of interactions & quadratic effects: R² = 0,97 R²adj = 0,94 AICc = 327 This looks to overfitting to me..? However R²adjusted is still close to R²...

2. Stepwise & block = fixed effect (stepwise does not accept random effects?):10 fixed effects: 3 fixed block effects and 7 parameters --> 5 main effects, 1 interaction and 1 quadratic effect. Consideringt block as a random effect, before making the model I transformed the 3 fixed block effects to 3 random effects, is this correct? Making the REML model I get R² = 0,90 R²adj = 0,87 AICc = 256

The two models are clearly different! I would prefer the second model with lower #effects & AICc and still R² = 0,9. There are too few runs to create a testset so what is your opinion?

Remark: in the 2nd, stepwise procedure, instead or assigning the 3 fixed block effects as 3 random effects I also can create a REML model by taking up the 9 level block as one random effect, is this the right way?

Thanks for input! Frank

[#2]

Hi Frank,

You will find discussion on random vs fixed effects for blocked experiments in Goos & Jones, Optimal Design of Experiments. There is a free chapter download available that is on the subject of a blocked RSM. There is another chapter in the book about a blocked screening design that is also relevant.

I don't think there is a simple answer to your question. However, I would say that it is agreed that estimating blocks as random effects is more "statistically proper". When you create a blocked design with Custom Design it should give you a Fit Model dialog with the block effect as a randon effect and REML fitting.

In your approach #2 it doesn't make sense to me to take the 3 fixed block effects that you have determined to be significant and then change them to random effects. You should, as you say, take the 9-level block as a single random effect.

36 runs for a blocked RSM with 7 factors seems quite a small number of runs. I am not sure how you designed this in Custom Design. I think it would suggest more rusn than that.

Regards,
Phil

[#3]

Hi Phil, it was quite easy to create a 9 block 7 factor custom DOE with 36 runs; model terms are main and powers, alias terms are 2nd interactions.

When I try to fit a response surface by selecting stepwise I get a JMP Alert that "random effects are not suported by stepwise"?? So I must first make a stepwise fixed block analysis and then make a model with random block.. is this a jusitified way?

[#4]

Thanks, Frank.

So the model specified in Custom Design had just the main effects and quadratics? The 2-factor interactions were not in the model? In which case, 36 runs sounds more sensible.

You are correct that random effects are not supported in stepwise or in the Gen Reg platform (you might want to add this in the Wish List). Probably the best method is to fit the REML model with the block random effect and all the other effects and do a "manual" backwards stepwise regression from the Effect Summary. It sounds like you may have already done this.

One thing to consider is how big the block effect is relative to the residual error - look in REML Variance Component Estimates in the fit model report. If the variance estimate 95% lower and upper limits include 0, you might want to take out the block effect altogether. Then you can use stepwise as usual.  

It sounds like you were also trying to fit a mode with significant interactions, although you didn't specify these in the model in Custom Design. So it will depend how good the experiment is for estimating these effects.

Phil

[#5]

I should clarify that taking out the block effect effect is something that you can try as part of exploring different possible models. Personally, I would feel more comfortable having the blocknig effect in the final model, even if it is not judged to be significant.

[#6]

I disagree with Phil's advice about the choice of treating the blocking effect as fixed or random. It depends. There are many cases in which a fixed effect interpretation is meaningful and justifiable. In your case, I agree with you and Phil that the Day effect is best interpreted as a random effect.

You add a blocking factor if you want to treat the block effect as fixed. You do not add a blocking factor is you want to treat the block effect as random. You select the "Group runs into random blocks of size 4" option in the Design Generation section of Custom Design instead of adding a blocking factor.

[#7]

Sorry but I am confused by the comments above.I don't undersrand and still have no answer on my question.

I created a nice 36 run blocked DOE with 9 4-run randomized blocks. When I look to the color map with main, interactions and powers everything is nearly blue indicating no or only few correlation among quadratic and interaction effects. Power calculations also are fine so this looks like a nice DOE to me? Only question is how to analyze? In the book Optimal design by Goos and Jones it is recommended to extract the random variation generated by the daily blocks by assigning them as random effects and using REML so I did. However by standard least squares and backward I get a completely different model (17 effects!) compared to stepwise (see above) But! using stepwise I have to analyse all factors fixed - which is wrong because block is a random effect -  and then create a model assigning block as random; I get a nice lower effect model different from the first with R² = 0,9 but is this way of analyzing the DOE OK i.e.first block = fixed and then random?? Which model is now the best guess? How to check?

[#8 · Accepted solution]

Hi Frank, I am sorry that you have not found the replies to be useful. We are trying to help! Unfortunately, there is nothing that brings more confusion than fixed verus random effects!

Having said that, I think you have already answered your own question.

"In the book Optimal design by Goos and Jones it is recommended to extract the random variation generated by the daily blocks by assigning them as random effects and using REML so I did." Mark and I both agree that is the appropriate analysis for this. But you don't seem to like the answer that gives you.

We all agree that using stepwise and blocks as fixed effects is not appropriate in this case: "using stepwise I have to analyse all factors fixed - which is wrong because block is a random effect".  But you prefer the answer that it gives you.

"Which model is now the best guess? How to check?" You can decide which model is "best". It depends on what "best" means to you.

[#9]

Quoting George Box: "All models are wrong. Some are useful." Pick the one that you find useful, but don't think that it is "the truth". Another quote from Box: "Never fall in love with a model."

So, the hard-core traditionalist approach:

The design that is built depends on the model to be estimated. Changing the model means you may not have the best design any longer. You have a random block. You created the design as a random block. Your analysis should have a random block.

The non-traditional approach:

You could certainly use your stepwise approach, but realize that by treating the random effect as fixed, you are changing your error term. Therefore, the model that stepwise comes up with is not using the same error term as your first analysis. If you could use the proper error term, you may end up with different selected model terms. The design was not built for this analysis, so why would you have tried it??

Finally, to help you answer your question of which model is best:

You ultimately have two competing models. How do their predicted values compare? Are there regions of the design space that give you drastically different predictions? As with all models from a DOE, you need to verify it. Pick a couple of conditions to test. Try to include locations where the models differ quite a bit. Run those tests. Which model does better? Ultimately, YOU decide which model is best. I would base that decision on real-world performance.

[#10]

I brought up the fact that when you used Custom Design you did not specify a model including the interactions. Why did I mention this? Because I was wondering how good the experiment is for estimating the interactions that you are fitting in your models. I was wondering if this is contributing to your challenge in finding the best model.

You say the design has good power and correlation for these effects. In which case, don't worry about this.