If you fit a block as a fixed effect in a saturated model, and its effect is not significant after you reduce it via stepwise regression, do you need to add it back to the model?

[#1]

This was posted to the JMP blog yesterday where Phil Kay describes reducing a saturated model with a block effect. He says the random block should be added as a fixed effect 1st in this specific case, then after you reduce the model via stepwise regression, you should re-fit the reduced model & add the block as a random effect. In his example the block was significant after he reduced the model. I'm wondering if the block does come up as significant, is there any reason to add it back into the model as a random effect, or can you just leave it out all together?

https://community.jmp.com/t5/Phil-Kay-s-Blog/Model-Selection-for-Designed-Experiments-with-Blocks/ba...

[#2]

Here are my thoughts on the topic of blocking.

First, blocking is a technique to handle those factors that you are not willing to control.  I label those factors noise.  The reasons for not willing to control factors is threefold:

1. There is no capability or technology or the factor is too complex (e.g., customer expectations, supplied materials)

2. The cost to control is prohibitive (e.g., environmental conditions)

3. It would be inconvenient or impractical to control (e.g., variations in power source)

Whether to treat the block as a fixed or random effect depends on your ability to assign the factors to the block (confound factors with the block).  If you have done due diligence and have identified the specific noise factors, then confounding those factors with the block would allow for assignment of the block as a fixed effect (this also allows assigning block-by-factor effects to the model for assessment of factor robustness).  If you have not identified the specific noise factors confounded with the block, then treat the block as a random effect.  Blocking is a fantastic technique as it allows the experimenter to increase the inference space without negatively affecting the precision of the design.

I also do not believe stepwise is the appropriate method for building a model in DOE.  You determine what DOE to run as a function of which model you want insight into.  You should be using a subtractive approach to model refinement (not additive).  Start with a saturated model (your hypotheses) and remove insignificant terms to determine a reasonable model.

I'm confused by your query; "I'm wondering if the block does come up as significant, is there any reason to add it back into the model as a random effect, or can you just leave it out all together?"

If the block is significant, it means there are noise factors confounded with the block are of interest.  You can either disaggregate the block to determine what those specific factors are and then decide to manage them or you can iterate to be robust to them.  You would not want a model with a block effect as this is nonsensical (you can't manage the block).

Here is a paper you might want to read:

Sanders, D., Leitnaker M., and McLean R. (2002) “Randomized Complete Block Designs in Industrial Studies” Quality Engineering, Vol. 14, Issue 1

[#3]

Sorry for the confusion I meant to say : 

"I'm wondering if the block does NOT come up as significant, is there any reason to add it back into the model as a random effect, or can you just leave it out all together?"

What do you mean by "disaggregate the block"?

[#4]

Block, just like any other terms in the model, if found insignificant both practically and statistically can be removed from the model.  Of course all of the terms removed from the model will be pooled into the experimental error affecting P-values (they will likely rise as the MSE will get smaller).

If block is significant, you will need to determine which of the possibly many noise factors confounded with the block are active.  I call this disaggregating the block.

[#5 · Accepted solution]

Hi @mjz5448,

I haven't watched the video (yet), but I think that Phil's point of view may be more nuanced than this summary.

I have mixed feelings (no pun intended) about this possible approach, @statman already gave a lot of information.

The decision to have fixed or random block effect is set before the analysis, depending on how you consider this blocking factor :

• If the level can be reproducible or if the specified levels are the only ones of interest, then you can treat it as a fixed block effect and it will affect response mean (bias).
• If the levels are a sample from a larger population or if you are not really interested in these two (or more) particular levels, but variation across the population, then treat it as a random effect and it will affect response variance.

Concerning your specific question, I believe the question is intended in the case of a insignificant random block effect. The random block effect captures a part of the variation due to the block effect, so removing it would increase the unexplained variance of your experiments, resulting in less precise fixed effects estimates and higher p-values. Even if not statistically significant, I would not remove the random effect from the model, as this random block is essential to capture part of the experimental noise (day-to-day variation, equipment variability, etc...).

Hope this clarify the use of random block effect,

[#6]

Hi Victor, 

You're correct, I meant to ask what to do if the block is NOT significant. Sorry for the confusion. 

So even if the random block is not significant, you always typically leave it in the model? Doesn't the fact that it's not significant mean that the random noise isn't too great? I wonder if fitting models both with & without the block effect and comparing error is appropriate? 

[#7]

I agree with @Victor_G regarding leaving in your random block. If you decide to treat block as fixed you could fit with and without the block and compare your other parameter estimates of interest to show if and how the inclusion/exclusion of the blocking factor biases these other estimates.

If you treat block as random and you fit with and without and compare the error, MOST of the time the model with the block will have less error since the inclusion of the blocking factor "takes on" some of the unexplained variance. I say most because I believe a negative variance component estimate would do the opposite but not entirely sure on that.

[#8]

FYI Victor - Phil responded to my question on his blog post and likewise suggested leaving the random block effect in the model, even if it isn't significant.