When should you only evaluate main effects? (Not main + 2nd Order Interaction)

[#1]

Hi,

I'd like to know when does a situation call for evaluation of only main effects, while ignoring interaction effects.

Me and a colleague have different approaches to how screening designs are performed so I'd like to understand what others think.

We can assume the following:

• This is a screening design (Let's say 4 factors - 2 Level)
• Nothing is known of the interactions up to this point and in theory, any interaction could exist
• Material (Experiment #) and time is not limited
• Follow up DoE's can be performed to augment the model

I appreciate efficiency, and will always gear experiments toward the "Greedy scientist" mentality when possible, however, I will not do this while compromising the model itself. My concern is that when interactions could be present, they may not be identified if a main-effects only design is implemented. If a factor is eliminated from the screen due to the main effect being found insignificant, can't that risk identifying the true optimal if there was a potential that a positive interaction was present?

[#2]

Here are my thoughts:

First a clarification, when you are running factorial designs, the interactions are present, they just may be aliased and therefore not specifically assignable.  Your concerns are legitimate.  However, if you commit to sequential experimentation, errors you make in interpretation of initial experiments will be uncovered in the next iteration.

There is no one correct approach to experimentation.  Given the situation you pose I have the following comments:

1. 4 factors is not much of a screening design.  You can run 4 factors in resIV in 8 treatments.  I think screening designs have ≥5 factors in which case you start realizing the benefits of fractionating and sequential design.

2. I always start with SME (domain knowledge) as a basis for determining resolution.  I suggest predicting the rank order of 1st and 2nd order effects.  Are the 2nd order effects possible/probable?  As 2nd order effects rise in your rank ordering, consider higher resolution designs.

3. Consider the design space.  Higher order effects (e.g., interactions and curvature) occur inside the space.  So the first objective is to move your space to "near" optimum (of course, no-one knows where optimum is...we rely on SME to provide guidance).   Then augment the space.  If you are far from optimum, it may not be efficient to study inside the space.  We all know the fastest way between two point is a straight line, so use factors at 2 levels to move the space.  This is the idea behind screening.  Experiment on lots of factors hoping to find factors that move the space quickly and focus on the significant factors.

4. Don't ignore noise.  In just about every case, the number of factors you manipulate is always a small subset of all factors.  What do you do with the others?  The correct answer is NOT to hold them constant. What strategy should you use to partition and assign the noise (e.g., repeats, replicates, blocks, split-plots, nesting)?  This is IMHO the most challenging part of experimentation and the least taught.  Unfortunately, the software does not give guidance.

In all cases, design multiple experiments.  Compare and contrast.  What will each do in terms of assigning, confounding and restricting factor effects?  Weigh this against resources and pick one.

"The best design you'll ever design is the design you design after you run it"

[#3]

Thanks for the thorough explanation. Just to confirm, if an interaction is confounded by the screening design, will it still identify each factor in the interaction as significant? This can then be followed up with further experiments to search for confounded factors through interactions?

Example:

Main effect screen:

Model: X1, X2, X3, X4

        Significant factors found: X1, X2, X3

D/I optimal:

Model: X1, X2, X3, X1*X2, X2*X3 (Potentially X1^2, X2^2, X3^2)

And in this example, there is reasonable evidence to say that X4 did not provide any interaction with X1-X3?

Thanks

[#4 · Accepted solution]

That is a great question, but unfortunately the answer is not definitive.  While we do consider principles of scarcity, hierarchy and heredity (in order for a higher order effect to be active at least 1 parent should be active) to be guiding principles, they are not absolutes.  Interestingly, F=ma...m (mass) and a (acceleration) main effects are not in the model.  Only the interaction.

In some cases the alias structure can be known and therefore you can surmise what possible (confounded) effects are active.  These can be separated in subsequent studies (e.g., fold over).  Experiments with partial aliasing are more difficult to disaggregate.  While what you actually will do in the next experiment iteration won't be decided until you analyze the first experiment, you should predict all possible outcomes and be anticipate all possibilities.