As part of our ongoing series of blogs about testing software, we’re going to take a bit of a detour to revisit the fundamental principles of factorial effects.
Particularly, when it comes to screening designs, underlying which properties we look for in a design can be summarized nicely by the fundamental principles of designed experiments. We even outline them in our documentation.
- Effect hierarchy: This principle specifies that main effects are more likely to be more important than second-order effects, which are more likely to be important than third-order effects and so on. In addition, effects of the same order are equally likely to be important.
- Effect sparsity: According to this principle, only a small number of effects in a factorial experiment are likely to be significant. Most effects are assumed to be negligible, which simplifies the model selection process.
- Effect heredity: This principle states that an interaction effect is likely to be significant only if at least one of its parent (main) effects is also significant. This specification helps in ruling out incompatible models during the model search process.
Now, if you’ve done enough experiments, you likely have examples where these principles do not hold. However, there is empirical evidence that these are reasonable assumptions to make when you’re experimenting (see Li et al. 2006 for a review).
The effect hierarchy principle is the underlying basis of many of the design choices in fractional factorial designs through resolution and minimum aberration that you may have learned if you took a design of experiments course. Combined with sparsity, where we want to find the most important factors, these are active research areas (see for example Definitive Screening Designs and mixed-level screening designs). Similarly, when you use the Custom Design platform, the default model is a main effects model.
I want to point out that for the sparsity principle, the “likely to be significant” is more a reflection of identifying the “most” important effects. If you have a sufficiently large run size, many (possibly most) of your factorial effects will be statistically significant – I like to think of the sparsity principle as getting the largest effects, getting the biggest bang for your buck for changing your response by changing certain corresponding inputs.
For the effect heredity principle, I typically only think about it when it comes to modeling. But we will see that it, too, is relevant to testing.
Why the detour? Is this related to testing?
While we could have simply pointed you to the documentation, which is in my next blog post, we’ll see how we can frame the testing problem using these same principles.
Li, X., Sudarsanam, N. and Frey, D.D., 2006. Regularities in data from factorial experiments. Complexity, 11(5), pp.32-45.
