Accepted Solutions
Hi @frankderuyck,
Space-filling designs and optimal designs (no matter if it is for mixture or other factors type, they can be used in both cases) are two completely different DOE mindsets: Model-based vs. Model-agnostic.
- Model-based : This type of designs of experiments involves postulating a specific model a-priori (with main effects, interactions terms, etc...), and then generating points in order to optimize the estimation of the model's coefficients. This approach assumes a specific model form and is useful when the underlying model is well-known or when there is prior knowledge about the system being studied. It may requires fewer points, and is particularly advised if you want to easily have access to explainable/interpretable results.
Example : My response Y is heavily influenced by statistically significants main effects from factor A and B, and by the statistically significant 2-factors interaction between A and B.
- Model-agnostic : This type of designs of experiments involves distributing points uniformly/homogeneously (and randomly) in the experimental space and then fitting different models to these points to obtain the best predictive model. This approach may require a high number of points depending on the complexity of the model tested (Regression, SVM, Neural Networks, Tree-based models,...), does not assume any specific model, and is useful when the underlying model is unknown or complex. But this approach may be sensitive to noise and the use of ML models may lead to overfitting situations...
Both approaches have their advantages and disadvantages. Model-agnostic designs can be more flexible and robust to model misspecification (as they don't require any), but they may require more data to achieve comparable accuracy to model-based designs. Model-based designs can be more efficient and require less data, but they are more vulnerable to model misspecification and may be less robust in the face of complex or unknown underlying models.
The evaluation can also be different between these two DoE mindsets, with more emphasis on statistical significance with optimal designs (except for Response Surface Models and Mixture designs), vs. predictive accuracy (RMSE and other predictive performance metrics) with model agnostic designs (you're trying to have the best predictions in your experimental space, not find a significant factor or terms in a model).
Please find below a slide from presentation that I give on this specific topic, I hope it will help you make it easy/accessible for students :
I hope this answer will help you
Completely agree with you @statman, depending on the objective and information about a product/process, the two design methods can be used sequentially : screen factors first with a model-based optimal approach, and then use a model-agnostic approach to optimize a predictive model in the eventuality of non-linear response.
Here is a slide from Synthace about the different designs and their possible complementarity :
There was also a paper on this topic of combining different DoE methodologies : https://community.jmp.com/t5/Discovery-Summit-Americas-2020/DOE-Gumbo-How-Hybrid-and-Augmenting-Desi...
Hi @frankderuyck,
Definitive Screening Design is a "textbook" (classical) design, used for a screening case with high number of factors involved and the possibility to test interaction and quadratic effects, following the basic principles of DoE : effect sparsity, effect hierarchy and effect heredity. I wouldn't categorize this design as an "optimal" design, as the structure is "fixed" (and so are the terms screened by this design).
Concerning your conditions,
- Main effects are completely orthogonal to each other, to 2-factors interactions and to quadratic effects.
- No, there are aliases between quadratic and interaction effects :
Here is an extract from the presentation of Tom Donelly about the desirable properties of DSDs I attached earlier :
I hope this answer will help you.
You can also search for presentations about DSD from Bradley Jones in the JMP Community, he clearly demonstrates when/where to use DSDs, and the conditions to fulfill (only continuous and few 2-levels categorical factors, high number of factors, no constraint in the experimental space, easy to change factors, ...).
Hi @frankderuyck,
Space-filling designs and optimal designs (no matter if it is for mixture or other factors type, they can be used in both cases) are two completely different DOE mindsets: Model-based vs. Model-agnostic.
- Model-based : This type of designs of experiments involves postulating a specific model a-priori (with main effects, interactions terms, etc...), and then generating points in order to optimize the estimation of the model's coefficients. This approach assumes a specific model form and is useful when the underlying model is well-known or when there is prior knowledge about the system being studied. It may requires fewer points, and is particularly advised if you want to easily have access to explainable/interpretable results.
Example : My response Y is heavily influenced by statistically significants main effects from factor A and B, and by the statistically significant 2-factors interaction between A and B.
- Model-agnostic : This type of designs of experiments involves distributing points uniformly/homogeneously (and randomly) in the experimental space and then fitting different models to these points to obtain the best predictive model. This approach may require a high number of points depending on the complexity of the model tested (Regression, SVM, Neural Networks, Tree-based models,...), does not assume any specific model, and is useful when the underlying model is unknown or complex. But this approach may be sensitive to noise and the use of ML models may lead to overfitting situations...
Both approaches have their advantages and disadvantages. Model-agnostic designs can be more flexible and robust to model misspecification (as they don't require any), but they may require more data to achieve comparable accuracy to model-based designs. Model-based designs can be more efficient and require less data, but they are more vulnerable to model misspecification and may be less robust in the face of complex or unknown underlying models.
The evaluation can also be different between these two DoE mindsets, with more emphasis on statistical significance with optimal designs (except for Response Surface Models and Mixture designs), vs. predictive accuracy (RMSE and other predictive performance metrics) with model agnostic designs (you're trying to have the best predictions in your experimental space, not find a significant factor or terms in a model).
Please find below a slide from presentation that I give on this specific topic, I hope it will help you make it easy/accessible for students :
I hope this answer will help you
Hi @frankderuyck,
Definitive Screening Design is a "textbook" (classical) design, used for a screening case with high number of factors involved and the possibility to test interaction and quadratic effects, following the basic principles of DoE : effect sparsity, effect hierarchy and effect heredity. I wouldn't categorize this design as an "optimal" design, as the structure is "fixed" (and so are the terms screened by this design).
Concerning your conditions,
- Main effects are completely orthogonal to each other, to 2-factors interactions and to quadratic effects.
- No, there are aliases between quadratic and interaction effects :
Here is an extract from the presentation of Tom Donelly about the desirable properties of DSDs I attached earlier :
I hope this answer will help you.
You can also search for presentations about DSD from Bradley Jones in the JMP Community, he clearly demonstrates when/where to use DSDs, and the conditions to fulfill (only continuous and few 2-levels categorical factors, high number of factors, no constraint in the experimental space, easy to change factors, ...).