Video 1: Challenge - Beyond One Best Model: What a DSD Can Really Tell You
The Q2 session of the Experimenter’s Club featured a lively discussion on George Box’s famous reminder that "all models are wrong, but some are useful."
It began with a user whether unexpectedly strong interaction and quadratic effects in a 6-factor Definitive Screening Design (DSD) should be trusted. What followed was a thoughtful exchange showing that the real issue is ,” but how to interpret a design space in which many plausible models coexist.
Community Question: https://community.jmp.com/t5/Discussions/quot-Surprising-quot-results-in-an-DSD-Design/td-p/941518
LinkedIn post from Victor: https://www.linkedin.com/posts/victorguiller_experimentersclub-designofexperiments-share-74534405586...
Video 2: Discussion
The conversation highlighted that in a DSD with a limited number of runs, it is impossible to estimate every main, interaction, and quadratic effect simultaneously without structural tradeoffs. As a result, partial aliasing, multicollinearity, and model multiplicity are natural features of the problem rather than signs of failure.
The discussion emphasized that different estimation methods and selection criteria, such as R², adjusted R², AICc, BIC, and RMSE, can lead to different but still defensible models.
A key takeaway is that experimenters should not focus only on finding a single best-fitting model. Instead, they should compare multiple strong candidates, look for effects that appear consistently across good models, apply effect hierarchy and domain knowledge, and consider augmentation when confirmation is needed.
In that sense, the design process becomes less about reacting to surprising results and more about disciplined model judgment in real-world experimental work.
