Recording Experimenters' Club Q2 2026_Beyond One Best Model: What a DSD Can Really Tell You

[#1]

Video 1: Challenge -  Beyond One Best Model: What a DSD Can Really Tell You

The Q2 session of the Experimenter’s Club featured a lively discussion on George Box’s famous reminder that "all models are wrong, but some are useful."

It began with a user questioning whether unexpectedly strong interaction and quadratic effects in a 6-factor Definitive Screening Design (DSD) should be trusted. What followed was a thoughtful exchange showing that the real issue is not whether one model is “correct,” but how to interpret a design space in which many plausible models coexist.

Community Question: https://community.jmp.com/t5/Discussions/quot-Surprising-quot-results-in-an-DSD-Design/td-p/941518

LinkedIn post from Victor: https://www.linkedin.com/posts/victorguiller_experimentersclub-designofexperiments-share-74534405586...

Video 2: Discussion

The conversation highlighted that in a DSD with a limited number of runs, it is impossible to estimate every main, interaction, and quadratic effect simultaneously without structural tradeoffs. As a result, partial aliasing, multicollinearity, and model multiplicity are natural features of the problem rather than signs of failure.

The discussion emphasized that different estimation methods and selection criteria, such as R², adjusted R², AICc, BIC, and RMSE, can lead to different but still defensible models.

A key takeaway is that experimenters should not focus only on finding a single best-fitting model. Instead, they should compare multiple strong candidates, look for effects that appear consistently across good models, apply effect hierarchy and domain knowledge, and consider augmentation when confirmation is needed.

In that sense, the design process becomes less about reacting to surprising results and more about disciplined model judgment in real-world experimental work.

[#2]

Thanks @maria_astals for the recording !
Here is the detailed process to create a Raster Plot from a DoE:

1. Go to Fit Model platform (or click on the script Model in the DoE table), specify the response(s) and terms in the assumed model (if these  panels are not already pre-filled) and choose the Personality Stepwise. Click on Run.
   
   
   

2. Go to the red triangle next to Stepwise Fit, select All Possible Models, and define the maximum number of possible terms in the models as well as number of best models to see. Check the option Restrict to models where interactions imply lower order effects (Heredity Restriction). Click OK.

   
   
   

3. Once the calculations are done, a new table All Possible Models appears in the report. RIght-click on this table, and choose the option Make Combined Data Table. You then have a new table opened with the results from the Stepwise platform:

   
4.  To identify each model individually and keep track of their performances, we create a new model ID column on the new table. To do this, right-click on the first column of the table, and select New Formula Column > Row > Row. Each model will then have a unique ID with the new column "Row". Change the Modeling type to Nominal.
   
   
5. Select the column "Model", and click on the red triangle in the upper right position on the left of the datatable (or click in the menu Cols), choose Utilities > Text to Columns. Define the delimiter as "," and check the option Make Indicator Columns.
   
   
   
6. Click on menu Table > Stack. Select all the new indicator columns and click on Stack Columns. You can rename the Label column as "Terms" and the Data column as "Presence". Click OK.
   
   
   
   
7. You can then add some formula column (or using the Recode option) to identify which terms are main effects, interaction effect or quadratic effects to prepare the plot. With the example Bounce Data, I used a formula column when recoding:
   

   Map Value(
   	:Terms,
   	{"Silane", "Main effect", "Silane*Silane", "Quadratic effect", "Silica",
   	"Main effect", "Silica*Silane", "Interaction effect", "Silica*Silica",
   	"Quadratic effect", "Silica*Sulfur", "Interaction effect", "Sulfur",
   	"Main effect", "Sulfur*Silane", "Interaction effect", "Sulfur*Sulfur",
   	"Quadratic effect"},
   	Unmatched( :Terms )
   )

8. Add a Value Order column property to the Terms type column, so that the terms in the model appear with the main effects first, then interaction effects, then quadratic effects:
   
   
   
9. You can then use Graph Builder and create the Raster plot !
   
   1. Use "Row" (or Model ID column) as Y variable.
   2. Use "Terms" as X variable.
   3. Add "Terms type" as a nesting X variable (in the location below X).
   4. Change the graph style to "Heatmap".
   5. Use one of the model's performance metric (for example RMSE) as Color variable and choose an appropriate color theme (like green to red).
   6. Right-click on the Y variable, choose Order by, and select your performance metric (ascending for AICc, BIC and RMSE, descending for Rsquare).
   7. Add a Local Data Filter on the column "Presence" and select only value different from 0 (1: term is present in the model).
   8. You can also add a filter on the number of models displayed by filtering/reducing the range of values for the performance metric. 
   9. Once everything is set, click on Done. The Raster plot is ready !
      
      

As you can see, the creation of the Raster plot involves a lot of different operations. I'm planning on creating an add-in to automatize the process, to reduce the manual (and painful !) work needed to get this great visualization. If anyone is willing to help and contribute to this add-in, please join !