Detection Limit in Definitive Screening Design

[#1]

I'm running a DSD DOE in JMP18 Pro with 2 categorical factors (Factor A (level A1, A2), Factor B (level B1, B2) and 2 continuous factors (Factor C, Factor D). My goal is to select the better Level in each Factor A, and B (e.g., either A1 or A2, B1 or B2) including any two-way interaction terms for all factors and estimate the quadratic terms in Factor C, D.

I chose to add 8 more runs with blocks to add more power to my model and resulted in total of 22 runs. 

After running the experiment, about 1/3 of my runs were below detection limit. I tried to use the "Fit model" --> "Generalized Regression/LogNormal" with censored data to analyze my data. 

My questions are:

1/ What should I put in Construct Model Effects? Should I only include the Main Effects in the model or should I have all main effects + two-way interaction + quadratic terms (for Factor C, D)? 

2/ In Generalized Regression, what is the best estimation method? By default, the Estimation Method was Two Stage Forward Selection. Should I try other methods and use AICc or BIC to evaluate which model works best? (I tried, and Ridge/Holdback gave me the lowest AICc, should i use it?)

3/ When working with DSD, I usually use the Fit Definitive Screening Platform as my go-to, which has 2 stages of evaluating terms in the analysis. Can I do this with Detection Limit or Left-Censored Data? 

Appreciate for all your help!

[#2]

There is not enough information provided and without any SME, there is little basis for critique.  I do have the following general comments, IMHO:

1. DSD is a screening design methodology.  You only have 4 factors in your experiment.  This is NOT the intended use for DSD.  Although you can run DSD with categorical factors, that is not the strength of DSD which is intended to provide quadratic information inside the design space.

2. Whenever you are running an experiment, the design you use is a function of your hypothesized model.  That is the model is predicted from your SME and tested via the experiment.  I always start with a saturated model and remove insignificant/uninteresting terms from the model (AKA subtractive method of model building).  You should include all estimable terms in your initial model.

3. Which statistics you use to evaluate your model is a personal choice.  I use several, NOT ONE, to evaluate the model as I remove terms (e.g., R-Square-R-Square Adjusted delta, R-Square adjusted magnitude, RMSE, residuals plots, p-values).  Always with the assistance of SME.

4. I am always careful to diagnose the data before quantitative analysis.  This is why I ALWAYS use the sequence: Practical significance, graphical analysis and lastly quantitative analysis (PGQ).  If the data makes no sense or does not change of any practical significance, you can't fix it with quantitative analysis.

5. Blocking is not a strategy to increase the power of the design (while that may happen, that is not why it was invented, see Fisher).  Blocking is used to increase the inference space while not decreasing the precision of the experiment.

[#3]

Thank you very much for your input. I'm pretty new to DOE and I haven't too many types of design yet so I thought DSD would be a good option for my purposes. I actually tried to design with Custom Design: with same factors, and included all main effects, two-way interactions and quadratic terms, + blocking = 20 runs, which is pretty similar to DSD. At that point, I was debating which one to use, and I chose the DSD with a little bit higher runs thinking that would give me better power. 

The reason I used blocking was because we can't run the entire experiment in one go (maximum about 12 runs each time), so I decided to break it into 2 blocks (and conveniently this is an option in DSD) when i designed the model

[#4]

I, of course, am a huge supporter of experimental design.  I applaud your efforts to run one.  I do suggest you read some books and take some classes to get a better understanding of the different experimental situations and strategies. Pay particular attention to how you handle noise and restrictions in an experiment.

[#5 · Accepted solution]

Hi @D_P2025,

@statman already gave a lot of useful information about the use of Definitive Screening Design and model building regarding your topic.
I will try to provide additional comments and answer some of the other questions you had :

• When creating designs, it's best to build several designs with different methodologies (for example in your case, build a classical screening/factorial design, a D-Optimal and other relevant screening design strategies) and different runs size, and compare the performances and characteristics of your design using the Compare Designs platform. This will give you more information about the relative performances of each design, and their limitations regarding the aliasing structure, power to detect active effects, and relative prediction variance profile).
• When using the Custom Designs platform with the factors type you have, you can automate the creation of several designs in terms of runs size, optimality criterion, etc... by using the Design Explorer :
  

  There are a lot of options you can try to build several designs according to a model, and compare the relative performances of the design easily thanks to an enhanced table with all informations :

  

• When building models, as mentioned by statman, you should start with all effects possible from your assumed model (saturated model), and use a subtractive approach based on specific criteria (R²/R²adjusted, RMSE, terms p-values, AICc/BIC, ...) related to your objective. In your case, to make the detection of interactions and quadratic effects possible, you need to add these terms in the model effects. Note that all terms won't be estimable with a DSD, so only the most active/important one will be detected based on principles of effects sparsity, hierarchy and heredity. 
• There is no "best" estimation methods in Generalized Regression, as it will be dependant on your objective, design characteristics, and data available. I would recommend trying several estimation methods, and compare their performances as well as the terms included in the models estimated by each method, to see how well these methods agree :
  Example with R²/R²adjusted : 
  

  Example of terms agreement between each estimation method : 

  

  (Visuals from my presentation in the french User group : Découverte des plans OML (Orthogonal Main Effects Screening Designs for Mixed Level Factors) pour le...

  
  I would recommend trying other Estimation Method Options like Pruned Forward Selection and Best Subset Selection (only for low number of factors and acceptable number of terms in the model !) with the "Enforce Effect Heredity" option checked from Advanced Controls. Another great option for screening design is to use The Fit Two Level Screening Platform that can help build relevant models if the assumptions behind the use of this platform are met : Overview of the Fit Two Level Screening Platform
  Which criterion/criteria you use to evaluate and compare models depends on your objective. 
• Ridge is a penalized regression method, and has the ability to minimize coefficients for correlated features (close to 0 but not 0). This property enables Ridge regression to be used on datasets with many correlated features, as the negative impact of correlated features is minimized, and enables to reduce overfitting. As a penalization is introduced, you'll also get biased coefficient estimates. Ridge enables to consider all features, and tends to be most effective when there are a large number of variables with large and comparable effects. 
  I don't think Ridge regression is relevant in most DoE contexts, as you tend to have low aliases between factors and a large difference in effect size between terms.
• The Fit Definitive Screening platform is not able to handle censored data or responses with detection limits. Distribution and Generalized Regression platforms are able to handle this type of data : Detection Limits 
  So if working with this type of data, it may be wiser to use Generalized Regression platform to benefit from the correct handling of the data. If you want to learn more about the possible impact of not using the Detection Limits property in the Generalized regression platform, you can watch the presentation Limits of Detection (LoD) - New in JMP Pro 16.  

Hope this additional answer may help you,

[#6]

Thank you very much for your reply.

• I ran Compare Design, but it only compared the main effect terms, i guess maybe it was because of the DSD structure?

• I typically use Custom Design with D-Optimal for Screening design, and I-Optimal for Optimization design. But i haven't had any situations which i would prefer A-Optimal. Could you explain more on this?
• I thought DSD can be used to evaluate the two-way interaction and estimate the quadratic terms as long as the active terms are not too many. that's why I added couple more runs to the design to increase the power of the model can detect most of the terms.
• The link you shared is very helpful, thank you for sharing.
• Thank you for explaining in simple terms. I dont have data science background so this is very helpful.
• I read online there are other algorithms to evaluate Censored Data, i.e., Beta-substitution when handling left-censored data. Do you know if JMP has this? As my sample size is not large, i'm afraid ML method might not be the best fit. 

[#7]

Hi @D_P2025,

Concerning your additional questions :

• By default, when evaluating or comparing a Definitive Screening Design, the model effect will be only comprised of main effects. You can add some interaction or quadratic effects terms, in the limit of your degrees of freedom available (related to the number of independent runs in your design). A DSD is not able to fit a full response surface model (with all main effects, quadratic effects and interaction terms), other designs such as Central Composite Designs or Box-Behnken are more appropriate to fit this complete model. As DSD is typically used in a screening stage, it relies on effects sparsity, hierarchy and heredity principles to identify the few important active terms from the many negligible, and thanks to it's good projection properties, may be able to fit a full response surface model if the active terms are not too many.
• A-optimality criterion helps minimize the average variance of the parameter estimates of your assumed model. This criterion helps minimize correlations between your terms (that would otherwise inflate parameters estimates variance), which also helps model selection, as it becomes much easier to determine which effects are statistically significant. Moreover, you can also adjust weights for estimating more precisely certain effects (for example putting more emphasis on main effects vs. interaction effects) with A-Optimal designs using the option A- Optimality Parameter Weights.
  Bradley Jones gave an excellent talk on this topic : 21st Century Screening Designs (2020-US-45MP-538)
  Jonathan Stallrich also compared in 2022 the properties of D- and A-Optimal designs regarding the minimization of estimation variance and bias, and found that A-Optimality criterion offer less candidate designs with similar or better properties than D-Optimality. The paper is available here : https://arxiv.org/abs/2210.13943 

For these reasons, A-Optimality is the recommended optimality criterion used by JMP when using two-level designs for main effects and two-factor interactions.

• Yes adding more runs increase power and allow second order effects to be more easily detected. It also helps model selection.
• Sorry, I'm not familiar with methods involving censored data. Other platforms in JMP may help you, like Nonlinear platform.

Concerning the analysis of DSDs, there are other discussions dealing about situations where different estimation methods and model selection have been used :

Setting Stage 1 P value in Analyis of DSD at a high level to dedect active effects

DSD Interpretation and Next Steps 

Significance of factors in Definitive Screening Design 

Hope this complementary answer will help you,

[#8]

For the modeling, I recommend starting with the full response surface model, and then use Two-Stage Forward Regression in Generalized Regression.  You should still should be able to use the censoring of the data, but a simple approach is to use substitute the reporting or detection threshold for the results.  

[#9]

I have no experience with censoring but with regular, good quality data, 22 DSD runs with 4 factors should be enough to give you at least a direction to understanding and correct modeling. I am curious to know the result when, in the DSD fit model platform, you increase the treshold P-value from 0,1 to 0,4. Did you also try SVEM?