Design of Experiments Club

Video 1: Introduction to JMP 19    Video 2:  Including covariates and/or uncontrolled supplementary variables in DOE set-up and analysis.   Video 3: Calculating power of your sample size in DOE, particularly for in vivo (i.e., animal studies) where replicates are common and necessary.   Video 4: How to choose the appropriate design for a DoE experiment?   Q3 DOE Club Q&A compiled How should covariates and uncontrolled supplementary variables be handled in the setup and analysis of a DOE? Designed Experiments often face variables outside the planned factors called covariates. Some are measurable but not controlled (covariates, e.g., baseline weight, pH), while others are uncontrolled noise (e.g., humidity, operator effects). Ignoring these can inflate error and obscure true factor effects. DOE Approach: Covariates: If measurable, add them into the model. This reduces unexplained variation, sharpens estimates, and increases power. Uncontrolled supplementary variables: If not measurable, manage through blocking (grouping similar conditions), randomization (spreading nuisance effects evenly), and replication (estimating variability). Solution: Identify likely covariates during the Define stage, measure them consistently, and include them in the model. For unmeasurable noise factors, rely on blocking, randomization, and replication. This ensures cleaner signal detection and reliable conclusions, especially in complex biological or manufacturing DOE. Reference: https://community.jmp.com/t5/JMP-Blog/What-is-a-covariate-in-design-of-experiments/ba-p/361517 https://community.jmp.com/t5/Discussions/DOE-with-covariate/td-p/825007 https://community.jmp.com/t5/Discussions/Unexpected-change-in-controlled-variable-in-DOE/td-p/729169 https://community.jmp.com/t5/Discussions/Mixture-DOE-with-covariate-batch-effects/td-p/718593 https://community.jmp.com/t5/Learn-JMP-Events/Developer-Tutorial-Handling-Covariates-Effectively-when/ev-p/809996 https://community.jmp.com/t5/JMPer-Cable/New-in-JMP-16-Improved-covariate-handling-in-DOE/ba-p/361540 https://community.jmp.com/t5/Discussions/How-to-Account-for-Uncontrolled-Factors-Using-a-Definitive/td-p/666426         How can we calculate the power of a DOE sample size, particularly in in vivo (animal) studies where replicates are common and necessary? In in vivo studies, sample size is limited by cost, ethics, and feasibility. Too few animals lead to low power (risk of missing real effects), while too many waste resources and may raise ethical concerns. DOE Approach: Power is the probability of detecting a true effect given sample size, effect size, variability, and significance level. Replicates are especially important in animal studies because biological variability is high. JMP’s Power and Sample Size platform lets you simulate different scenarios before running the study. Solution: Estimate expected variability from pilot data or literature. Use power analysis to determine the minimum number of replicates needed to achieve acceptable power (commonly 80%). If resource constraints limit sample size, design the DOE to focus on the most critical factors, and consider blocking or covariates to reduce noise. This balances statistical rigor with ethical and budget realities. Reference: https://community.jmp.com/t5/Discussions/How-to-use-sample-size-and-power-for-factorial-DoE-to-determine/td-p/380872 https://community.jmp.com/t5/Discussions/Power-calculations-to-determine-sample-and-replication-of/td-p/552299 https://community.jmp.com/t5/Discussions/Sample-Size-for-DOE/td-p/237085 https://community.jmp.com/t5/Discussions/Power-Equation/td-p/53219 https://community.jmp.com/t5/Discussions/Should-I-consider-power-analysis-in-DOE/td-p/501063 https://community.jmp.com/t5/Discussions/DoE-How-to-treat-replicate-measurements/td-p/18588 https://community.jmp.com/t5/Discussions/Power-Analysis-in-Design-Evaluation-DOE/td-p/65472 https://community.jmp.com/t5/Discussions/Doe-and-replications/td-p/565281 https://community.jmp.com/t5/Learning-Center/Sample-Size-Explorers-DOE/ta-p/414885 https://community.jmp.com/t5/Discussions/quot-Animals-quot-Split-Plot-DOE/td-p/546140   When drafting a DOE exploration path, how do we determine the number of experiments required to achieve specific goals (such as coverage of design space or reduction of uncertainty), and how can this be balanced against resource constraints? When planning a DOE, there is always tension between the ideal number of runs (to cover the design space and reduce uncertainty) and the maximum feasible runs due to time, cost, or ethical limits. Without careful planning, you risk either underpowered results or wasted resources. DOE Approach: Use design diagnostics (like prediction variance profiles, design space coverage, and aliasing structures) to quantify how well a candidate design explores the factor space. Apply power analysis to check if the chosen sample size can detect effects of practical importance. Explore sequential/iterative designs (like Bayes Opt) starting smaller and expanding only if needed, instead of committing all resources upfront. Solution: Define the experimental goals clearly (effect detection, response surface mapping, or robustness testing). Then use DOE tools to evaluate candidate designs against those goals under the given resource ceiling. If the maximum allowable experiments do not provide sufficient coverage or power, either narrow the study scope to critical factors and responses, or adopt a staged design strategy. This ensures that even with constraints, the DOE produces meaningful, defensible insights.   https://community.jmp.com/t5/Discussions/how-JMP-design-the-number-of-runs-for-Default-under-DOE-custom/td-p/662940 https://community.jmp.com/t5/Discussions/JMP-Custom-DOE-vs-Full-Factorial-DOE/td-p/256413 https://www.jmp.com/content/dam/jmp/documents/en/support/jmp111/DOE_Guide.pdf https://community.jmp.com/t5/Learn-JMP-Events/Understanding-DOE-Capabilities-in-JMP/ev-p/809981         What is the best way to choose the appropriate experimental design for a DOE study? Selecting an inappropriate design risk wasting resources and failing to answer the study’s questions. DOE Approach: Match design to objective: Screening : Fractional Factorial, Definitive Screening Design (DSD). Optimization/curvature modeling :  Response Surface Designs (Central Composite, Box–Behnken). Mixture/formulations:  Simplex lattice, Simplex centroid, or Mixture Space-Filling designs. Robustness/noise: Taguchi or robust designs. Custom Design in JMP: When standard catalog designs don’t fit (due to unusual factor types, constraints, or resource limits), JMP’s Custom Design platform builds an experiment tailored to your factors, model, and number of runs. Always check diagnostics (alias structure, prediction variance, design efficiency) to validate the choice. Solution: Define the experimental goal clearly (screening, optimization, robustness, mixtures). Use classic designs where appropriate, but rely on JMP’s Custom Design to handle complex, real-world scenarios. This ensures flexibility, efficient use of resources, and alignment with the actual scientific question. https://www.synthace.com/blog/types-of-doe-design-a-users-guide https://ilssi.org/how-to-choose-the-right-design-for-a-designed-experiment-doe/ https://community.jmp.com/t5/Abstracts/Design-of-Experiment-s-Crucial-Step-0-Choosing-the-Right-DOE/ev-p/738877 https://www.jmp.com/content/dam/jmp/documents/en/support/jmp111/DOE_Guide.pdf https://www.itl.nist.gov/div898/handbook/pri/section3/pri33.htm         JMP 19 DOE resources Bayesian Optimization in JMP 19: JMP Bayesian Optimization platform overview: https://www.jmp.com/en/software/bayesian-optimization Community introduction to Bayesian Optimization in JMP Pro 19: https://community.jmp.com/t5/JMPer-Cable/Introducing-the-Bayesian-Optimization-platform-in-JMP-Pro/ba-p/897807 JMP 19 New Release overview including Bayesian Optimization: https://www.jmp.com/en/software/new-release/new-in-jmp JMP Pro 19 Bayesian Optimization YouTube demo: https://www.youtube.com/watch?v=9rOqTZHpRqA DOE whitepapers in JMP (general and JMP 19): JMP DOE advantage whitepaper: https://www.jmp.com/en/resources/white-papers/doe-advantage JMP official whitepapers page (DOE papers included): https://www.jmp.com/en/resources/resource-listings/by-type/white-papers Detailed JMP DOE whitepaper PDF: https://www.jmp.com/content/dam/jmp/documents/en/white-papers/jmp-whitepaper-doe-advantage.pdf Additional DOE whitepaper PDF: https://www.jmp.com/content/dam/jmp/documents/en/white-papers/103044-doe.pdf DOE-related information and new features in JMP 19: JMP 19 Design of Experiments Guide PDF: https://www.jmp.com/content/dam/jmp/documents/en/support/jmp19/design-of-experiments-guide.pdf JMP DOE capabilities overview page: https://www.jmp.com/en/software/capabilities/design-of-experiments What's new in JMP DOE (JMP 19) with new explorers and enhanced tools: https://community.jmp.com/t5/JMPer-Cable/What-s-new-in-JMP-DOE/ba-p/898558       ... View more

Next Sessions are:  23 Sep | Q3: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/23-sep 25 Nov | Q4: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/25-nov   Question 1: How do I handle situations where my analysis shows no significant effects or interactions? Chandra: Use Graphical Exploration Profiler, Contour Plots, and Interaction Plots in JMP can help spot trends that aren’t statistically significant yet may warrant further study. Victor: Depends on the objective, prior knowledge and assumed model/complexity of the system under study + budget constraints Definitive Screening Designs and OML designs (Mixed-Level Screening Designs) help investigating curvature (quadratic) effects early in the screening stage Simon: if I already have existing DSD data and I want to augment them to a full RSM, how do I deal with the lack of randomization since I add on additional runs which I didnt measure together with the first data set? Victor: Use blocking when augmenting the design Phil: FitDSD will not work with missing runs unfortunately Simon: can you show the augment using blocking in the software? Victor: I added this default check in the Wish list: https://community.jmp.com/t5/JMP-Wish-List/DoE-Augmentation-Option-quot-Group-new-runs-into-separate-block/idi-p/665019   Question 2: How do I choose between different DOE types (e.g., full factorial, fractional factorial, response surface)?   Question 3: What to do when you lose a run of DOE   Victor: You could also run the model (if possible) without this run, and see if the model is able to approximate the response from the last added run?  And if you must expand ranges of your factors because the ranges are too narrow, you can augment your design instead of starting from scratch again to benefit from the information already gathered. David: The challenge is that I am already on the limit of runs and I can not go with one run less. So, I need to "augment" the design somehow. Any takes on whether you must use blocking?  Kevin: A common pitfall I see, and fall into myself, is to try to limit the number of factors at the start, to make things manageable (i.e. reduce the number of runs). So then on gutfeel eliminate these, instead of screening them out based on DoE.  Of course there is a limit, not to screen every thought that comes to mind, but it is often helpful to get a few experts in the meeting and brainstorm a starting set of factors. Victor: You have a design with saturated model? You can still maybe use some tools adapted for saturated designs, like the Fit Two-Levels Screening platform (The Fit Two Level Screening Platform) to try creating a relevant model.  And/Or as you mentioned, you could augment your incomplete first design and use Blocking for the new runs to prevent/detect any shift/variation between the first batch of experiments and the second one. CK: can you also run one that was run previously to help account? Simon: I don’t know if we already touched the topic, but I feel limited at final data analysis and model judgment. I learned at the JMP DoE course to analyse the ANOVA and lack of fit. What other and maybe better option do I have to judge my final model and maybe use the statistics of this model to plan a better experiment? Victor: There are many model metrics to use for model comparison/validation depending on your objective: R²/R² adjusted, p-values, RMSE, Information criteria, ... More discussion : https://community.jmp.com/t5/Discussions/Model-reduction/m-p/823421/highlight/true#M100286   Kevin: It is a tough question indeed David. Once idea to test/try, without knowing more about your case, but try to use a tool like Partial Least Squares that can handle missing data. Put your planned experiment in the X block, and then the Y predictors are what you measured, and leave the value you have missing in the Y column (or matrix if you have multiple responses).   PLS can make a prediction for you, though it might not be the best, it will be better than dropping out that run completely.   Also, what would have been your next step after this analysis, if you had the data point that is missing? Phil: I'd just repeat the missing run like Kim says. It's not ideal because "noise" factors could have changed but I don't think there is a better option. Putting it in a separate block makes no sense. A block with 1 run is not useful.  CK: centerpoints are good but could you repeat a previously successful run? Kevin: could you try simulating your problem? and in the simulation see the effect of dropping a run, and bringing it back in a later block, simulated with, or without a block effect. Create a fake system, where you know the truth, simulate experimental data that replicates your scenario. David: And then? how would you proceed? Kevin: Then do the analysis on the simulated data and see how important this missed run is to the insights you would (or would not) have gotten.   Since you know the truth in the simulated system, you should see that back in your analysis of the simulated data.  David: by "the truth" in the system, do you mean which effects are present? Kevin: Yeah, if that is the intention of your experiments is to discover which effects are at play in the range you are operating in. David: I would have fitted the data to get the model with the lowest AICc to describe the system Kevin: So, it sounds like you are in an early phase: screening out factors, and learning about the system. David: Exactly! That's why I feel that every run matters Kevin: In that case; the question to check in the simulation: would the potential screening or knowledge learned (e.g. factor A make response 1 decrease), would that knowledge learned been wrong without that single missing run, or would you have gotten the same info? In other words, it is not an issue having that missing run.   Question 4: Making use of the Box Cox Transformation option within a DOE analysis Victor: to know if a transformation is needed, you can check the residuals from your model first Diana: Box Cox transformation: I recommend rounding to the nearest big number (still inside the confidence interval of good transforms) because it will be much easier to later explain to the audience. Much easer to say "it is a log transformation" than "it is to the power of -0.132342356". Phil: Re box-cox transformation: JMP suggests a lambda number which is the power to which the response is raised as the transformation. 0 is equivalent to a log transform. 2 would be squaring the response. -1 is taking the inverse of the response. A log transform makes sense in some cases, e.g. where the response is a count of cell growth and therefore varies over orders of magnitude. I think it is hard to justify other transforms so I would avoid. The only exceptions are transforms that help when the response is on percent scale or on a 0 to 1 scale. Logit and logit pct transforms can be useful.   Question 5: What is the purpose of centre points, and determining appropriate factor ranges Bill: DOEs are notoriously sparse and methods like tree methods or SVEM are good options for analysing your DOE data. Victor: Topic of centre points also mentioned in previous DoE Club : https://community.jmp.com/t5/Design-of-Experiments-Club/Recordings-DOE-Club-Q1-2025/m-p/842727   Question 6: Data that doesn’t follow any DOE Principles. Victor: You can use the Augment design platform.  This helps build trust with the people having already done experiments with OFAT Chandra: To capture interactions and curvature after an OFAT study, use JMP’s Augment Design feature by specifying the same factors and a fuller model (main effects, interactions, quadratics). This generates additional runs that, when combined with your OFAT data, complete the design. Also adding a blocking factor to separate the original and new runs for better modelling.    Comment from Caleb King: I wanted to share something that is somewhat related to some things that were mentioned (Kim mentioned rounding factor levels to a nicer level and there's of course the discussion around using centerpoints to help with estimating pure error). I have a colleague at NASA who's run into situations where testers have a hard time achieving the specified design settings but can record the actual settings with high precision. There are methods available to use this situation to estimate pure error outside of using centre points. I've attached a presentation I gave 3 years ago on this topic (we had tried coming up with our own; turns out we were a few decades too late) that contains a summary and references for research in this area. Might be useful to you all.  (see attachement) ... View more

Question 1: DOE model analysis     Comments: @Phil_Kay  @Victor_G : Stepwise selection might not be the default approach to recommend.  It is helpful as a guide and comparison, but I would prefer to compare the outcomes with GenReg and other platforms  Raster plots can also help see the frequencies of terms inclusion on many models, to see how they compare : https://community.jmp.com/t5/JMP-Wish-List/Raster-plots-or-other-visualization-tools-to-help-model/idi-p/730968   @bdclark23 : Multiple model approaches with a preponderance of evidence that consistently show the same effects across different models, each with their own biases and assumptions.   Question 2: “Covariate Approach” of applying constraints   Comments @Victor_G : About constraints : https://community.jmp.com/t5/JMP-Blog/Demystifying-Factor-Constraints/ba-p/619898#U619898 @Ryan_Lekivetz : https://community.jmp.com/t5/Abstracts/Candidate-Set-Designs-Tailoring-DOE-Constraints-to-the-Problem/ev-p/756629 @Phil_Kay : I posted about this on LinkedIn recently as well: https://www.linkedin.com/posts/drphilkay_candidate-set-approach-for-complex-experiments-activity-7290301797410168833-JxRT?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAGJ9cABNhzDhYBtevawkkMdSNm3keFRm68 Love this discussion. It's relatively easy to jump (JMP!) to the technical answer. But important to challenge the "why" as well.  @Chandramouli_R :    Video 3: Full factorial design: How many center points you recommend?   @Phil_Kay : I've posted about centre points before as well: https://www.linkedin.com/posts/drphilkay_doebyphilkay-doebyphilkay-designofexperiments-activity-7056885385095729152-pu9F?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAGJ9cABNhzDhYBtevawkkMdSNm3keFRm68 @Victor_G : Centre points can help detecting curvature, but doesn't allow estimation of several quadratic effects Q: How to add center points in a custom design? A:    @Victor_G : About scoping designs and their use as preliminary runs to check design space : Scoping Designs | Prism   Video 4: How can I reset / delete factors in the easy DoEs without changing the experiment design?   Comments:  @Chandramouli_R : https://community.jmp.com/t5/Discussions/Hard-to-change-factors-center-points-and-replicates/td-p/224712  @Victor_G :  In "analysieren" you can select the effects in and out of the model Tab "analysieren" --> Try "Bestes Model"  Or create a custom model with "Fit Model" using the JMP datatable with your results @Jonas_Rinne : I think you can also do this by going into the flexible mode in Easy DOE @Victor_G : Fit your responses separately.  But I think there are many ressources to help you create and compare models. OFAT has no model behind, it's only a comparative approach, so not very helpful to guide decision-making   Video 5: Analytic Culture for Using DOE   Comments:  @Uwe_Weber : Use the Profiler and they will understand @anchildress : We have the most success at Eastman when we focus on relationship building with customers/clients, starting small to build trust with you and your techniques. Once they see some wins or successful DOEs with colleagues perhaps they are more trusting of DOE.   Results and cost savings are the most convincing technique @DB1 : Changing a culture: Patience. Work around the edges. When you can convince one scientist or engineer, they can be more convincing of DOE use to their teammates than us. As Bea says, one really good DOE dataset in a company can convince a lot of people if used diplomatically and audience-aimed.     ... View more

Video 1: Various ways to create models: model building, comparison and selection     Comments: @Victor_G - Some visualization and propositions for model comparison and selection (raster plots) proposed by Goos : https://community.jmp.com/t5/JMP-Wish-List/Raster-plots-or-other-visualization-tools-to-help-model/idi-p/730968   @martindemel - bootstrapping is what I thought of when saying tree based models, sorry fo rbeeing unprecise SVEM - Self Validating Ensemble Models: Discovery Summit Talk - SVEM: A Paradigm Shift in Design and Analysis of Experiments (2021-EU-45MP-779) - JMP User Community Support page - https://www.jmp.com/support/help/en/18.1/index.shtml#page/jmp/overview-of-selfvalidated-ensemble-models.shtml   @Jonas_Rinne - SVEM might be a method which uses some kind of validation without violating the design structure   @Victor_G - Or use statistical analysis, or Machine Learning algorithms robust to overfitting (random forests, SVM, ...)  Replicates are more used for explainability (sort the random error/variation/statistical error from the factors influences), I wouldn't use them for predictive modeling, particularly for Machine Learning modelling   Video 2: Mixture design with constraints    Comments:  @Bill_Worley - Mixture models can be highly non-linear and it may be good to start off with a space-filling mixture doe to try and better understand the non-linearity. @Victor_G Or do it sequentially : classical mixture design up to 2nd/3rd order, and then augment with Space-Filling to have points inside the experimental space.   Video 3: Number of trials : can we use the minimal advised by JMP in Optimal Design?    Comments:  @Bill_Worley - Try Augment Designs @Paul_J - This makes me wonder ... How does JMP decide the recommended number of runs? ... Could that give us an idea of why to use something more than the minimum? @Victor_G - Very often, 4 extra runs than minimum number of runs  Design Generation @Jonas_Rinne -  It also depends on your choices. If you have chosen some terms as "if possible" it will try to build these effects in with the added runs coming from the minimum. Otherwise it will add replicates for instance. JMP is trying to distribute the additional runs as best as possible based on your choices.  @martindemel - consider that you will get also information about the error estimate @Victor_G -  Not only correct model, but you can evaluate "natural" variability (random noise)  Without extra runs, you fit a model with the assumption that you have no experimental or measurmeent noise ! Completely agree with Brian, it all comes with a price / compromise   Video 4: Resources   Next DOE Club Session: https://www.jmp.com/en_gb/events/users-groups/users-group-meetings/doe-community-of-practice-multi-registration.html Time to Innovate: https://www.jmp.com/en_ca/events/live-webinars/time-to-innovate/time-to-innovate-28nov2024-doe.html DOE Club in the Community: https://community.jmp.com/t5/Design-of-Experiments-Club/gh-p/doe-grouphub Lifecycle DOE - https://www.mdpi.com/2306-5354/11/11/1089 ... View more