Design of Experiments Club

Introduction to JMP 19 new features with focus on Bayesian Optimization Further information: https://www.jmp.com/en/software/new-release/new-in-jmp Question 1: If we don't have historical data set, can we start a Bayesian Optimization? Victor: Yes, you can start with Space-Filling runs to cover your experimental space  Jonas : It needs at least one row in a data set as well as some initial column information. It will add more runs to explore more of the design space in case you start with only one or a few rows. Yes, basically the platform needs some metadata and column informations to be able to run (design role and coding at least)  Victor: I would start with a simple basic design first Maria: Developer Tutorial on Bayesian Optimisation: https://community.jmp.com/t5/Learn-JMP-Events/Developer-Tutorial-Bayesian-Optimization/ev-p/894910?… Victor: You can try Scoping Designs | Prism to check the variability of your response and its variability regarding the factors Winfried: It is i the name: this is particularly useful, if you believe you can start soon to optimise. If you are still early in the process of understanding your problem, use other design tools, like screening factors or space filling designs! Victor: Scoping Design DoE JMP Add-In - JMP User Community Winfried: Be aware of time trend with any sequential method like BO! Remember, why we are generally use randomisation!  Victor: Also important to emphasize that DoE is not a static, one-design-answers-all-questions methodology; like Bayesian Optimization, you can progress by iterations, by augmenting your design step by step : start with a screening design, augment with higher order effects, then augment with space-filling ... The possibilities are immense, and you can combine these steps with Bayesian Optimization  Winfried: Be aware of time trend with any sequential method like BO! Remember, why we are generally use randomisation! ;) Yes, no protection in BO like Blocking or other time-trend robust designs that we can find with DoEs !  Victor: overview of optimization methods : The Evolution of Optimization Methods | by Victor Guiller | Medium Question 2: Is there a tool like prediction profiler which you could use for design spaces with many factors? Victor: What is the practical difficulty behind using Prediction Profiler for 6+ factors? Optimize? You might reduce your factors space based on the importance of factors on your response(s) Anthony: could reduce based on effect size as well. the practical significance question. if some are very low compared to the others you could reduce on those at first. Victor: PCA might not work well using independent factors Anthony: PCA I’ve used for reducing dimensionality in Y's. i haven't used it for multiple x's. Victor: You can also use the Simulator, create a new dataset with predictions, and create your own visualizations.   Question 3: Including different start material product quality in the DOE designs. You can use random blocks to allocate different batch/product qualities across your DOE runs? Jonas: Covariate factors maybe? Anthony: treat as a factor, covariate, blocking all come to mind as options Manlio: you could use blocking for different materials Victor: It depends if you want to assess the impact of batch on mean response vs. variance response (fixed vs. random effects in the model), how representative these batch qualities are, and your objectives    Question 4:  How does Bayesian Optimisation handle iteration of factor addition and removal as the learning journey goes on? Design and noise factors change from experiment to experiment in my current workflow. I'm curious how BayesOpt would approach this.   Further resources:  Developer Tutorial: Bayesian Optimization: https://community.jmp.com/t5/Learn-JMP-Events/Developer-Tutorial-Bayesian-Optimization/ec-p/894910#M833 Topic suggestions for 2026: https://community.jmp.com/t5/Design-of-Experiments-Topic/idb-p/doe-grouphubidea-board Subscribe to get the new sessions in 2026: www.jmp.com/newswire ... View more

Hello Everyone, I am a masters student currently finishing my thesis. The objective of the thesis is to increase vertical axis wind turbine efficiency by adding winglets and optimizing the winglets design parameters to maximize efficiency (using DOE). The design parameters of these winglets consist of 7 continuous parameters, and 1 categorial parameter (2-level). I am lost on how to proceed after finishing the 22 runs of DSD (I still have few runs to finish but I am planning ahead because I am running out of time). I've read that augmenting the design is a good approach to fit an RSM, but can someone help me on that? Especially that I have a categorial factor I am not sure how to fit a RSM with this factor (should there be 2 equations each for a level of this categorial factor, or combined in 1 equation?) Thank you so much for your help! ... View more

Video 1: Introduction to JMP 19    Video 2:  Including covariates and/or uncontrolled supplementary variables in DOE set-up and analysis.   Video 3: Calculating power of your sample size in DOE, particularly for in vivo (i.e., animal studies) where replicates are common and necessary.   Video 4: How to choose the appropriate design for a DoE experiment?   Q3 DOE Club Q&A compiled How should covariates and uncontrolled supplementary variables be handled in the setup and analysis of a DOE? Designed Experiments often face variables outside the planned factors called covariates. Some are measurable but not controlled (covariates, e.g., baseline weight, pH), while others are uncontrolled noise (e.g., humidity, operator effects). Ignoring these can inflate error and obscure true factor effects. DOE Approach: Covariates: If measurable, add them into the model. This reduces unexplained variation, sharpens estimates, and increases power. Uncontrolled supplementary variables: If not measurable, manage through blocking (grouping similar conditions), randomization (spreading nuisance effects evenly), and replication (estimating variability). Solution: Identify likely covariates during the Define stage, measure them consistently, and include them in the model. For unmeasurable noise factors, rely on blocking, randomization, and replication. This ensures cleaner signal detection and reliable conclusions, especially in complex biological or manufacturing DOE. Reference: https://community.jmp.com/t5/JMP-Blog/What-is-a-covariate-in-design-of-experiments/ba-p/361517 https://community.jmp.com/t5/Discussions/DOE-with-covariate/td-p/825007 https://community.jmp.com/t5/Discussions/Unexpected-change-in-controlled-variable-in-DOE/td-p/729169 https://community.jmp.com/t5/Discussions/Mixture-DOE-with-covariate-batch-effects/td-p/718593 https://community.jmp.com/t5/Learn-JMP-Events/Developer-Tutorial-Handling-Covariates-Effectively-when/ev-p/809996 https://community.jmp.com/t5/JMPer-Cable/New-in-JMP-16-Improved-covariate-handling-in-DOE/ba-p/361540 https://community.jmp.com/t5/Discussions/How-to-Account-for-Uncontrolled-Factors-Using-a-Definitive/td-p/666426         How can we calculate the power of a DOE sample size, particularly in in vivo (animal) studies where replicates are common and necessary? In in vivo studies, sample size is limited by cost, ethics, and feasibility. Too few animals lead to low power (risk of missing real effects), while too many waste resources and may raise ethical concerns. DOE Approach: Power is the probability of detecting a true effect given sample size, effect size, variability, and significance level. Replicates are especially important in animal studies because biological variability is high. JMP’s Power and Sample Size platform lets you simulate different scenarios before running the study. Solution: Estimate expected variability from pilot data or literature. Use power analysis to determine the minimum number of replicates needed to achieve acceptable power (commonly 80%). If resource constraints limit sample size, design the DOE to focus on the most critical factors, and consider blocking or covariates to reduce noise. This balances statistical rigor with ethical and budget realities. Reference: https://community.jmp.com/t5/Discussions/How-to-use-sample-size-and-power-for-factorial-DoE-to-determine/td-p/380872 https://community.jmp.com/t5/Discussions/Power-calculations-to-determine-sample-and-replication-of/td-p/552299 https://community.jmp.com/t5/Discussions/Sample-Size-for-DOE/td-p/237085 https://community.jmp.com/t5/Discussions/Power-Equation/td-p/53219 https://community.jmp.com/t5/Discussions/Should-I-consider-power-analysis-in-DOE/td-p/501063 https://community.jmp.com/t5/Discussions/DoE-How-to-treat-replicate-measurements/td-p/18588 https://community.jmp.com/t5/Discussions/Power-Analysis-in-Design-Evaluation-DOE/td-p/65472 https://community.jmp.com/t5/Discussions/Doe-and-replications/td-p/565281 https://community.jmp.com/t5/Learning-Center/Sample-Size-Explorers-DOE/ta-p/414885 https://community.jmp.com/t5/Discussions/quot-Animals-quot-Split-Plot-DOE/td-p/546140   When drafting a DOE exploration path, how do we determine the number of experiments required to achieve specific goals (such as coverage of design space or reduction of uncertainty), and how can this be balanced against resource constraints? When planning a DOE, there is always tension between the ideal number of runs (to cover the design space and reduce uncertainty) and the maximum feasible runs due to time, cost, or ethical limits. Without careful planning, you risk either underpowered results or wasted resources. DOE Approach: Use design diagnostics (like prediction variance profiles, design space coverage, and aliasing structures) to quantify how well a candidate design explores the factor space. Apply power analysis to check if the chosen sample size can detect effects of practical importance. Explore sequential/iterative designs (like Bayes Opt) starting smaller and expanding only if needed, instead of committing all resources upfront. Solution: Define the experimental goals clearly (effect detection, response surface mapping, or robustness testing). Then use DOE tools to evaluate candidate designs against those goals under the given resource ceiling. If the maximum allowable experiments do not provide sufficient coverage or power, either narrow the study scope to critical factors and responses, or adopt a staged design strategy. This ensures that even with constraints, the DOE produces meaningful, defensible insights.   https://community.jmp.com/t5/Discussions/how-JMP-design-the-number-of-runs-for-Default-under-DOE-custom/td-p/662940 https://community.jmp.com/t5/Discussions/JMP-Custom-DOE-vs-Full-Factorial-DOE/td-p/256413 https://www.jmp.com/content/dam/jmp/documents/en/support/jmp111/DOE_Guide.pdf https://community.jmp.com/t5/Learn-JMP-Events/Understanding-DOE-Capabilities-in-JMP/ev-p/809981         What is the best way to choose the appropriate experimental design for a DOE study? Selecting an inappropriate design risk wasting resources and failing to answer the study’s questions. DOE Approach: Match design to objective: Screening : Fractional Factorial, Definitive Screening Design (DSD). Optimization/curvature modeling :  Response Surface Designs (Central Composite, Box–Behnken). Mixture/formulations:  Simplex lattice, Simplex centroid, or Mixture Space-Filling designs. Robustness/noise: Taguchi or robust designs. Custom Design in JMP: When standard catalog designs don’t fit (due to unusual factor types, constraints, or resource limits), JMP’s Custom Design platform builds an experiment tailored to your factors, model, and number of runs. Always check diagnostics (alias structure, prediction variance, design efficiency) to validate the choice. Solution: Define the experimental goal clearly (screening, optimization, robustness, mixtures). Use classic designs where appropriate, but rely on JMP’s Custom Design to handle complex, real-world scenarios. This ensures flexibility, efficient use of resources, and alignment with the actual scientific question. https://www.synthace.com/blog/types-of-doe-design-a-users-guide https://ilssi.org/how-to-choose-the-right-design-for-a-designed-experiment-doe/ https://community.jmp.com/t5/Abstracts/Design-of-Experiment-s-Crucial-Step-0-Choosing-the-Right-DOE/ev-p/738877 https://www.jmp.com/content/dam/jmp/documents/en/support/jmp111/DOE_Guide.pdf https://www.itl.nist.gov/div898/handbook/pri/section3/pri33.htm         JMP 19 DOE resources Bayesian Optimization in JMP 19: JMP Bayesian Optimization platform overview: https://www.jmp.com/en/software/bayesian-optimization Community introduction to Bayesian Optimization in JMP Pro 19: https://community.jmp.com/t5/JMPer-Cable/Introducing-the-Bayesian-Optimization-platform-in-JMP-Pro/ba-p/897807 JMP 19 New Release overview including Bayesian Optimization: https://www.jmp.com/en/software/new-release/new-in-jmp JMP Pro 19 Bayesian Optimization YouTube demo: https://www.youtube.com/watch?v=9rOqTZHpRqA DOE whitepapers in JMP (general and JMP 19): JMP DOE advantage whitepaper: https://www.jmp.com/en/resources/white-papers/doe-advantage JMP official whitepapers page (DOE papers included): https://www.jmp.com/en/resources/resource-listings/by-type/white-papers Detailed JMP DOE whitepaper PDF: https://www.jmp.com/content/dam/jmp/documents/en/white-papers/jmp-whitepaper-doe-advantage.pdf Additional DOE whitepaper PDF: https://www.jmp.com/content/dam/jmp/documents/en/white-papers/103044-doe.pdf DOE-related information and new features in JMP 19: JMP 19 Design of Experiments Guide PDF: https://www.jmp.com/content/dam/jmp/documents/en/support/jmp19/design-of-experiments-guide.pdf JMP DOE capabilities overview page: https://www.jmp.com/en/software/capabilities/design-of-experiments What's new in JMP DOE (JMP 19) with new explorers and enhanced tools: https://community.jmp.com/t5/JMPer-Cable/What-s-new-in-JMP-DOE/ba-p/898558       ... View more

Next Sessions are:  23 Sep | Q3: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/23-sep 25 Nov | Q4: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/25-nov   Question 1: How do I handle situations where my analysis shows no significant effects or interactions? Chandra: Use Graphical Exploration Profiler, Contour Plots, and Interaction Plots in JMP can help spot trends that aren’t statistically significant yet may warrant further study. Victor: Depends on the objective, prior knowledge and assumed model/complexity of the system under study + budget constraints Definitive Screening Designs and OML designs (Mixed-Level Screening Designs) help investigating curvature (quadratic) effects early in the screening stage Simon: if I already have existing DSD data and I want to augment them to a full RSM, how do I deal with the lack of randomization since I add on additional runs which I didnt measure together with the first data set? Victor: Use blocking when augmenting the design Phil: FitDSD will not work with missing runs unfortunately Simon: can you show the augment using blocking in the software? Victor: I added this default check in the Wish list: https://community.jmp.com/t5/JMP-Wish-List/DoE-Augmentation-Option-quot-Group-new-runs-into-separate-block/idi-p/665019   Question 2: How do I choose between different DOE types (e.g., full factorial, fractional factorial, response surface)?   Question 3: What to do when you lose a run of DOE   Victor: You could also run the model (if possible) without this run, and see if the model is able to approximate the response from the last added run?  And if you must expand ranges of your factors because the ranges are too narrow, you can augment your design instead of starting from scratch again to benefit from the information already gathered. David: The challenge is that I am already on the limit of runs and I can not go with one run less. So, I need to "augment" the design somehow. Any takes on whether you must use blocking?  Kevin: A common pitfall I see, and fall into myself, is to try to limit the number of factors at the start, to make things manageable (i.e. reduce the number of runs). So then on gutfeel eliminate these, instead of screening them out based on DoE.  Of course there is a limit, not to screen every thought that comes to mind, but it is often helpful to get a few experts in the meeting and brainstorm a starting set of factors. Victor: You have a design with saturated model? You can still maybe use some tools adapted for saturated designs, like the Fit Two-Levels Screening platform (The Fit Two Level Screening Platform) to try creating a relevant model.  And/Or as you mentioned, you could augment your incomplete first design and use Blocking for the new runs to prevent/detect any shift/variation between the first batch of experiments and the second one. CK: can you also run one that was run previously to help account? Simon: I don’t know if we already touched the topic, but I feel limited at final data analysis and model judgment. I learned at the JMP DoE course to analyse the ANOVA and lack of fit. What other and maybe better option do I have to judge my final model and maybe use the statistics of this model to plan a better experiment? Victor: There are many model metrics to use for model comparison/validation depending on your objective: R²/R² adjusted, p-values, RMSE, Information criteria, ... More discussion : https://community.jmp.com/t5/Discussions/Model-reduction/m-p/823421/highlight/true#M100286   Kevin: It is a tough question indeed David. Once idea to test/try, without knowing more about your case, but try to use a tool like Partial Least Squares that can handle missing data. Put your planned experiment in the X block, and then the Y predictors are what you measured, and leave the value you have missing in the Y column (or matrix if you have multiple responses).   PLS can make a prediction for you, though it might not be the best, it will be better than dropping out that run completely.   Also, what would have been your next step after this analysis, if you had the data point that is missing? Phil: I'd just repeat the missing run like Kim says. It's not ideal because "noise" factors could have changed but I don't think there is a better option. Putting it in a separate block makes no sense. A block with 1 run is not useful.  CK: centerpoints are good but could you repeat a previously successful run? Kevin: could you try simulating your problem? and in the simulation see the effect of dropping a run, and bringing it back in a later block, simulated with, or without a block effect. Create a fake system, where you know the truth, simulate experimental data that replicates your scenario. David: And then? how would you proceed? Kevin: Then do the analysis on the simulated data and see how important this missed run is to the insights you would (or would not) have gotten.   Since you know the truth in the simulated system, you should see that back in your analysis of the simulated data.  David: by "the truth" in the system, do you mean which effects are present? Kevin: Yeah, if that is the intention of your experiments is to discover which effects are at play in the range you are operating in. David: I would have fitted the data to get the model with the lowest AICc to describe the system Kevin: So, it sounds like you are in an early phase: screening out factors, and learning about the system. David: Exactly! That's why I feel that every run matters Kevin: In that case; the question to check in the simulation: would the potential screening or knowledge learned (e.g. factor A make response 1 decrease), would that knowledge learned been wrong without that single missing run, or would you have gotten the same info? In other words, it is not an issue having that missing run.   Question 4: Making use of the Box Cox Transformation option within a DOE analysis Victor: to know if a transformation is needed, you can check the residuals from your model first Diana: Box Cox transformation: I recommend rounding to the nearest big number (still inside the confidence interval of good transforms) because it will be much easier to later explain to the audience. Much easer to say "it is a log transformation" than "it is to the power of -0.132342356". Phil: Re box-cox transformation: JMP suggests a lambda number which is the power to which the response is raised as the transformation. 0 is equivalent to a log transform. 2 would be squaring the response. -1 is taking the inverse of the response. A log transform makes sense in some cases, e.g. where the response is a count of cell growth and therefore varies over orders of magnitude. I think it is hard to justify other transforms so I would avoid. The only exceptions are transforms that help when the response is on percent scale or on a 0 to 1 scale. Logit and logit pct transforms can be useful.   Question 5: What is the purpose of centre points, and determining appropriate factor ranges Bill: DOEs are notoriously sparse and methods like tree methods or SVEM are good options for analysing your DOE data. Victor: Topic of centre points also mentioned in previous DoE Club : https://community.jmp.com/t5/Design-of-Experiments-Club/Recordings-DOE-Club-Q1-2025/m-p/842727   Question 6: Data that doesn’t follow any DOE Principles. Victor: You can use the Augment design platform.  This helps build trust with the people having already done experiments with OFAT Chandra: To capture interactions and curvature after an OFAT study, use JMP’s Augment Design feature by specifying the same factors and a fuller model (main effects, interactions, quadratics). This generates additional runs that, when combined with your OFAT data, complete the design. Also adding a blocking factor to separate the original and new runs for better modelling.    Comment from Caleb King: I wanted to share something that is somewhat related to some things that were mentioned (Kim mentioned rounding factor levels to a nicer level and there's of course the discussion around using centerpoints to help with estimating pure error). I have a colleague at NASA who's run into situations where testers have a hard time achieving the specified design settings but can record the actual settings with high precision. There are methods available to use this situation to estimate pure error outside of using centre points. I've attached a presentation I gave 3 years ago on this topic (we had tried coming up with our own; turns out we were a few decades too late) that contains a summary and references for research in this area. Might be useful to you all.  (see attachement) ... View more