Design of Experiments Club

Next Sessions are:  23 Sep | Q3: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/23-sep 25 Nov | Q4: https://www.jmp.com/en/events/europe/users-groups/doe-club/2025/25-nov   Question 1: How do I handle situations where my analysis shows no significant effects or interactions? Chandra: Use Graphical Exploration Profiler, Contour Plots, and Interaction Plots in JMP can help spot trends that aren’t statistically significant yet may warrant further study. Victor: Depends on the objective, prior knowledge and assumed model/complexity of the system under study + budget constraints Definitive Screening Designs and OML designs (Mixed-Level Screening Designs) help investigating curvature (quadratic) effects early in the screening stage Simon: if I already have existing DSD data and I want to augment them to a full RSM, how do I deal with the lack of randomization since I add on additional runs which I didnt measure together with the first data set? Victor: Use blocking when augmenting the design Phil: FitDSD will not work with missing runs unfortunately Simon: can you show the augment using blocking in the software? Victor: I added this default check in the Wish list: https://community.jmp.com/t5/JMP-Wish-List/DoE-Augmentation-Option-quot-Group-new-runs-into-separate-block/idi-p/665019   Question 2: How do I choose between different DOE types (e.g., full factorial, fractional factorial, response surface)?   Question 3: What to do when you lose a run of DOE   Victor: You could also run the model (if possible) without this run, and see if the model is able to approximate the response from the last added run?  And if you must expand ranges of your factors because the ranges are too narrow, you can augment your design instead of starting from scratch again to benefit from the information already gathered. David: The challenge is that I am already on the limit of runs and I can not go with one run less. So, I need to "augment" the design somehow. Any takes on whether you must use blocking?  Kevin: A common pitfall I see, and fall into myself, is to try to limit the number of factors at the start, to make things manageable (i.e. reduce the number of runs). So then on gutfeel eliminate these, instead of screening them out based on DoE.  Of course there is a limit, not to screen every thought that comes to mind, but it is often helpful to get a few experts in the meeting and brainstorm a starting set of factors. Victor: You have a design with saturated model? You can still maybe use some tools adapted for saturated designs, like the Fit Two-Levels Screening platform (The Fit Two Level Screening Platform) to try creating a relevant model.  And/Or as you mentioned, you could augment your incomplete first design and use Blocking for the new runs to prevent/detect any shift/variation between the first batch of experiments and the second one. CK: can you also run one that was run previously to help account? Simon: I don’t know if we already touched the topic, but I feel limited at final data analysis and model judgment. I learned at the JMP DoE course to analyse the ANOVA and lack of fit. What other and maybe better option do I have to judge my final model and maybe use the statistics of this model to plan a better experiment? Victor: There are many model metrics to use for model comparison/validation depending on your objective: R²/R² adjusted, p-values, RMSE, Information criteria, ... More discussion : https://community.jmp.com/t5/Discussions/Model-reduction/m-p/823421/highlight/true#M100286   Kevin: It is a tough question indeed David. Once idea to test/try, without knowing more about your case, but try to use a tool like Partial Least Squares that can handle missing data. Put your planned experiment in the X block, and then the Y predictors are what you measured, and leave the value you have missing in the Y column (or matrix if you have multiple responses).   PLS can make a prediction for you, though it might not be the best, it will be better than dropping out that run completely.   Also, what would have been your next step after this analysis, if you had the data point that is missing? Phil: I'd just repeat the missing run like Kim says. It's not ideal because "noise" factors could have changed but I don't think there is a better option. Putting it in a separate block makes no sense. A block with 1 run is not useful.  CK: centerpoints are good but could you repeat a previously successful run? Kevin: could you try simulating your problem? and in the simulation see the effect of dropping a run, and bringing it back in a later block, simulated with, or without a block effect. Create a fake system, where you know the truth, simulate experimental data that replicates your scenario. David: And then? how would you proceed? Kevin: Then do the analysis on the simulated data and see how important this missed run is to the insights you would (or would not) have gotten.   Since you know the truth in the simulated system, you should see that back in your analysis of the simulated data.  David: by "the truth" in the system, do you mean which effects are present? Kevin: Yeah, if that is the intention of your experiments is to discover which effects are at play in the range you are operating in. David: I would have fitted the data to get the model with the lowest AICc to describe the system Kevin: So, it sounds like you are in an early phase: screening out factors, and learning about the system. David: Exactly! That's why I feel that every run matters Kevin: In that case; the question to check in the simulation: would the potential screening or knowledge learned (e.g. factor A make response 1 decrease), would that knowledge learned been wrong without that single missing run, or would you have gotten the same info? In other words, it is not an issue having that missing run.   Question 4: Making use of the Box Cox Transformation option within a DOE analysis Victor: to know if a transformation is needed, you can check the residuals from your model first Diana: Box Cox transformation: I recommend rounding to the nearest big number (still inside the confidence interval of good transforms) because it will be much easier to later explain to the audience. Much easer to say "it is a log transformation" than "it is to the power of -0.132342356". Phil: Re box-cox transformation: JMP suggests a lambda number which is the power to which the response is raised as the transformation. 0 is equivalent to a log transform. 2 would be squaring the response. -1 is taking the inverse of the response. A log transform makes sense in some cases, e.g. where the response is a count of cell growth and therefore varies over orders of magnitude. I think it is hard to justify other transforms so I would avoid. The only exceptions are transforms that help when the response is on percent scale or on a 0 to 1 scale. Logit and logit pct transforms can be useful.   Question 5: What is the purpose of centre points, and determining appropriate factor ranges Bill: DOEs are notoriously sparse and methods like tree methods or SVEM are good options for analysing your DOE data. Victor: Topic of centre points also mentioned in previous DoE Club : https://community.jmp.com/t5/Design-of-Experiments-Club/Recordings-DOE-Club-Q1-2025/m-p/842727   Question 6: Data that doesn’t follow any DOE Principles. Victor: You can use the Augment design platform.  This helps build trust with the people having already done experiments with OFAT Chandra: To capture interactions and curvature after an OFAT study, use JMP’s Augment Design feature by specifying the same factors and a fuller model (main effects, interactions, quadratics). This generates additional runs that, when combined with your OFAT data, complete the design. Also adding a blocking factor to separate the original and new runs for better modelling.    Comment from Caleb King: I wanted to share something that is somewhat related to some things that were mentioned (Kim mentioned rounding factor levels to a nicer level and there's of course the discussion around using centerpoints to help with estimating pure error). I have a colleague at NASA who's run into situations where testers have a hard time achieving the specified design settings but can record the actual settings with high precision. There are methods available to use this situation to estimate pure error outside of using centre points. I've attached a presentation I gave 3 years ago on this topic (we had tried coming up with our own; turns out we were a few decades too late) that contains a summary and references for research in this area. Might be useful to you all.  (see attachement) ... View more

Question 1: DOE model analysis     Comments: @Phil_Kay  @Victor_G : Stepwise selection might not be the default approach to recommend.  It is helpful as a guide and comparison, but I would prefer to compare the outcomes with GenReg and other platforms  Raster plots can also help see the frequencies of terms inclusion on many models, to see how they compare : https://community.jmp.com/t5/JMP-Wish-List/Raster-plots-or-other-visualization-tools-to-help-model/idi-p/730968   @bdclark23 : Multiple model approaches with a preponderance of evidence that consistently show the same effects across different models, each with their own biases and assumptions.   Question 2: “Covariate Approach” of applying constraints   Comments @Victor_G : About constraints : https://community.jmp.com/t5/JMP-Blog/Demystifying-Factor-Constraints/ba-p/619898#U619898 @Ryan_Lekivetz : https://community.jmp.com/t5/Abstracts/Candidate-Set-Designs-Tailoring-DOE-Constraints-to-the-Problem/ev-p/756629 @Phil_Kay : I posted about this on LinkedIn recently as well: https://www.linkedin.com/posts/drphilkay_candidate-set-approach-for-complex-experiments-activity-7290301797410168833-JxRT?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAGJ9cABNhzDhYBtevawkkMdSNm3keFRm68 Love this discussion. It's relatively easy to jump (JMP!) to the technical answer. But important to challenge the "why" as well.  @Chandramouli_R :    Video 3: Full factorial design: How many center points you recommend?   @Phil_Kay : I've posted about centre points before as well: https://www.linkedin.com/posts/drphilkay_doebyphilkay-doebyphilkay-designofexperiments-activity-7056885385095729152-pu9F?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAGJ9cABNhzDhYBtevawkkMdSNm3keFRm68 @Victor_G : Centre points can help detecting curvature, but doesn't allow estimation of several quadratic effects Q: How to add center points in a custom design? A:    @Victor_G : About scoping designs and their use as preliminary runs to check design space : Scoping Designs | Prism   Video 4: How can I reset / delete factors in the easy DoEs without changing the experiment design?   Comments:  @Chandramouli_R : https://community.jmp.com/t5/Discussions/Hard-to-change-factors-center-points-and-replicates/td-p/224712  @Victor_G :  In "analysieren" you can select the effects in and out of the model Tab "analysieren" --> Try "Bestes Model"  Or create a custom model with "Fit Model" using the JMP datatable with your results @Jonas_Rinne : I think you can also do this by going into the flexible mode in Easy DOE @Victor_G : Fit your responses separately.  But I think there are many ressources to help you create and compare models. OFAT has no model behind, it's only a comparative approach, so not very helpful to guide decision-making   Video 5: Analytic Culture for Using DOE   Comments:  @Uwe_Weber : Use the Profiler and they will understand @anchildress : We have the most success at Eastman when we focus on relationship building with customers/clients, starting small to build trust with you and your techniques. Once they see some wins or successful DOEs with colleagues perhaps they are more trusting of DOE.   Results and cost savings are the most convincing technique @DB1 : Changing a culture: Patience. Work around the edges. When you can convince one scientist or engineer, they can be more convincing of DOE use to their teammates than us. As Bea says, one really good DOE dataset in a company can convince a lot of people if used diplomatically and audience-aimed.     ... View more

Video 1: Various ways to create models: model building, comparison and selection     Comments: @Victor_G - Some visualization and propositions for model comparison and selection (raster plots) proposed by Goos : https://community.jmp.com/t5/JMP-Wish-List/Raster-plots-or-other-visualization-tools-to-help-model/idi-p/730968   @martindemel - bootstrapping is what I thought of when saying tree based models, sorry fo rbeeing unprecise SVEM - Self Validating Ensemble Models: Discovery Summit Talk - SVEM: A Paradigm Shift in Design and Analysis of Experiments (2021-EU-45MP-779) - JMP User Community Support page - https://www.jmp.com/support/help/en/18.1/index.shtml#page/jmp/overview-of-selfvalidated-ensemble-models.shtml   @Jonas_Rinne - SVEM might be a method which uses some kind of validation without violating the design structure   @Victor_G - Or use statistical analysis, or Machine Learning algorithms robust to overfitting (random forests, SVM, ...)  Replicates are more used for explainability (sort the random error/variation/statistical error from the factors influences), I wouldn't use them for predictive modeling, particularly for Machine Learning modelling   Video 2: Mixture design with constraints    Comments:  @Bill_Worley - Mixture models can be highly non-linear and it may be good to start off with a space-filling mixture doe to try and better understand the non-linearity. @Victor_G Or do it sequentially : classical mixture design up to 2nd/3rd order, and then augment with Space-Filling to have points inside the experimental space.   Video 3: Number of trials : can we use the minimal advised by JMP in Optimal Design?    Comments:  @Bill_Worley - Try Augment Designs @Paul_J - This makes me wonder ... How does JMP decide the recommended number of runs? ... Could that give us an idea of why to use something more than the minimum? @Victor_G - Very often, 4 extra runs than minimum number of runs  Design Generation @Jonas_Rinne -  It also depends on your choices. If you have chosen some terms as "if possible" it will try to build these effects in with the added runs coming from the minimum. Otherwise it will add replicates for instance. JMP is trying to distribute the additional runs as best as possible based on your choices.  @martindemel - consider that you will get also information about the error estimate @Victor_G -  Not only correct model, but you can evaluate "natural" variability (random noise)  Without extra runs, you fit a model with the assumption that you have no experimental or measurmeent noise ! Completely agree with Brian, it all comes with a price / compromise   Video 4: Resources   Next DOE Club Session: https://www.jmp.com/en_gb/events/users-groups/users-group-meetings/doe-community-of-practice-multi-registration.html Time to Innovate: https://www.jmp.com/en_ca/events/live-webinars/time-to-innovate/time-to-innovate-28nov2024-doe.html DOE Club in the Community: https://community.jmp.com/t5/Design-of-Experiments-Club/gh-p/doe-grouphub Lifecycle DOE - https://www.mdpi.com/2306-5354/11/11/1089 ... View more

Question 2:  How to avoid Pitfalls    Comments @Victor_G :  1. Do not assume any conclusions/interpretations outside of your design space  2. And validate the model(s) with validation points!  @Stats_Like_Jazz :  Lesson 1: be brave when setting ranges.  Lesson 2 : there might be a lot of "good" models for your response(s). Choose a relevant criterion (or several ones) linked to your objective(s) to evaluate and select model(s).  Lesson 3: if you have ranges or combination to avoid use the disallowed combination filter.  I have a 4 minute video on how to do just that: https://youtu.be/0ILaduCaszE @ryan_cooper:  I'd say another pitfall (particularly for beginners) is overthinking the DOE.  Anything is better than one-factor-at-a-time.  You can build on your DOE knowledge as you do more of them - you don't have to know everything to get started.    Question 3: Mistakes using Definitive Screening Designs (DSD)   Comments:  @Jonas_Rinne  Do not try to learn everything in the first try. People trend to try to bring everything in in the first design. Most big designs do not work without a lot of prior knowledge.  @Victor_G   Also think iteratively. It's very rare to have all responses and conclusions with only one design. @KimRas2  Not adding enough potential factors  @Steph_Georges  Not Randomizing Runs: If the order of experimental runs is not randomized, lurking variables (such as time or temperature drift) may introduce bias into the results. Here is an article from Bradley Jones about "Proper and Improper use of DSD": https://community.jmp.com/t5/JMP-Blog/Proper-and-improper-use-of-Definitive-Screening-Designs-DSDs/ba-p/30703     Question 4:  Moving from DoE to Taguchi experiments - robust design   Comments:  @Victor_G  Depending on where/how you want to find robust process design, there may be better alternatives today with Optimal designs. And if you have several 2-levels categorical factors, try using OML designs instead: Mixed-Level Screening Designs (jmp.com) @KimRas2  IMH, Taguchi is a less modern form of DoE, where you have to buy expensive books on  stadardized DoE designs.   ... View more