Design of Experiments Club

Video 1: Various ways to create models: model building, comparison and selection     Comments: @Victor_G - Some visualization and propositions for model comparison and selection (raster plots) proposed by Goos : https://community.jmp.com/t5/JMP-Wish-List/Raster-plots-or-other-visualization-tools-to-help-model/idi-p/730968   @martindemel - bootstrapping is what I thought of when saying tree based models, sorry fo rbeeing unprecise SVEM - Self Validating Ensemble Models: Discovery Summit Talk - SVEM: A Paradigm Shift in Design and Analysis of Experiments (2021-EU-45MP-779) - JMP User Community Support page - https://www.jmp.com/support/help/en/18.1/index.shtml#page/jmp/overview-of-selfvalidated-ensemble-models.shtml   @Jonas_Rinne - SVEM might be a method which uses some kind of validation without violating the design structure   @Victor_G - Or use statistical analysis, or Machine Learning algorithms robust to overfitting (random forests, SVM, ...)  Replicates are more used for explainability (sort the random error/variation/statistical error from the factors influences), I wouldn't use them for predictive modeling, particularly for Machine Learning modelling   Video 2: Mixture design with constraints    Comments:  @Bill_Worley - Mixture models can be highly non-linear and it may be good to start off with a space-filling mixture doe to try and better understand the non-linearity. @Victor_G Or do it sequentially : classical mixture design up to 2nd/3rd order, and then augment with Space-Filling to have points inside the experimental space.   Video 3: Number of trials : can we use the minimal advised by JMP in Optimal Design?    Comments:  @Bill_Worley - Try Augment Designs @Paul_J - This makes me wonder ... How does JMP decide the recommended number of runs? ... Could that give us an idea of why to use something more than the minimum? @Victor_G - Very often, 4 extra runs than minimum number of runs  Design Generation @Jonas_Rinne -  It also depends on your choices. If you have chosen some terms as "if possible" it will try to build these effects in with the added runs coming from the minimum. Otherwise it will add replicates for instance. JMP is trying to distribute the additional runs as best as possible based on your choices.  @martindemel - consider that you will get also information about the error estimate @Victor_G -  Not only correct model, but you can evaluate "natural" variability (random noise)  Without extra runs, you fit a model with the assumption that you have no experimental or measurmeent noise ! Completely agree with Brian, it all comes with a price / compromise   Video 4: Resources   Next DOE Club Session: https://www.jmp.com/en_gb/events/users-groups/users-group-meetings/doe-community-of-practice-multi-registration.html Time to Innovate: https://www.jmp.com/en_ca/events/live-webinars/time-to-innovate/time-to-innovate-28nov2024-doe.html DOE Club in the Community: https://community.jmp.com/t5/Design-of-Experiments-Club/gh-p/doe-grouphub Lifecycle DOE - https://www.mdpi.com/2306-5354/11/11/1089 ... View more

Question 2:  How to avoid Pitfalls    Comments @Victor_G :  1. Do not assume any conclusions/interpretations outside of your design space  2. And validate the model(s) with validation points!  @Stats_Like_Jazz :  Lesson 1: be brave when setting ranges.  Lesson 2 : there might be a lot of "good" models for your response(s). Choose a relevant criterion (or several ones) linked to your objective(s) to evaluate and select model(s).  Lesson 3: if you have ranges or combination to avoid use the disallowed combination filter.  I have a 4 minute video on how to do just that: https://youtu.be/0ILaduCaszE @ryan_cooper:  I'd say another pitfall (particularly for beginners) is overthinking the DOE.  Anything is better than one-factor-at-a-time.  You can build on your DOE knowledge as you do more of them - you don't have to know everything to get started.    Question 3: Mistakes using Definitive Screening Designs (DSD)   Comments:  @Jonas_Rinne  Do not try to learn everything in the first try. People trend to try to bring everything in in the first design. Most big designs do not work without a lot of prior knowledge.  @Victor_G   Also think iteratively. It's very rare to have all responses and conclusions with only one design. @KimRas2  Not adding enough potential factors  @Steph_Georges  Not Randomizing Runs: If the order of experimental runs is not randomized, lurking variables (such as time or temperature drift) may introduce bias into the results. Here is an article from Bradley Jones about "Proper and Improper use of DSD": https://community.jmp.com/t5/JMP-Blog/Proper-and-improper-use-of-Definitive-Screening-Designs-DSDs/ba-p/30703     Question 4:  Moving from DoE to Taguchi experiments - robust design   Comments:  @Victor_G  Depending on where/how you want to find robust process design, there may be better alternatives today with Optimal designs. And if you have several 2-levels categorical factors, try using OML designs instead: Mixed-Level Screening Designs (jmp.com) @KimRas2  IMH, Taguchi is a less modern form of DoE, where you have to buy expensive books on  stadardized DoE designs.   ... View more

Question 1: Random factors in DOE from @Stats_Like_Jazz    Comments:    @Victor_G  1. The use of random effects in the model enable to explain a part of the total variance, which is attributed to the random effect. This then may enable to have a better R² 2. You could use Operator as a random effect, and also use "Quality and Process" platforms (MSA, Control Charts, ...) to evaluate and visualize influence of operator (on the mean response and variance)  @Steph_Georges  Additional info about Blocking when designing experiments: https://community.jmp.com/t5/Mastering-JMP/Using-Blocking-When-Designing-Experiments/ta-p/277697 @ptolomey  1. Can you please show how to use "Reliability Life testing" platform to determine sample size for Reliability experiments? 2. Find out an energy activation parameter for Arrhenius model @WeightedMouse47  What was the original purpose of the experiment? @Jonas_Rinne  Was it a logistic regression model? Since you said pass/fail? @Victor_G  Maybe there are other y's that could be linked to this PASS/FAIL Y response, and be used as proxy with more information (continuous responses for example) ? There are too many terms for the model I think @jszarka  It's pretty clear from your studentized residuals by row number that the last set of 6 and the 6 before it is distinctly different.  That will be attributed to your Whole Plot effect. @KimRas2  There is definitely a pattern in your residuals, Christian.   ... View more

Thanks to all participants! Please feel free to elavorate more or ask questions below.    Video 1: What are the consequences (logically and mathematically) of not following the DOE in the order specified? - Richard Lacaba   Comments @Victor_G  - You can record the order of the experiments in the table and see if you have an effect of the experiment order on the response. @bdclark23  - I get this question often and provide this from Montgomery to my clients: Randomization is the cornerstone underlying the use of statistical methos in experiment design. By randomization we mean that both the allocation of the experimental material and the order in which the individual runs of the experiment are to be performed are randomly determined. Statistical methods require that the observations (or errors) be independently distributed random variables. Randomization usually makes this assumption valid. By properly randomizing the experiment, we also assist in “averaging out” the effect of extraneous factors that may be present.   Video 2: Augment Design (Sequential DOE) - John Szarka ( @jszarka )     Question from Thalita Coppini Soares ( @THCS :(  Could you run different ranges of the variables from the 1st experiment in this scenario?   Answers: @Jonas_Rinne  -  Yes you can change that as well if needed. @Victor_G  - Yes, you can change the ranges of the factors. This enables to focus in the optimal area (reduce the range, restrict the experimental space) or move the design space a little further if you have no interesting area to search/optimize for @Paul_J  - Or maybe adding a new factor with the next set? @Jonas_Rinne  - Here is a community post where you can check out how to do that: Solved: How do I add a new continuous factor to a DoE via design augmentation? - JMP User Community. Note that this factor had to be constant in your previous experiments.    Question from Thalita Coppini Soares ( @THCS :(   so, start with a PB to reduce the number of experiments, and then run a full design just with the main factors with narrow ranges?   Answer @Victor_G  - Or with another screening design, in the example here DSD could be helpful: no constraints and high number of continuous factors, so it could help better estimate main effects and possibly detect interactions and quadratic effects. And once your screening is done, why not adding higher order effects (interactions, quadratic effects, ...) in the augmentation on the narrow ranges to expand your knowledge about the system and have a better/more accurate model?   Video 3: Is it reasonable to keep main effects in the DoE Model even when they're not significant? How reliable is the effect size estimation of these "insignificant" factors? - Johann-Christoph Dettmann ( @JohannD124 )   Video 4: Integration of DOE with other methods: ML, chemoinformatics, etc... - Victor Guiller ( @Victor_G )   Comments @ruskicar - SVEM (self-validating ensemble modelling) is an approach that is sometimes used to model space-filling DOE’s @Victor_G - Design of experiments and machine learning with application to industrial experiments | Statistical Papers (springer.com) Design of experiments and machine learning with application to industrial experiments Statistical Papers - In the context of product innovation, there is an emerging trend to use Machine Learning (ML) models with the support of Design Of Experiments (DOE). The paper aims firstly to... Design choice and machine learning model performances - Arboretti - 2022 - Quality and Reliability Engineering International - Wiley Online Library @Paul_J  - Can JMP "optimal" designs be based on mechanical models?  @maria_astals - Space-Filing Designs: https://www.jmp.com/support/help/en/18.0/index.shtml#page/jmp/overview-of-spacefilling-designs.shtml @Victor_G   - Example of Space-Filling designs with Support Vector Machine analysis on mixture design : https://community.jmp.com/t5/Online-Abstracts/Synergy-Between-Design-of-Experiments-and-Machine-Learning-for/ec-p/708147#M2   Video 5: Modeling DoE results which include whole plots, especially the random error added to the Whole Plot effect. - Drejc Kopač ( @ruskicar ) ... View more

Hosted by @HadleyMyers , DOE enthusiasts shared discusions about several topics.  Feel free to continue the discussions below!    Video 1: Introduction   Video 2: Dealing with outliers   Comments:  @Victor_G : And also separate repetitions from replications to better understand if the problem comes from measurement process (repetitions), or from the system (with possible uncertainty in fixing factors levels, seen through replications)   Video 3: Uses of covariate factors   Comments:    @Victor_G : Jed Campbell from JMP did a great blog post about non-linear factors constraints : https://community.jmp.com/t5/JMP-Blog/Demystifying-Factor-Constraints/ba-p/619898 Demystifying Factor Constraints Isn't Define Factor Constraints something I should just leave alone or pretend doesn't exist? A wise man does nothing by constraint. -Marcus Tullius Cicero If you've felt intimidated by the Define...   @Paul_J : Did that all the time when I worked in food and chem industries.   @Bill_Worley : I'm a big proponent of using covariate candidate sets.   @Victor_G : Covariate sets are very useful in chemistry, when dealing with chemical properties as inputs for example   Video 4: best practices on setting ranges   Comments:  @Victor_G : Use the broad range in screening (possibly with centre points), and refine the range in exploration/optimization.   @KimRas2 : Be bold!     @shs: What works for me, if you start from scratch and you are unsure if your borders work, start with the experiment which you think might be the most critical. Here you waste only one experiment.  @Victor_G : Scoping designs are also very useful to define appropriate ranges and check variability in design space : https://prismtc.co.uk/resources/blogs-and-articles/scoping-designs You only try tests with -1 levels for all factors, centre point(s) and +1 levels for all factors. And you can very easily augment your scoping design into a factorial design   @Nicola_Nuti: Agreed, increasing effect size gives more power as well.    @Jonas_Rinne : The idea of scoping designs is super helpfull to check if your ranges make sense. Especially when you have many factors.   @SDF1: Even trials that don't work still provide very useful information for the DOE and the whole exploratory process.   @shs : With too narrow settings, you can never be sure about curvature., which is often the case with temperature.   @Paul_J : setting the ranges is good example of why to keep the science in the stats.   @Bill_Worley : If you have some historical data you can build a model and use the Design Space Profiler to help determine optimal ranges.   @Victor_G : Yes, DoE is a method change, but with more predictability about the outcomes, learnings and with planning made a lot easier (compared to trying One Factor At A Time and relying on best guess).   @SDF1 : Depending on how "new" the research is, sometimes a "pre-DOE" is necessary to determine what are the workable ranges for the factors that you're testing so that when you actually do sit down to create a more detailed/complex DOE, you're more likely to have all runs be successful.   Video 5: Modern DOE´s   @Victor_G : Really like the metaphor with the roof, thanks for sharing!   @shs : I rarely use classical design anymore, as they are large in experimental runs. Using concepts like SVEM with small designs, or definite screening designs, etc.. can give you designs which can cover large amount of factors with few runs, and that convinces people when they first see how much the experimental efforts can drop using modern designs. @Victor_G :  I often evaluate and compare modern designs with a possible "classical" counterpart, to better show the benefit of the modern design   @KimRas2 : Functional DoE?   @ryan_cooper : I often like to refer to some of Phil Kay's points in this Community discussion.  (The title of the question is misleading, however.)  Some discussion on why most things can be done with DSD and Custom design. https://community.jmp.com/t5/Discussions/Reason-against-using-Custom-Design-DOE-for-everything/td-p/407995   Video 6: Functional Data Explorer   @Victor_G : Functional DoE with curve output : https://community.jmp.com/t5/Discovery-Summit-Europe-2021/Use-of-Functional-Data-Explorer-in-a-Mixture-Design-for/ta-p/349208 You can use curve data (functional data) as inputs and/or output.   Lyndal asked: Does the standard JMP license cover functional DoE analysis? I thought that needed JMP Pro? Yes, it´s only in Pro   @Victor_G : As an input, you can use coefficients from functional analysis, and link these coefficients to your outputs   @Phil_Kay : @Ben_BarrIngh has some expertise in that field. Ben will be at Discovery Summit in Manchester if you able to make it.           ... View more