topic Re: Adding a complex constrain to a Custom Design with 4 factors in Discussions

Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Sat, 10 Jun 2023 23:54:35 GMT

Hello,

I am creating a Custom Design with 4 factors (Factors A, B, C & D).

I have a question about a constraint that has to do with factors B and D.
- Factor B is actually the number of reactions done in sequence in the same experiment. This can be 1 reaction, 2, 3 or more reactions in sequence. I set the range as 1-3 (not higher because I am not interested at the moment in >3 reactions in sequence).
- Factor D is the time I wait in between the reactions in sequence. Thus, I only define Factor D when Factor B is >1. The range is 0.1 to 10 seconds. Factor D would be 0 if Factor B is 1. In other words, when the number of reactions done in sequence (factor B) is just 1, Factor D doesn't exist because it was just 1 reaction and thus, I don't need to wait any seconds since there is no other reaction in sequence performed.

How can I define this constraint in my Custom Design model?

Thank youuuu!!
PS: I use a Mac.

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Fri, 23 Sep 2022 11:57:37 GMT

Hi @ADouyon,

Welcome to the Community !

Looking at your problem and options, here is what's recommended :

First, define your factor B as a discrete numeric, because the number of reactions is a whole number (you can't have 1,7 reactions for example, only 1, 2 or 3 in your case study),
Factor D should have a lower level fixed at 0 if you want to have a waiting time of 0 when there is only one reaction.
Since Factor D (time) is in your design a continuous factor (you can have any reaction time you want in the range specified), I would recommend to add the quadratic term D*D in the model, since for (chemical) reactions, time has not often a linear relation with the yield/output.
Depending if the response can be measured "online" without any degradation, perhaps it would make sense to record the response at several time intervals, in order to have a modelization of response vs. time (with the input factors) ?
Last but not least, I'm not familiar with your topic, but choosing different settings for factors may also solve your problem (changing waiting time for reaction time for example).

Based on your inputs, you can go into "Use Disallowed Combinations filter" ans select the range you want to exclude from your design (here, 1 =< B < 2 AND D > 0, see screenshot constraints for example). JMP should then offer you a design that adress this specific constraint (see screenshot Matrix and Experimental Space).

There may be better (and smarter) ways to do it, or more appropriate designs/factors configuration to think about this topic, but this should answer your needs and question.

I hope it will help you,

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Fri, 23 Sep 2022 14:17:28 GMT

Hello @Victor_G ,

Thank you very much for your quick response! This was very helpful!
I have a couple follow up questions:

1- When Factor B (the discrete numeric factor that is the number of biological reactions) is 2 or higher (3), then the Factor D cannot be zero. How can I add this second constraint in the model?
I tried it but I am not sure I did it right (screenshots attached).

2- The model automatically added the quadratic term B*B with estimability = if possible. Do you think I should keep this term? (I am trying to make this first design not too large)

Thank you!!

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Fri, 23 Sep 2022 15:15:48 GMT

Hi @ADouyon,

I'm glad this first answer helped you.

Concerning your other questions :

On the second screenshot, it seems that the two constraints are well specified. I also forgot to do it myself in my first answer, but you can change the format of your B factor in the filter constraint by clicking on the red triangle, then "Modeling type" = Categorical/Ordinal : this is easier to create constraints and avoid using things like 0<= B < 1,99 when we simply want to have B fixed at 1 (see screenshot constraints2). Looking at your design matrix and mine, constraints seem to have been respected, so it should work fine.
Yes, you should keep it, since JMP adds it and specifies its estimability as "If Possible" by default so it won't require more runs, but the runs will be better organized in the experimental space, so that in case you have enough degree of freedom in you model, JMP will try to estimate the quadratic term of your discrete numeric factor. And compared to a Categorical factor, having a Discrete Numeric factor means that JMP knows there is a relationship between the levels (they are ordered), compared to a pure Categorical factor (no relationships between levels, levels are nominal). There was a video explaining the difference between Discrete Numeric and Categorical: Dear Dr. DOE, isn’t Discrete Numeric just Categorical? - JMP User Community.

If you can afford more runs, I would suggest to add some replicate runs, to increase the power of your main effects (increasing the ability to detect a significant main effect if it is indeed present). You may have a severe experimental budget constraint, but comparing several design with several sample sizes can help you choose the most optimal situation for your needs.

I hope this follow-up will help you as well,

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Mon, 26 Sep 2022 20:43:06 GMT

Thank you so much, @Victor_G !! Much appreciate it. Your answers are super helpful!!

I took a couple courses on JMP DOE and read that we shouldn't invest >25% of our efforts on the first experiment. For us, time is the most limited resource we have. I was wondering, is it common to generate more than one design with different sample sizes for comparison, like you mentioned? Shouldn't the designs give the same result at the end? would you mind expanding a little more on that?

Thank you!!!
Best,

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Tue, 27 Sep 2022 16:47:38 GMT

Hi @ADouyon !

I'm glad the answers were helpful. I don't know how common this practice may be, but I personnaly prefer to spend more time comparing different designs and sample sizes than going to the lab and do the experiments quickly, to finally see that I may have forgotten a constraint, or that my design is not well suited enough for my needs.

--> To illustrate this, I really like this quote of Ronald Fisher : "To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of."

Different designs may lead to different interpretations of the results, depending on which terms are in the different models. For example, screening designs with main effects only may help filter relevant factors for the follow-up of a study, but may clearly lack precision and response(s) predictivity in presence of 2 factors interactions and/or quadratic effects (possible lack of fit).

You have several options to generate different designs and compare them:

Change (add/remove) the terms in the model (2 factors interactions, quadratic effects, ...) or their estimability (you can switch the estimability from "Necessary" to "If Possible" for 2FI and Quadratic effects).
Vary the sample sizes (number of experiments) in the design by adding replicate runs : are the extra runs relevant and useful enough to have a gain on estimates precision or relative variance prediction over the experimental space ?
Change the optimality criterion (depending on your target : D-optimality for screening, I-optimality for good response prediction precision, Alias-optimality to improve the aliasing between terms in the model...)
Change the type of design (if possible !)...

For the moment, to compare different DoEs you have to create them in JMP and then go to "DoE", "Design Diagnostics", and then "Compare Designs". But ... Spoiler alert ! : In JMP 17, it will be a lot easier to compare several designs (see screenshots), as a "Design Explorer" platform will be included in the DoE creation, so no need to manually create them one by one : you can create several designs in seconds, and filter them based on your objectives and decision criteria.

And finally, take advantage of "iterative designs" and don't expect or try to solve all your questions with one (big) design. It may be more powerful and efficient to start small with a screening design for example, and then use Augmentation to search for more details, like interactions and possible non-linear effects. Finally, if your goal is to optimize something, you can once again augment your design to fit a full Response Surface Model. At each step, you gather more knowledge about your system, and make sure that the next augmented design will bring even more knowledge, without wasting experiments/runs, time and ressources.

"Augment design" platform is really a powerful (and often underestimated) tool to gather more data and gain more understanding and build more accurate models without losing any previous information.

I hope this answer will help you,

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Wed, 28 Sep 2022 11:24:04 GMT

One additional comment to mention a great ressource about the sequential nature of experiments, by Prism Training & Consultancy : Article: The Sequential Nature of Classical Design of Experiments | Prism (prismtc.co.uk)

This article explains very well the process behind DoE, from creating and assessing the experimental scope for factors, to screening and then optimization. This is part of a serie of article that may have an interest for you.

Happy reading !

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Thu, 29 Sep 2022 13:41:02 GMT

Thank you incredibly, @Victor_G! Very much appreciated! Your comments are extremely useful and clear, thank you!

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Thu, 29 Sep 2022 13:42:26 GMT

Thanks once again, Victor! :)

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Thu, 29 Sep 2022 15:18:23 GMT

Thanks a lot for your kind comments and feedback @ADouyon !
I'm happy if my answers were helpful and clear enough ! :)

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Thu, 13 Oct 2022 12:05:37 GMT

Hello @Victor_G again! I have a quick question; do you know why I cannot save my model constraints like we can do with the factors and responses?
thank you!
Best,

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Thu, 13 Oct 2022 12:34:44 GMT

Also, @Victor_G do you know how I can specify in the model that my factor values cannot have decimal numbers (ie. 0.5)? Two of my factors are variables that I set up in an instrument. But, the instrument doesn't let you use decimal units because it has to be a whole number. The instrument lets the user change the value of these factors in increments of 1 unit (ie. 21, 22, 23...).

Can I add this in my constraints somehow as well?

Thank you in advance @Victor_G !
Best,

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Thu, 13 Oct 2022 13:17:34 GMT

Hi @ADouyon,

It seems that you can save and load constraints only if it is done through the option "Specify Linear Constraints".

What you can do in your case, is

to generate your DoE entirely with the constraint you added, and then you'll see that a new script in your datatable called "Disallowed Combinations" (see screenshot Disallowed_Combinations-datatable-script.jpg).
If you right click on it, then click "Edit", you can copy the JSL code of the disallowed combinations (see Disallowed_Combinations-datatable-script-Edit.jpg) and...
Use it (paste it) in the Custom Design creation, in the "Use Disallowed Combinations Script" option (see screenshot "Use-disallowed-combinations-script.jpg).

There might be a smarter and more automatic way to do this (via JSL scripting maybe ?), but this approach work (and you don't need to manually recreate your disallowed combinations).

Hope it will help you ! :)

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Thu, 13 Oct 2022 13:52:34 GMT

Hello @ADouyon,

For the factors that can't take any decimal value, you have two options :

The first and easy one may be to change the numbers without decimal values in the datatable, by changing the format to "Fixed Dec" and Dec = 0 (see screenshot). But the design may lose a little bit of its optimality, and the optimal values for these factors from the model for the prediction profiler may have decimals.
The other way is to treat those factors as "discrete numeric", with specified levels (see my previous comments for more info about Discrete Numeric factor type). This way looks better to me, as this "constraint" is directly taken into account in the model. In your case it might not be interesting to use all the possible levels of your equipment (with an increment of 1) in the levels of the discrete numeric factor, but choosing at least 3 levels (min, medium and max) may be interesting in order to look at possible quadratic effects. In the prediction profiler, it will be treated as categorical factor, but keeping in mind the ordinality of the levels (like 2 < 5 < 7 ... see screenshot for Factor 4), so you can have a look at the trend between each levels.

There might be other options to do it, so don't hesitate to create a new post and check for other users' inputs and responses.

At the end, choose the option you're the most confortable with.
Hope it helps you,

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Fri, 14 Oct 2022 19:46:53 GMT

Thank you very much @Victor_G! I just tried it and was able to do it :)

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Fri, 14 Oct 2022 19:52:32 GMT

Thank you again @Victor_G for always responding so fast and with such clear answers!! Much appreciated!

(I may try to make an additional post for that one; just out of curiosity :))

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Tue, 18 Oct 2022 21:22:07 GMT

Hi again @Victor_G,

I have a follow up a couple follow-up questions:

1- If I want to maximize my response to the highest possible value, I still have to define the 'lower limit' for the response, right? But, does this 'lower limit' matter much? Does it matter if I set this to 10 or 50 or 100? (I don't know how to go about this 'lower limit' value ... as I really just care about getting the max response possible).

2- This DOE I am working on is for a biological reaction. Usually, in science/biology, we perform replicate experiments for each condition (run); that means we perform each experimental run in duplicates (or triplicates). I understand, that this is not practical: If my DOE consists of 18 runs total, it does not make sense for me to duplicate all 18 runs, because that would be a total of 36 runs, which would defeat the purpose of me using DOE...
How do I go about deciding the number of replicate runs, then? How to decide if I need just 2 or 4 or 10 replicates?

3- On that same note, in science/biology, researchers usually add some negative and positive control reactions to their experiments as well. That is a reaction we expect to fail because we know in advance it will fail and a reaction we expect to succeed because we have prior knowledge that one works well. How do DOE users go about this in general?

Thank you very much @Victor_G for your help!!!
Best,

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Wed, 19 Oct 2022 08:57:35 GMT

Hi again @ADouyon,

The "lower limit" is the lowest acceptable solution for your situation (it defines a possible and acceptable range of values from your lowest limit to your highest limit), and may have an impact if you have several responses. In the case of an unique response, it doesn't matter much because JMP will either way try to optimize the response (maximize/minimize or match value), so the lower limit will be in any case avoided at all cost by finding the right factors settings (in the case of "Maximize"). The change will be in the slope of the desirability function (see screenshot Profiler-1 done with lower limit 350 and Profiler-2 done with lower limit 450 on the use case "Bounce Data" from JMP).

If you have several responses, changing the slope of one of your response desirability profile may change the optimum found, because JMP will try to maximize the overall desirability value (taking into account all responses desirabilities functions), so having a steeper slope for a response means that this response may have a bigger influence on the desirability optimization solution (because you'll reduce the range of acceptable values, see comparison of profiler optimization with screenshots Profiler-desirability-1 and Profiler-desirability-2 on the dataset Tiretread.jmp from JMP data use cases). Also take into account that the importance fixed for each of your response will also greatly influence the optimum found : the bigger the response importance, the bigger the influence of this response on the optimum situation found.
More infos on the desirability profile can be found here : The Desirability Profile (jmp.com)
The use and number of replicates is determined by your goal, power for effects, precision expected for your experiments/predictions, ... Be careful in JMP there are two options : "number of replicate runs" (in Custom Design), meaning here the number of run(s) to repeat, and "number of replicates" (in Classical Design), meaning he number times to perform each run (in addition to the original runs) : Re: Custom Design Number of Replicates - JMP User Community.

It's difficult without knowing all of this (and without domain expertise) to know what may be the more suitable choice for your case, but I would recomment to try different options (no replicate runs, 2 replicate runs, 4 replicate runs, 8 replicate runs...) and use the "Compare Designs" platform (found in DoE -> Design Diagnostics) to evaluate which number of runs would be optimal for your case (biggest impact for the lowest number of replicates). Attached you can find the different comparisons made with your DoE settings for 0, 2, 4 and 8 replicate runs for Power, Variance and Correlation comparisons, with the individual datatable created for each DoE. Here for 2 and 4 replicate runs, JMP ends up with the same default number of experiments (18) with similar properties for these designs. However, we can clearly see a gap between 0 and 2/4 replicate runs in terms of power, variance prediction profile (almost divided by a factor 2 in the case of 2 or 4 replicate runs compared to 0 replicate runs) and correlation matrix/color map (aliasing between terms in the model), so the 18 runs design (either 2 or 4 replicate runs) may be a good compromise here to have reliable results and precision without augmenting too much the experimental costs (27 experiments for a 8 replicate runs design !).

If you really want to compare with a duplicate/triplicate... design (to have replicates and not only replicate runs), you can create the original design proposed by JMP (with 9 experiments, without any replicate runs), and then augment it (in DoE -> Augment Design) by selecting the option "Replicate". JMP will then ask you the number of times to perform each run, so if you enter "2", you'll end up with your duplicate design with 18 runs. Note that in the case here, the two options (replicates and replicate runs) bring very similar design performances for the same number of runs (here tested with 18 runs in total).
I don't know precisely how to handle this. If you don't know in your experimental space which conditions will make your experiment succeed/fail, one option could be to test your experimental space first with a "Scoping design" (Scoping Designs | Prism (prismtc.co.uk)) to check if your factors conditions are possible in the lab (and then augment this design to a screening one, and perhaps after augment it to an optimization one with Response Surface models).

If you already know which factors conditions will make your experiment fail or succeed, you can perhaps include the settings and create appropriate runs in your DoE, and include or add these runs as well in your analysis, either as part of the design, or as validation points (excluded for the analysis of the design, but added at the end to check predictions of the design match your knowledge).

I hope these answers will help you (sorry for the long post),

PS: I have 9 runs in my original design with no replicate runs, this is because I didn't add the terms X1*X1 and X1*X2 in the model, but the general presentation and comparison of design may still be valid (you may have to verify to check it, if needed I can try to change the designs and provide a valid comparions and screenshots today).

Re: Adding a complex constrain to a Custom Design with 4 factors

Victor_G — Wed, 19 Oct 2022 13:52:34 GMT

Hi @ADouyon,

With the correct designs and models, the comparisons are a bit different:

For power, there seems to be no significant improvement between designs with no replicates runs, or 2 or 4 replicate runs. Having 8 replicate runs (for a total of 27 experimental runs) or replicate entirely the original design (for 36 experiments in total) does improve the power.
For variance prediction, having 2 replicate runs (for the same number of total runs : 18) seems to lower more efficiently the variance over the entire space than with having 4 replicate runs (or none). Having 8 replicate runs (for 27 experiments !) does lower the prediction variance over the experimental space a little bit, but it is quite similar to what you get with 2 replicate runs. And for sure, if your replicate your original design (and double the number of experiments from 18 to 36...), your prediction variance will be divided by two.
In terms of efficiencies and aliasing, replicating the entire design will give you the same aliasing/correlation between terms, whereas adding replicate runs can change and lower some aliasing. Efficiencies are quite similar for designs with the same number of runs, and will fore sure increase when you augment your number of experiments.

Hope this part will help you and not bring additional confusion,

Re: Adding a complex constrain to a Custom Design with 4 factors

ADouyon — Thu, 20 Oct 2022 16:01:06 GMT

Hi @Victor_G,

Following up on your response from Sept 27 about comparing multiple designs, which you said will be easier to do in JMP17. I was wondering, do you know when we will be able to use this new version of JMP17? It seems a great feature!

1- Also, to make sure I understand this. When you say that you like to compare designs first. You mean on JMP, not in the lab. So you prefer to always compare different designs with JMP DOE before physically testing the design, correct?

2- I don't recall seeing anything about DOE design comparisons in the "Custom Design of Experiments" course. Would it be possible @Victor_G to provide more information on how you go from "DOE > Design Diagnostics > compare designs" to 1) the power, 2) variance prediction profile and 3) correlation matrix/color map? (I tried it but I am confused on what exactly I should select...)

3- On the same note, what is the meaning of this "Design Diagnostics" information (see screenshot) that one can see right before making the table?

4- Last question :) Why when I am making the design again, now JMP chose two runs with 'X1' data points as 1.93 and 1.89 (two separate runs) that are not centered at all. X1 is set up as a continuous factor with a range of 1-3. I am okay with the decimals for this factor, however, I wonder why JMP chose those two values so close to each other instead of values more spread out apart in the experimental space?

Thank you very much @Victor_G for your fast and detailed responses!! Super helpful, thank you!!! I am learning A LOT!