topic Re: Custom DoE screening design in Discussions

Custom DoE screening design

JMP38401 — Thu, 08 Jun 2023 21:05:14 GMT

I have two very basic questions about custom DoE screening design.

1. If I only wanted to know which main effects are significant to my responses and use the custom main effect screening design with the default run numbers given by JMP, how can I be sure whether it is the main effect or other confounding two-way interaction that is really significant? If the model analysis shows factor A is significant but A is partially correlated with B*C, what factor(s) should be included in the optimization design later? I suppose only the main effects that are significant such as factor A should be further investigated for the optimization design following the main effect screening design but will B and C be missing if B*C partially correlated with factor A but B and C are not significant main effects from the screening DoE?

2. when I include some uncontrolled factors in the model, JMP does not show the color map for correlation any more. How can I evaluate the design with uncontrolled factors?

Thanks

Re: Custom DoE screening design

Mark_Bailey — Tue, 27 Jul 2021 17:01:38 GMT

"If I only wanted to know which main effects are significant to my responses and use the custom main effect screening design with the default run numbers given by JMP, how can I be sure whether it is the main effect or other confounding two-way interaction that is really significant?"

Enter the main effects in the term list under Model. Make sure that all the two-way interactions are in the term list under Alias. Click the red triangle at the top and select Optimality Criterion > Make Alias Optimal Design.

"If the model analysis shows factor A is significant but A is partially correlated with B*C, what factor(s) should be included in the optimization design later? I suppose only the main effects that are significant such as factor A should be further investigated for the optimization design following the main effect screening design but will B and C be missing if B*C partially correlated with factor A but B and C are not significant main effects from the screening DoE?"

If you suspect higher-order terms are potentially active, then there are two ways that I recommend. First, add all the potential terms in the Model list. Select these terms and click on Necessary in the Estimability column, then select If Possible. This change will produce two results. The first result is that the minimum number of runs will decrease because it is determined by the number of parameters that must be estimated. (You need to guess the number of actually active terms and add runs to the minimum for them.) The other way is to use a screening design like Definitive Screening Design. This way will work as long as it is a true screening situation in which the key screening principles hold. How many factors to you have? How many are likely to be active?

I do not know the answer to your second questions, off the top of my head. Perhaps someone else has that knowledge.

Re: Custom DoE screening design

statman — Tue, 27 Jul 2021 17:08:06 GMT

@JMP38401 , Here are my thoughts:

1. There are three principles we rely on for fractional factorials and screen designs:

Scarcity: There are relatively few significant effects (analogous to the Pareto Principle)

Hierarchy: 1st order > 2nd order >> 3rd order, etc.

Heredity: In order for an interaction to be significant at least one parent must be significant

Regarding your question, my advice is to predict the rank order model effects (at least through 2nd order). Your predictions as to which effects you believe would be reasonable and likely will impact design resolution selection. If all of your 1st order effects rank above 2nd order effects, then lower resolution seems reasonable to begin the iterative process of investigation. In fact, this is the hierarchy principle. Expand the number of factors (1st order effects) by confounding higher order effects.

2. If you suspect interaction effects (≥2nd order), then you might want to bump resolution to IV+.

3. I know I don't represent the bulk of the thinking on optimal designs. I am not a huge fan of "partial confounding" to create a more efficient design as if there are instances that do not make sense in the data analysis, the next iteration can be a difficult choice (e.g., fold over designs don't work).

4. I don't completely understand your second question. If you have covariates in the data table, you can certainly see the correlation between the covariates and the design factors. Multivariate Methods>Multivariate will provide scatterplots and selecting the options (red triangle) you can get color maps. Of course if you have many, you can get VIFs by right-clicking in the parameter estimates out put table and adding >Columns>VIFs.

Re: Custom DoE screening design

JMP38401 — Tue, 27 Jul 2021 18:05:33 GMT

Thank you for the quick reply!

"Enter the main effects in the term list under Model. Make sure that all the two-way interactions are in the term list under Alias. Click the red triangle at the top and select Optimality Criterion > Make Alias Optimal Design."

I tried this and did not see any changes in Alias Matrix under Design Evaluation of the models. Did I miss anything or how do I know the model is optimized for Alias? Sorry if I misunderstood how the "Make Alias Optimal Design" works by assuming I will see a lower correlation effect under Alias Matrix or the color map of correlation.

"(You need to guess the number of actually active terms and add runs to the minimum for them.)"

I have 7 continuous factors with 3 of them HTC. I also have 3 uncontrolled factors that I will record their values during my DoE runs so totally 10 factors. When I created the screening design using Custom DoE platform, I have 11 Minimum Number of Runs (10 factors plus the intercept) and 18 runs as Default. There are 45 two-way interactions under Alias Terms. If I suspect 10 out of those 45 two-way interactions could be active terms, your suggestion is to add those 10 two-way interaction to the Model, change their Estimability to If Possible, and add 10 runs to the minimum 11 runs to make it 21 runs total or add the 10 runs to the default 18 runs to make it 28 runs total? Since I have HTC factors in the model, the definitive screening design does not seem to work.

Re: Custom DoE screening design

JMP38401 — Tue, 27 Jul 2021 18:12:00 GMT

Thank you for the quick response!

"2. If you suspect interaction effects (≥2nd order), then you might want to bump resolution to IV+."

I know how to find the design resolutions in Minitab but am not able to tell the design resolution from JMP Custom Design. Can you please let me know how to get the design resolution information in JMP?

"4. I don't completely understand your second question. If you have covariates in the data table, you can certainly see the correlation between the covariates and the design factors. Multivariate Methods>Multivariate will provide scatterplots and selecting the options (red triangle) you can get color maps. Of course if you have many, you can get VIFs by right-clicking in the parameter estimates out put table and adding >Columns>VIFs."

Sorry for the confusion. When I don't have any Uncontrolled factors (under factor Role column, it says Uncontrolled instead of continuous etc), I can see the color map of correlation and Alias Matrix under Design Evaluation. However, once I add any Uncontrolled factors, the Design Evaluation is gone and I am not able to see the Alias Matrix, color map of correlation and Power Analysis etc.

Re: Custom DoE screening design

Mark_Bailey — Tue, 27 Jul 2021 18:53:10 GMT

The Custom Design is very flexible. It is possible to define a main effects model and the two-interaction alias matrix and still get correlations. For example, the number of runs might prevent an orthogonal design to be made. Think of 3 factors in 7 runs.

Can you post a picture of the custom design elements (factors, model, alias matrix, design, and correlation map)?

Re: Custom DoE screening design

JMP38401 — Tue, 27 Jul 2021 19:09:17 GMT

Above is the design with 10 factors (7 continuous factors with 3 of them HTC. 3 more uncontrolled factors.

After I made the design, I don't see the Design Evaluation platform as I normally see when there is no Uncontrolled factors.

I had to click Make Table and then click DoE Dialog in order to see the Design Evaluation. However, when I checked the factors, X8, X9, and X10 became "Continuous" instead of the original "Uncontrolled". Not sure why and how that impacts the Design Evaluation. The last picture below shows the color map of correlation obtained this way.

My main question is whether this design is a good design to tell which main effect(s) out of those 10 factors are significant without worrying about their confounding with each other and other two-way interactions. My plan is to use those significant Main Effects out of this screening DoE for my next optimization DoE using RSM. Thank you very much!

Re: Custom DoE screening design

JMP38401 — Tue, 27 Jul 2021 19:11:32 GMT

Just to add one thing, the correlation between X1 (HTC) and X2 (HTC) in the color map is 0.033, which sounds too big to me. Is there a suggested value for maximum correlation in order not to worry about the Aliasing? Thanks!

Re: Custom DoE screening design

Mark_Bailey — Tue, 27 Jul 2021 20:25:01 GMT

All of the design evaluation information is a function of the design matrix, but the columns for the uncontrolled factors are empty.

Re: Custom DoE screening design

Mark_Bailey — Tue, 27 Jul 2021 20:27:12 GMT

It is a matter of signal to noise. A very large signal (non-zero parameter estimate) will be estimated well even if there is correlation. The correlation does not bias the estimate or the model. It inflates the standard error of the estimate. I consider 0.033 to be very small.

Re: Custom DoE screening design

Mark_Bailey — Tue, 27 Jul 2021 20:32:43 GMT

@statman is guiding you very well. My only additional comment from 'the sidelines' is that you are thinking in terms of old design methods. The design principles never change but the methods continue to improve. Custom design is a new method. It permits strategies that were impossible just two decades ago. But with the new capabilities comes the responsibility to learn both the principles and the ways of the new method.

For example, uncertainty in the active effects is an issue from the start of statistical DOE a century ago. After regular fractional factorial designs appeared for the sake of economy in screening experiments, the method dictated the strategy. It resulted in confounded estimates (perfectly correlated estimates).

Play close attention to @statman's questions about variance or noise. These are key questions that a greedy experimenter must address in order to get the most information with the most confidence from the smallest design.

Re: Custom DoE screening design

JMP38401 — Tue, 27 Jul 2021 22:41:54 GMT

Sorry for giving a wrong number, the correlation value is 0.33 instead of 0.033. I would think 0.33 will be too big but not sure how small is good enough. Thanks

Re: Custom DoE screening design

Mark_Bailey — Wed, 28 Jul 2021 12:58:50 GMT

There is no absolute answer. It depends. The impact of correlation is variance inflation, which is bad. Perfect correlation (confounding) inflates the variance to infinity. But otherwise it depends on the variance that your start with. Small variance might tolerate a large correlation while a large variance will tolerate very little inflation.

Re: Custom DoE screening design

JMP38401 — Wed, 28 Jul 2021 20:09:45 GMT

Follow up with the Custom DoE for screening main effects, below is the model fit results. Is it safe to say the factors with t-ratio smaller than 2 can be eliminated and only those factors with t ratios larger than 2 will be used for the optimization design using a RSM next? If p value is available for each factor, I would think the factors with p value smaller than 0.05 should be included in the RSM optimization DoE next. Any suggestions about this thought process will be appreciated. Thanks!

Re: Custom DoE screening design

statman — Thu, 29 Jul 2021 22:47:58 GMT

@JMP38401 sorry for the delay...It is difficult to look only at the output you shared and draw conclusions. It looks like you have an un-replicated design and your model is saturated. If this is the case, you might want to try getting Normal Plots for statistical significance (sometimes you have to ignore Length's PSE line), Pareto Plots for practical significance and Bayes Plots if you're into Bayesian philosophy (Fit Model>Red Option Triangle next to response>Effect Screening). The first thing I always do is check for practical significance. Did you create variation of any practical value? What is the smallest increment of change in the response that you think is of scientific or engineering value? After you establish this value, plot the line on the Pareto plot and identify which factors had a practically significant effect. Then look for these effects on the Normal plot (Daniels Plot).

As far as optimization goes, it is extremely difficult to provide advise with the amount of context you've given. Please realize optimization is far from just a statistical design. It requires interpretation from someone who understands the science/engineering. A couple of thoughts though:

1. You are not trying to create some incredibly complex non-linear model that describes everything. Models are meant to be efficient approximations that are useful for prediction.

2. You should NOT be doing optimization of design factors unless you thoroughly understand noise.

3. What did you mean by RSM? G.E.P. Box implies this is sequential experimentation. It is not one central composite design.

4. You also should be thinking multivariate. Doesn't do any good to optimize one Y at the sacrifice of others.

Re: Custom DoE screening design

JMP38401 — Thu, 29 Jul 2021 23:51:09 GMT

Thank you for the suggestions! I tried to get the Normal plot but all three plot options under Effect Screening were grayed out and not available to show.

"The first thing I always do is check for practical significance. Did you create variation of any practical value? What is the smallest increment of change in the response that you think is of scientific or engineering value? After you establish this value, plot the line on the Pareto plot and identify which factors had a practically significant effect. Then look for these effects on the Normal plot (Daniels Plot)."

I can estimate the standard deviations for all the factors and know the practically meaningful minimum shift in response but not sure about your suggestion, "After you establish this value, plot the line on the Pareto plot and identify which factors had a practically significant effect. Then look for these effects on the Normal plot (Daniels Plot)." Can you please elaborate a little bit more on how to do it and why to do it? Thanks!

Re: Custom DoE screening design

JMP38401 — Fri, 30 Jul 2021 00:08:14 GMT

I guess any factors with an estimated effect larger than the practically meaningful minimum shift on the Pareto chart are significant and they should deviate from the normal line on the Normal plot as well. Should both plots give the same conclusion?

Re: Custom DoE screening design

statman — Fri, 30 Jul 2021 03:25:12 GMT

Again, I can't really help without the data table.

Re: Custom DoE screening design

statman — Fri, 30 Jul 2021 03:26:41 GMT

No, you can have effects show as significant on the Normal plot and yet not have any practical significance.

Re: Custom DoE screening design

JMP38401 — Sat, 31 Jul 2021 18:19:52 GMT

Questions about blocking for this design...

Since I will need multiple days to complete all the runs, I would like to block the design. However, since I have HTC factors in my design, I don't see the option for "group runs into random blocks of size". I can add a blocking factor into the factor list but I am not sure whether it is the right way to do it since the blocking factor (day) will be treated as fixed blocking.