topic Analysis of a Mixture DOE with stepwise regression in Discussions

Analysis of a Mixture DOE with stepwise regression

frankderuyck — Mon, 10 Jul 2023 08:28:24 GMT

In Mixture case studies (webcast & JMP documentation library) I notice that backward regression is used to analyse the results. Why is stepwise regression not used with forward regression or all possible models in regualr (non Pro) JMP? Is stepwise not possble because of the special nature of mixture models?

Re: Analysis of a Mixture DOE with stepwise regression

statman — Mon, 10 Jul 2023 13:49:55 GMT

Here are my thoughts, mostly philosophical in nature:

My first thought is to NOT do stepwise for experimental design, ever. The experiment is designed to investigate factors and a predicted model for those factors (e.g., first order, first order plus second order, etc.). The model should be predicted á priori. Stepwise is a procedure to uncover relationships that may not be known or predicted.

Mixture designs, for the most part, are optimization designs. That is, the components of the mixture are already known to be important and now you are trying to find the best "area" of the surface to operate in. You should already know the terms of the model and are optimizing those terms. There may also be a fair amount of collinearity in mixture designs which makes stepwise challenging.

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Mon, 10 Jul 2023 13:57:27 GMT

How then find the best model, backward regression? All possible models in stewise?...

Re: Analysis of a Mixture DOE with stepwise regression

statman — Mon, 10 Jul 2023 18:07:06 GMT

The approach depends greatly on where you are in the knowledge continuum (e.g., screening, optimizing, whether you are explaining or predicting). When experimenting in the screening phase, I do indeed recommend starting with the full model (saturated if possible) and removing insignificant terms from there (backwards or subtractive model building). When you are optimizing, the questions are a bit different (like, in general, mixture designs). You should already have a good idea of 1st and 2nd order linear and non-linear. Statistical significance is less important (it has already been established). You are "fine tuning" the level setting.

Re: Analysis of a Mixture DOE with stepwise regression

Victor_G — Mon, 10 Jul 2023 19:58:18 GMT

Hi @frankderuyck,

I agree with the "philosophical" point of view from @statman concerning Stepwise regression, more oriented for "non-designed" datasets in order to uncover some factors and effects in the absence of a-priori model. Traditional model selection techniques using p-values are not useful for mixtures : due to multicollinearity between estimated effects (aliases between effects), standard errors of the estimates are quite large, resulting in misleading and distorted p-values.

For (model-based) mixture designs, Forward Stepwise regression may be possible, but with safeguards and caution (for example, force main effects in the model and no intercept). Two options may be interesting to consider for Mixture designs, and highlighted by Dr. Philip J. Ramsey in one of his presentation, "Analysis Strategies for Constrained Mixture and Mixture Process Experiments Using JMP Pro 14" :

Traditional Forward selection using the pseudo factor method of Miller
Traditional Forward selection using fractionally weighted bootstrapping and auto-validation: SVEM in JMP Pro, Generalized Regression platform (Gotwalt and Ramsey)

Dr Ramsey does not recommend the use of "All Possible Models" for mixtures in part due to the need to force the pure component terms in every model. From a pragmatic and practical perspective, "All possible models" method can be highly demanding in terms of computations (and not very effective), as all possible models with various number of effects are constructed, no matter the hierarchy and heredity between effects. So there is a quite large portion of the models created that may be not very interesting (and relevant) to consider, but that are still created and evaluated in an agnostic and "brute-force" way.

Since there is an a-priori model assumed for model-based mixture designs, backward regression is safer to use with a validation method based on information criterion like AICc (Generalized Regression platform in JMP Pro, or Backward Stepwise in JMP).

Since you're more interested in predictive performance than in factors screening, you can also do "manually" the backward regression, by starting from the full model with all the supposed effects from the model (JMP, Standard Least Squares), and remove terms as long as it helps RMSE (prediction errors) of the model to decrease.

For model-agnostic mixture designs (like Space-Filling designs), the use of Machine Learning methods, efficient and effective in interpolation, may be very useful to build a predictive model on the homogeneously distributed points (but overfitting may happen quickly) : SVM, Neural Nets, k-Nearest Neighbors, Gaussian Process, ...

At the end, validation runs may be necessary in order to validate and estimate the predictive performances of the model.

I hope this additional response will help you,

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Mon, 10 Jul 2023 20:44:57 GMT

I saw in the webinars that the -logworth(p-value) pareto plot is used for the backward regression; if p-values are not suitable for mixtures is this correct? Note: I need to give a training to non-jmp pro users so I can't use generalized, machine learning..

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 07:16:20 GMT

I am analysing a three factor FLA 1, FLA 2 and FLA 3 mixture DOE based on a full cubic Scheffé model. Pushing the standard least squares buttallon and trying to remove low effect parameters using backward regression baseo on the logworht pareto (P-values for mixture analysis?) I get the result below: should I also remove all selected effect or only non-contained effects?

I also did the analysis with "All Possible Models" in Stepwise and get a very nice model based on AICC ranking, maybe a wrong model (will P-value based backward regression generate a right model...?) however very useful to specify optimal output Y settings!

I definitely agree that because of strong collinearity in mixture analysis a validation column is necessary and using generalized regression & actual machine learning tools tools is the right approach; I will have a close look at SVEM. As for my non-pro users DOE training I have to rely on classical methods; "All Possible Models" looks not too bad as shown above? Of course for more complex cases, with > 3 mixture components process effects yielding too many models, a seqential approach is definitely necessary; however, due to strong collinearity, making a good selection of parameters to proceed to next sequential step will be quite difficult.

Re: Analysis of a Mixture DOE with stepwise regression

Victor_G — Tue, 11 Jul 2023 11:20:44 GMT

Hi @frankderuyck,

No matter if your DoE is a factorial or mixture one, if there is an assumed model a-priori, you should respect the three principles behind Design of Experiments :

Effect sparsity: The effect sparsity principle actually refers to the idea that only a few effects in an experiment will be statistically significant. Most of the variation in the response is explained by a small number of effects, thus it is most likely that main effects (single factor) and two-factor interactions are the most significant responses in a factorial experiment. It is also called the Pareto principle.
Effect hierarchy: The hierarchy principle states that the higher degree of the interaction, the less likely the interaction will explain variation in the response. Therefore, main effects explain more variation than 2 factors interactions, 2FIs explain more variation than 3FIs, … so priority should be given to the estimation of lower order effects.
Effect heredity: Similar to genetic heredity, the effect heredity principle postulates that interaction terms may only be considered if the ordered terms preceding the interaction are significant. This principle has two possible implementations : strong or weak heredity. Strong heredity implies that an interaction term can be included in the model only if both of the corresponding main effects are present. Weak heredity implies that an interaction term is included in the model if at least one of the corresponding main effects is present.

In your case, when trying to remove these effects, you are in conflict with the effect heredity principle : you can't remove main effect FLA1 and interaction FLA2xFLA3 unless you already have removed all other effect terms containing these terms. So here you should keep them.

As stated before, creating models based solely on p-value for mixture designs may be very misleading.

The second option you tried with AICc (or BIC ?) criterion seems more reasonable, as AICc (or BIC) is an information criterion trying to "balance" the complexity of the model with its accuracy (the lower, the better).

Regarding your screenshots, it seems that models with higher number of terms may be useful in your case, as model number 9 has lower RMSE and lower BIC. It may also correct the residuals plot that seems to show some kind of heteroskedasticity (bigger residuals for higher flavour score).

Hope this supplementary comment will help you,

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 11:39:27 GMT

Hi Victor, I definitely agree on your heridity principles and that's why in this case backward regression based on p-values does not work here. So I used All Possible Models with forward AICC selection with very nice results. I am now working on another mixture case with a process variable and agian all possible models method generates a - maybe wrong - but very useful model! For non-pro JMP users I recommend using stepwise all possible model method for mixture DOE analysis.

Re: Analysis of a Mixture DOE with stepwise regression

statman — Tue, 11 Jul 2023 14:13:18 GMT

Hmmm, you have one or two mixture experiments and conclude "I recommend using stepwise all possible model method for mixture DOE analysis". I suggest you use caution with this recommendation or, as Danial puts it, you will have the failure of "premature generalization".

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 15:06:06 GMT

At least the models I get satisfy Cox pragmatic statement "Wrong but useful to meet my goal" What is a better alternative in regular, non JMP pro? Will these models be better?

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 15:11:55 GMT

For pro users I definitely recommend machine learning methods with validation

Re: Analysis of a Mixture DOE with stepwise regression

Mark_Bailey — Tue, 11 Jul 2023 16:35:20 GMT

My notion might have been expressed previously in this discussion, but I want to state it explicitly. First of all, we would not test the main effect parameter estimates. Why not? The constraint that the component proportions sum to one has ramifications beyond the mathematics of the linear model. An effect is a change in the response when a factor changes its level. The changes in the factor levels are independent in a non-mixture experiment, so the effect of changing a factor is uniquely determined in a properly designed experiment. It is impossible to change one mixture component independently from all the other components. So we cannot uniquely assign the effect to one component in a mixture experiment.

Second, we cannot test the main effect parameter estimates. JMP incorrectly tests the parameter estimate for the main effects against a null hypothesis that the parameter is zero. The main effect parameter is the mean response plus the mean change in the response of a pure blend so the null hypothesis is the mean response, not zero. Furthermore, the joint tests are impossible because the mean response is part of all the main effect parameters.

The true mean response in this simulated data set is 20. The true main effect parameters, which include the mean response, are 10, 20, and 30 for X1, X2, and X3, respectively. The parameters are significantly different from 0, but that is the wrong null hypothesis. Each of the main effect estimates is flagged with the note that the tests are confounded.

We don't ask about the main effects, but we can and should ask about the higher-order terms because they tell us about the shape of the surface and are testable.

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 17:24:08 GMT

Very interesting Marc, what mixture DOE result analysis method do you recommend for non JMP pro users? In my example above backward regression with the logworth pareto did not work; AICc forward with stewise all possible models worked fine.

Re: Analysis of a Mixture DOE with stepwise regression

Mark_Bailey — Tue, 11 Jul 2023 17:35:44 GMT

I do not recommend a particular method for selecting the linear model for a mixture experiment. I reiterate @statman's advice to consider all a priori knowledge while designing the experiment and again while analyzing the experiment.

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 20:01:00 GMT

This means that we should focus on interior points in the ternary plot trying to estimate the higher order terms? So a space filling DOE is a better option than trying to estimate a two way interaction model?

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Tue, 11 Jul 2023 20:03:48 GMT

I fully agree and that's what I did in my example above, did I use the right approach?

Re: Analysis of a Mixture DOE with stepwise regression

Victor_G — Tue, 11 Jul 2023 20:18:44 GMT

Just to clarify how the points are generated when using a mixture optimal design (with the case of 3 mixture factors) :
- Points at the vertices help estimate main effects parameters,
- Points at the middle of edges help estimate 2-factors interactions.
- Point in the centre is used for 3rd degree interaction.

Space-filling design create randomly and homogeneously distributed points in the experimental space, in the absence of any a-priori model.
If you already have an idea about the type of model you will use (like a two-way interaction model), an optimal design may be more useful and effective, as the generation of points will be optimized for the parameters estimation of the effect terms you have specified in your assumed model (unlike the "randomness" of space-filling design points generation).

Space-filling designs are useful in the absence of any a-priori knowledge about a possible model, with a probability of non-linear response surface, and have more flexibility regarding the modelling possibilities in case of points that are non-measurable (stability problems, very high or low values, ...).

Hope this clarify the difference between the use of these two types of methodologies.

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Wed, 12 Jul 2023 06:16:50 GMT

Suppose you are testing a mixture with 5 components and want to know which components have an effect on the response, how do you screen out these effective ingredients? Is this possible due to the confounding?

Re: Analysis of a Mixture DOE with stepwise regression

frankderuyck — Wed, 12 Jul 2023 07:02:17 GMT

I suppose that even with space filling designs screening will be very difficult?

Remark: I have a wide experience in non mixture DOE so my questions above may look naive for mixture experts. So far my conclusion is that mixture DOE requires experience and as analysis methods are less clear cut than non-mixture, for scientists it looks little artificial. I'm preparing for a lot of questions in a coming DOE training where formulations are important. Thanks all for you inputs, is a great help!