Which alias terms to include in fit model

[#1]

Good Day

I have performed a main effects screening design experiment to determine which of my factors have the most significant effect on the mechanical properties of a material. My factors were:

L -> 5 level discrete numeric

R -> 2 level categorical

S -> 2 level categorical

T -> 2 level categorical

I used the default number of runs and added one replicate run as this was suggested by a colleague and tried my best to minimize correlation values when using the design evaluation tools, but had to keep the runs at a minimum as I did not have a lot of material to work with. The two factor interactions were automatically excluded from the design if I remember correctly and were added as alias terms.  

When I now run my analysis after entering my experimental data using the fit model platform and effect screening, the effects which are significant vary depending on which alias terms I include in the fit model platform. I also cannot include all the two factor interactions (e.g L I can only include in a two factor interaction term twice) as most of the effects are just not shown in the resulting effect summary then. Does this have something to do with the degrees of freedom in the design? Is it correct to try and include as many of the two factor interaction terms as possible with priority given to the ones with highest correlation with main effects in the alias matrix? In my alias matrix the 4 highest correlations are as follows:

L*(T*R) - 0.69

L*(T*S) - 0.58

L*(S*R) - 0.58

R*(T*L) - 0.56

where the factors in brackets are the two factor interaction term partially aliased with the main effect in front of the bracket. 4 more correlations follow these having values around 0.4 and the rest of the design includes terms with very low correlation values.

Are these alias terms the reason why the effect L pops up as being the only significant factor (p<0.05) in most cases when I include different two factor interactions?  Should I include the 4 listed above (with the highest correlations) as my best bet at finding which factors are significant if there are any? Also do I need to include two factor interactions in the fit model analysis at all?

[#2]

I'm a bit confused , so here are some questions/comments:

1. If you are running a screening design, why are you testing a factor at 5 levels?  This would allow you to estimate a 4th order polynomial which would be more likely associated with an optimization design.

2. I assume you used Custom Design and added the model terms you were most interested in?

3. I don't understand what you mean by "The two factor interactions were automatically excluded from the design if I remember correctly and were added as alias terms."  You can't have terms that are aliased in the fit model or you will get singularity.

4. I also don't understand "added one replicate run"?  Did you do this to get 1 DF for error for your statistical test?  How did you choose which treatment to replicate?  Realize the statistical test MSfactor/MSerror (F-test) assumes the errors are unbiased, randomly distributed and hopefully REPRESENTATIVE of the true random errors (not sure how you get this with 1 replicated treatment?).  And the residuals meet NID(0,variance) assumptions.

5. Each coefficient in the model is conditional and its magnitude, and sometimes the sign, change.  They depend upon the other variables in the model (and noise).

6. Don't understand "priority given to the ones with highest correlation with main effects in the alias matrix".  If your model variables are correlated this is multicollinearity.  You don't want this.  Make sure you check VIFs in the parameter estimates table.

Perhaps attach your JMP table and the community can have a look.

[#3]

Hi and thank you for taking the time to respond to my questions!

1 and 2): I was referred to a colleague for help who has experience in experimental design. I started off with a custom design but was told to rather do a classical screening design as I am trying to find out which factors have the most significant effect on my material mechanical properties so that we can take those and focus on them in further research.

3 and 6): I meant that during creation of the screening design only the main effects were included in the model and the two factor interactions were added as alias terms. After I gathered my research results and I had to analyze using the fit model platform I was confused about whether I should add alias terms into the fit model section before I run the analysis as the alias matrix and the colour map showed partial aliasing and correlation of some main effects and two factor interactions but not between the main effects themselves. I understand the alias matrix to reveal which main effects may be biased by certain two factor interactions? and from what I have read I thought that these two factor interactions should then be included in the fit model section? If I just run the fit model analysis without adding alias terms and get my results, what should I look at to know whether the main effects indicated as significant are really significant? I am not familiar with singularity but will go and read up on this.

4): I was told to manually add one replicate run to the screening design after the table was generated but am not sure how this helps.

I am attaching my JMP design table here.

[#4]

My first comments are, I don't understand:

• the process being experimented on (did you create a process map of the experiment?),
• the factors nor the hypotheses that led you to experiment on these factors (do you have a thought map),
• the response variables nor how much change in the response variables if of practical value,
• how the experiment was executed (did you keep track of run order?  Was it run in random order?),
• how you were handling all of the other variables that changed during the course of the experiment (that were not explicitly manipulated).

So my answers may be less than complete.

1. What you have attached is not what I would call a classical screening design as you have a factor (L) at more than 2 levels.  Since you did this, you have created a rather complex aliasing structure.  If you had stuck to 4 factors at 2-levels, you would have a straight forward alias structure resolution IV design in 8 treatments.  I don't understand why you would test just one factor at 5 levels.  This biases the results as there is more information regarding factor L than others in the study (4 degrees of freedom for L).

2. I still don't understand your use of the word alias "only the main effects were included in the model and the two factor interactions were added as alias terms".  You don't add terms that are aliased to the same model or the resultant analysis will invoke a singularity warning in JMP. In the table you provided, you have a total of 10 DF. Each 2-level factor is 1 DF and there are 4 DF for L (1 for linear, 1 for quadratic, 1 for cubic and 1 for quartic). The non-linear terms are partially aliased with 2nd order (and higher) linear effects.  Since L is being considered a continuous variable compared to the others in the model, this may create a mixed model scenario (Treated as a random effect vs. the other factors that are fixed effects).

There are multiple approaches to analysis...some do it additively by starting with 1 term and add terms terms to the model (stepwise).  Some prefer to start with a saturated model and remove terms deemed insignificant.  Both of these require an understanding and ability to interpret: R-square, R-square adjusted (and the delta between the 2), p-values, VIF's (for unbalanced designs), MS error and whether this is representative of the true error, practical significance, etc.  Realize when you leave a term out of the model it ends up in the error term and then becomes part of the MSerror estimate impacting the F-test. Removing insignificant terms from the model further biases the MSerror term (making p-values look even more significant). Daniel was the first to use normal plots to perform statistical tests unbiased by the MSerror term.

3. Adding 1 replicate run does little to provide a representative, unbiased estimate of the error, IMHO.  And which run was selected to replicate? Randomized replicates certainly has been used/recommended for many years to get an unbiased estimate of the error.  There are, perhaps, more efficient and effectives means of understanding the error, how representative it is of future conditions and therefore whether it is a good basis for a statistical test.

[#5]

Thank you once again for your swift response!

Regarding your bullet point comments: My experiment is investigating the effect of additives (two of the factors) as well as 3D printing settings (two factors) on 3D printed parts created with my material. Yes, I have done a thorough literature search and have sat down with my supervisor to discuss which variables at which levels I should include. Everything was well mapped out with sufficient data to back up decisions. All other settings could be controlled at default settings and those such as environmental changes during tests which may vary and are not included in the model were still measured to ensure that they stay more or less constant and do not influence the results significantly. 

1): I would have liked for factor L to be a 2 level factor however this was not possible as it represents the % additive in the material and for the specific additive we use a 1% difference in % additive between 0 and 10 % could have quite a large effect on the resulting material strength according to several literature studies. So my supervisor wanted me to include more than 2 levels.

2) As for the alias terms. When I create the screening design I only add main effects to the model and no alias terms/matrix is present under the design evaluation. However, as soon as I select the "Make Table" option and then go into the design evaluation again the two-factor interactions are suddenly listed under an "alias terms" tab and an alias matrix appears. Could this be because I am using the classic screening design option and then JMP picks up my 5 level factor when the table is created?

Thank you for your description of why my L factor is biased and how the error term is influenced - this has helped a lot!

3) The replicate run was chosen randomly by me and then manually added as the screening design does not allow an option for adding replicates. Would you suggest I perhaps add more random replicate runs to my design to try and get a more unbiased error estimate?

Thank you!

[#6]

Additional comments:

regarding your statement: "All other settings could be controlled at default settings and those such as environmental changes during tests which may vary and are not included in the model were still measured to ensure that they stay more or less constant and do not influence the results significantly. "

Having the settings controlled or held constant is NOT what you should be doing as this will result in an experiment run in an inference space that is likely not typical of future conditions.  Therefore your results are limited to those conditions.  Have you studied measurement system error?  For the additives, are they from one batch or multiple batches?  Is there any batch-to-batch variation?  You should have a strategy to handle those noise variables. To quote a great experimenter:

"Block what you can, randomize what you cannot" G.E.P. Box 

If you can identify all of the noise variables, there are excellent strategies available to partition and assign those sources of variation (e.g., BCBD, BIB, split-plots, repeats, covariates, etc.).  Holding them constant is only a good idea when you intend to do that forever.  If you cannot identify the noise variables, use randomization to get, hopefully, unbiased, representative estimates of the combination of all of those noise variables.  This estimate can subsequently be used for statistical tests.

I doubt that one treatment replicated will provide you with a reasonable estimate of noise, but that's just my experience.

I'm not sure I understand your first bullet point?  If there are constraints on the level setting for those variables, you might consider a mixture design.  The point of YOUR experiment is to validate the studies done in literature (which were done in a completely different design space) and provide insight into your hypotheses.  It is OK for factors to show their effects, large or small and also in relation to other factors in the study.  Your argument provides no justification for more than 2-levels (because your supervisors wants...)

Regarding your second bullet point, you have created a fractional factorial.  You do this by aliasing higher order effects with 1st order effects.  This true for any fractional design.  If you alias the 1st order effect with a 2nd order effect, you have Res. III design, If you alias 1st order effect with 3rd order, Res. IV etc.  The aliasing is a result of your wanting to economize on treatments as you are suspicious you are not at the optimum design space yet and you're not sure what factors have the greatest effect.  You can decide what model to run in JMP.  There are several schools of thought to accomplish this.  I tend to start with a saturated model and remove terms that are unimportant.  Others start with 1 variables and add variables sequentially (stepwise).  There is no right way, but you do need to know how to evaluate the models you build and JMP provides a plethora of statistics to help with this.

[#7]

Regarding the additional comments. You make a valid remark in saying that all settings are not always going to stay constant. However, currently in the 3D printing research field there is no ASTM/ISO standard that exists which one can follow when printing. Developing this standard would as you said not be allowed to assume that certain settings remain constant whilst others are chosen as variables. However, since there are so many settings that can be changed/investigated, most researchers (in literature) typically choose a few settings to vary whilst keeping other settings constant as I have done with mine. This is still very much a growing research field.

I have used the materials from the same batch throughout. It should be possible for me to identify some noise variables. If I can identify them: To use the strategies you mentioned (BCBD, spli-plot etc), were the noise variables supposed to be included in the model before I created my design or can I use these strategies now even after I have already done my experiments using my current design?

Response to your second paragraph: If I were to for instance do a mixture design instead, would it be possible that similar runs to my current runs could be generated so that I would not have to re-do all of my experiments? I have limited time and remaining resources (materials) and therefore I am just trying to find the best way possible of analyzing my results. If it is not possible to create a better design without having to re-do all runs then all of the issues you have brought up I can still include in my discussion on my experimental design.

Response to third paragraph: "You can decide what model to run" - were you referring here to when I was creating my design i.e. before generating the design table or when analyzing the results in the fit model platform?

Thank you!

[#8]

ASTM and ISO standards are rarely (if ever) done to provide optimum processes.  They are the result of collaboration between many individuals and organizations and often are negotiated.  Typically they are the minimally accepted criterion.

Even though settings remain constant, it does not mean the process does not vary!  For example, you might set the mold temperature of an injection molding process at a specified temperature.  But as the process runs, and hot material is injected into the mold, the mold temperature increases.  This may be countered by running a coolant through the mold, but all of this contributes to variation in the actual mold temperature.

The number of variables is not a restriction on your investigation.  Admittedly, you might be limited by resources, but there is no limit to the number of variables that can be investigated using sequential directed sampling and experimentation. The principles of Scarcity, Hierarchy and Heredity hold true.  A fundamental truth: In order to learn about the effect of a factor, that factor must vary in the study.  If it is constant in the study, there is no way to estimate its effect.  So, since you did your entire study with one batch, you have no data to support your conclusions will hold true when the batch changes (which it will eventually).

Strategies to handle noise in experiments should be planned and established prior to running the experiment.  There are times when we can take advantage of some strategies after the first experiment is run, but these are less than optimal.  For example, you can consider the first replicate of your experiment Block 1 and then replicate the entire experiment again for the 2nd Block, but you will need to be very mindful of the noise that was constant within block and make sure that noise changes between block.

There are times you can salvage runs for use in other situations, but you must always be aware of the "block" effect.  Not sure what you mean by "I am just trying to find the best way possible of analyzing my results".  

Yes, you decide what model to enter into JMP for analysis (fit model).  This should be done prior to designing the experiment so that you ensure the effects of most interest can be estimated by your experiment, but after you have the data you still choose what to include, or not, in the model for analysis.