Lack of fit for a custom DoE design with hard to change factors

[#1]

When I use Minitab to build a DoE model with two hard to change factors, I can see Lack of fit evaluation as long as I replicate both whole plot and subplot (Minitab gives me an option to replicate the whole plot and subplot during the design of the model)

However, when I use JMP Custom DoE platform, for the same set of factors with two HTC factors, I am not able to see the lack of fit analysis no matter what I do. I assume that is because the whole plot and the subplot were not replicated in my model but am not able to find a way to replicate both the whole plot and the subplot. 

[#2]

Hi there.

I have done a small amount of exploration of this. I don't know if I have an answer for you but this is what I found out:

1. As far as I can see, Custom Design does not allow you to specify replicates (i.e. centre points) when you have hard-to-change factors. This makes sense to me. I think the idea of replicate runs when you have hard-to-change factors, and therefore whole plots modelled as random effects, seems like a bad idea.
2. Even if you manually add replicate runs to your design, you will not see a lack-of-fit report when you model using the recommend REML method. If you switch to EMS (not recommended) you will see the lack-of-fit. There might be technical reasons why this is the case. But again this just seems to make sense. What does lack-of-fit mean when you have random effects?

In summary, you can achieve lack-of-fit tests for split plot designs if you create the replicates manually and use the EMS fitting method. But I would think hard first about whether you really want to do this.

I think it would be fair to say that the JMP DOE people do not recommend replicate runs, so the inability to add or use replicates in split plot designs in JMP might be a reflection of that. Replicated centre points is part of an old DOE methodology from the days before we had easy-to-use software, when experiment analysis was done by hand. In most experimentation, runs are expensive so replicates are wasteful. Better to use all runs to explore the different parts of the factor space. This is really the idea behind Custom Design.

If you still think that you should have the ability to add replicates for lack-of-fit in split plot designs you can add your request to the Wish List.

I hope that is helpful.

Phil

[#3]

My thoughts regarding some of Phil's ideas (which in general I agree with):

1. HTC factors should not be considered random effects (not sure where this comes from).  In order to get a formal statistical test of an HTC which is in the whole plot (1 factor), you compare the effect of the WP factor to the WP-by-replicate interaction (this is EMS).  Of course, you can determine practical significance without a formal statistical test.

2. I'm not sure why Phil is suggesting replicates are equal to center points "(i.e., centerpoints)"?, but perhaps I am misinterpreting him.

3. The issue is this...HTC factors will likely not be used to adjust a process.  We want to know if the HTC matters and perhaps what might be the "best" level to run at, but not manage the process with them.  So it makes sense to restrict them.  In fact, you might be inflating the noise in the experiment (decreasing the precision) by changing the HTC multiple times and introducing more noise than is normal.

4. There is a limit, statistically speaking, to the effectiveness of designs with multiple HTC's.

5. Replicated center points randomly run throughout the design space is actually a great idea to get an understanding of the consistency of noise or identify special causes in the noise.  It might offer a realistic estimate of the true MSE.  Not sure that some of the "old" ideas aren't actually quite good, but I guess I'm old.

6. To analyze split-plots I prefer Daniel/Box methods.  That is to create Normal, Pareto, Bayes plots of each plot.  This ensures you are comparing the whole plot to the noise associated with the WP and the sub plot factors to the noise associated with the sub plot.  This prevents MSE bias and the issues created by lack-of-fit decisions.

[#4]

I am not suggesting that centre points are the same as replicates. I was referring to the ability in JMP Custom Design to add centre-points, which will ensure you have replicates. This option is not there when you have hard-to-change factors. I hope that is clear, if it was not before, and is helpful in understanding how you can achieve your goals with JMP.

[#5]

I would recommend Optimal Design of Experiments by Goos and Jones for a discussion of why you should model experiments with hard-to-change factors using REML and random effects. And the problems with EMS. They (especially Peter Goos) are really the experts on split plot designs.

[#6]

Good suggestion.  I'm not sure what makes an "expert"?  Admittedly, I prefer a practical and graphical approach to analysis vs. quantitative analysis as these are easier to "show the results" to non-statisticians (engineers, scientists and managers), which is whom I deal with most of the time.  Last time I used the acronym REML to a non-statistician, they had quite the quizzical look on their face.  BTW, last time I checked, Minitab does not use REML, but I could be wrong.

I recommend you read :

Daniel, Cuthbert (1976) “Applications of Statistics to Industrial Experiments” Wiley (ISBN 0-471-19469-7)

Box, G.E.P., Stephen Jones (1992), “Split-plot designs for robust product experimentation”, Journal of Applied Statistics, Vol. 19, No. 1

Jones, Bradley, Christopher J. Nachtsheim (2009) “Split-Plot Designs: What, Why, and How”, Journal of Quality Technology, Vol. 41, No. 4, pp. 340-361

Bisgaard, Søren, (2000), “The Design and Analysis of 2 ^k-p X 2 ^q-r Split Plot Experiments”, Journal of Quality Technology, Vol. 32, No. 1, January

Bisgaard, Søren, Murat Kulahei, (2001), “Robust Product Design: Saving Trials with Split-Plot Confounding”, Quality Engineering, 13(3), 525-530

[#7]

Agreed. Practical and graphical. Keep it as simple as possible. Avoid talking about REML!