How to use sample size and power for factorial DoE to determine number of replicated treatments

[#1]

I'm trying to learn how to operate JMP's dialog boxes for sample size for use in experimental design. I'd like to understand how to determine the number of replicates necessary in a factorial experiment.  I'd like to know how many times to run each treatment combination.  

Assume there is no measurement error, no need for blocking or centerpoints, the response variable is continuous and everything is exactly as needed to meet all the assumptions for classic calculation of sample size (random stationary homoskedastic normally distributed errors, etc.).  

Suppose an experiment will have two factors at two levels, full factorial.  Thus, there are four treatment combinations.  I want to estimate the two main effects and the one interaction.  What are the steps to determine the number of replications of each treatment combination?

The same question applies for fractional factorial.  If I have 15 factors each at two levels and I want to run a fractional factorial of 16 treatment combinations, how do I determine the number of replicates of each treatment?

For the sake of sample data, let alpha = 0.05, beta = 0.05, sigma = 1, delta = 1.

I've used "Two Sample Means" dialog. There the static text says "Sample size is the total sample size; per group would be n/2".  If I use this calculator for factorial design, what's the number of replicates per treatment combination for full or fractional factorial?  Do I just divide the total sample size by the number of treatment combinations?

Now suppose I want a centerpoint in the 2^2 experiment.  That's five treatment combinations.  Assume I'll replicate all five treatments.  To design an experiment to test the significance of the centerpoint effect, I'll assume that effect is equal to delta.  How do I use the JMP dialogs to get the number of replicates per treatment?  Do I divide the "Two Sample Means" sample size by five?

If I want replication only at the centerpoint and no replication of the corner points, I'll assume the standard deviation that is revealed at the centerpoint is constant throughout the inference space.  The goal is still to estimate the significance of the main effects, the interaction and the centerpoint.  Assuming the same delta, how do I use the dialogs to get the number of replicates of the centerpoint combination?  Do I divide the "Two Sample Means" sample size by one?

I'm aware of the custom design platform and its power analysis and know how to operate it.  I would like to use DOE > Design Diagnostics > Sample Size and Power.  After all, those calculators are in the "DOE" menu, I'm trying to figure out how to use them for DoE.

[#2]

The sample calculators in the DOE menu are meant for simple cases with at most one factor. The power analysis is in included in the DOE platforms. the process is to generate the design, then evaluate the power. You can save the design in a data table and the compare the power, and other attribute, across more than one design.

See this section in JMP Help for explanations about how to perform a power analysis of a design. I see that you are familiar with these tools, though.

You can use Make Table from Full Factorial with the desired replication and number of center points, then select DOE > Design Diagnostics > Evaluate Design to get the Power Analysis.

[#3]

The proposed approach is a trial-and-error way of designing an experiment (pick a design, choose a number of replicates, see what the power is, iterate until happy with the balance of power with sample size).  Is this really necessary?  It's absolutely possible to do what I'm requesting by hand (or with Minitab) as a way of designing an experiment rather than evaluating an experiment.

[#4]

I apologize in advance for the following comments/questions:

What is the purpose of replication?  I believe there are several purposes:

1. To increase the inference space of the study,  

2. To make the study more representative of future conditions (reality),

3. To determine if the effects of factors studied under one set of conditions are consistent when those conditions change,

4. Robust design: identify noise-by-factor interactions early in design to take action early early to reduce the impact of those noise effects...

So to me, replication is done to ensure the experiment is representative of future conditions so I increase my confidence the results can be extrapolated into the future.

 “Unfortunately, future experiments (future trials, tomorrow’s production) will be affected by environmental conditions (temperature, materials, people) different from those that affect this experiment…It is only by knowledge of the subject matter, possibly aided by further experiments  (italics added) to cover a wider range of conditions, that one may decide, with a risk of being wrong, whether the environmental conditions of the future will be near enough the same as those of today to permit use of results in hand.”

Dr. Deming

That being said, why would you rely on a statistical formula to provide guidance as to whether the experiment is representative of reality?

[#5]

This is a different issue for a different discussion, it's doesn't address the original question.

[#6]

5. To gain degrees of freedom to estimate experimental error and then calculate statistical significance of the factor effects.

[#7]

You can only calculate the prospective power of a given design. It is a function of the model matrix, which, in turn, is a function of the design matrix. Without a design, you cannot compute the power.

Selecting the 'best' design is iterative, using 'trial and error,' but it is also practical by developing a small set of designs that span the choices in which you vary the complexity as you intended (e.g., number of runs, terms in the model, number of required center points, et cetera). Then you can compare them all simultaneously using DOE > Design Diagnostics > Compare Designs to narrow down the choice. It is a pretty quick process.

The choice is also 'good enough' because your assumptions that are used in the power analysis are wrong but they are good enough to get the job done (select a reasonable design).

How would you calculate the design by hand or with MINITAB, given the desired power? I think that the answer must rely on the special nature of a rigid factorial design.

[#8]

It's true that we can't calculate the power without specifying the design.  However, specifying the design can be done with a simple dialog and doesn't require actually generating a design matrix.  It just needs to give enough information to know how many total degrees of freedom are in the experiment and how many will be used to model the effects.  The remainder go to error, and replicates increase the number of degrees of freedom for error.  

This is not a statistical challenge, this is a user interface question.  I'm just looking for a productivity tool to simply answering the question, "how many replicates do I need in this particular experimental design in order to be 95% powerful?"   I want to specify power and let a computer calculate the sample size.

JMP has a dialog box for a two-sample test, Minitab has that and also has a dialog box for a factorial experiment.  Enter number of factors, number of corner points, number of center points, alpha, power, delta, sigma.  It will calculate the number of replications of each treatment.

For example, in Minitab I specify two factors, four corner points, delta = 1, power = 0.95, no center points, sigma = 1, alpha = 0.05. The answer is 14 replicates for each of the four treatments, for 56 total runs.

If I enter 15 factors, 16 corner points, delta = 1, power = 0.95, 0 center points, sigma = 1, alpha = 0.05, then the answer is 4 replicates for 64 runs total.  

To get N by hand is tedious and iterative because the number of degrees of freedom for error is what we're trying to solve for and there is not a closed-form equation for it.  It arises from the F value.  However, it can be done in a few cells in Excel.  It's still trial and error to get the answer, but this trial and error is less tedious than diddling around with a bunch of dialogs and buttons for creating replicated matrices and seeing the resulting power.

[#9]

I just noticed a JMP support topic saying that there is a command called DOE > Sample Size Explorers > Power > Power for ANOVA.  It seems to exists in version 16.  If this is the solution to my problem, then I need to try version 16!

[#10]

It turns out that the sample size explorer for ANOVA is not the answer for this DoE question.  That would be useful to determine a sample size for the situation of a nested ANOVA to detect variation between subgroups in excess of variation within subgroups.  Itʻs not straightforward to connect that to DoE and determining sample sizes for factor effects.

However, the Sample Size calculator for Two Means can do what Minitab does, meaning you can input an effect size and a power and it will calculate a sample size.  The result matches the result from Evaluate Design (where you guess in advance which sample size might work and see how powerful it is).  Hereʻs how to do it:

Let r = the number of treatment combinations in a factorial experiment.  Use the Sample Size for Two Means and enter number of extra parameters = r - 2.  Enter alpha, sigma, delta and power, then let it calculate the Sample Size.  Then, the number of readings per treatment combination is Sample Size divided by (Sample Size / r).

The extra parameters are the degrees of freedom that are taken from the error term by terms in the factorial model.  The sample size analysis already has two terms (the mean and the effect in the two sample analysis so r = 2) so you need to specify how many more to subtract.