Why Include Replicates In DoE? (Design Comparison)

[#1]

Hi,

I understand the concept of what design replicates are supposed to do (Help with). Run replicates allow for understanding of general experimental error (Noise?).

However, when you compare custom designs generated with and without replicates (When maintaining the same total run number) you find that in most (if not all) cases, the design without replicates outperforms the design with replicates.

I created two Designs:

• Design 1 (Replicates)
  • #3 Factor, 2nd order Interaction and Quadratics
  • 13 Total Runs
  • 2 Repicates

• Design 2 (No Replicates)
  • #3 Factor, 2nd order Interaction and Quadratics
  • 13 Total Runs
  • 0 Repicates

When comparing the designs, evaluation indicates the following:

• Power Analysis: Design 2 > Design 1
  • Able to more readily detect quadratics
• Fraction of Design Space: Design 2 > Design 1
  • Prediction Variance is lower Across design space
• Correlations: Design 2 > Design 1
  • Lower correlation factor

1. Is there a quantifiable reason for choosing a design with replicates, over a design without replicates (When you limit your run #)?

Obviously performing at a higher run # (Design 2 + replicates) would lead to an improved design but if that's the case, why would I not just let JMP optimize for the best design again without replicates with the higher run number?

2. Is the concept of including replicates an artifact of classical designs where the newer, custom design platform can generally provide improved design algorithmically?

[#2]

Hi @MetaLizard62080,

I'm clearly not surprised by the situation and comparison you're showing. Let's start explaining why:

First, it's important to distinguish between Replicates and Replicate Runs. In JMP there are two options:

• In a "Classical design", JMP displays "Number of Replicates", meaning here the number of times to perform each run (so number of times to perform the whole design, in addition to the original runs/design). You also have this option in the "Augment" platform: Replicate a Design
• In a "Custom design" JMP displays "Number of Replicate Runs", meaning here the number of run(s) to repeat. You're using this option in your post.

Then, regarding your settings, you have designs with 3 factors, and an assumed Response Surface model (with all main effects, interactions and quadratic effects). For 3 factors, that means you have to estimate 10 terms: 1 intercept, 3 main effects, 3 two factor interactions and 3 quadratic effects.  So the minimum runs design should include at least 10 runs.

With the 3 extra runs you have, there are indeed 2 options:

1. Use these runs as replicate runs, to improve estimation of some effects and having the possibility to do a Lack of Fit Test.
2. Let the Custom design choose which other uns to add in the design, among the 27 unique combinations possible; since you assume a full RSM model, each factor can be seen at 3 levels, giving a number of unique factor combinations of 3^3 = 27.

The option 1 will enable to estimate some effects slightly better than without replicates. In your design, runs 2 and 5, and runs 6 and 11 are replicate runs. If you check the design comparison report, you can see that this design option 1 enable to have slightly lower relative std error of estimates for interaction effects (so a slightly higher than 1 relative estimation efficiency for these effects). Moreover, you'll have access to Lack of Fit test since you have replicate runs, which can help evaluate the adequacy of your model (among other visualizations like Actual by Predicted and residual analysis/visualization). 

The option 2 enable to test other factor combinations than the 10 absolutely required for the estimation of your model's effects. This enable to reduce correlations between terms, and enable to have a better coverage of your design space, thus reducing correlation between terms (Color Map on Correlations), and reducing globally the average variance prediction.

Concerning your questions, there is no definitive answer, as it depends on your needs (and habits):

• You might feel more confident to include replicate runs to be able to run the Lack of Fit test to evaluate the adequacy of your model, or include replicate runs to estimate more precisely specific effect terms. Note for this last point that A-Optimal designs may be a more suitable option, by defining through A- Optimality Parameter Weights the effects you want to estimate more precisely than others. See Why is the power of quadratic factors in DoE that low for more infos.
• You might want to reduce as much as possible correlations between effects to ensure a relatively unbiased estimation of these effects, so letting the Custom design choose the most relevant ones among all the combinations possible make sense.

For your second question, I think it depends on your activity domain, some domains always use replicate runs, or even replicates, as the variability of the systems is very high (biological systems for example). So it depends on your expected signal/noise ratio of your responses. In some industry, replicate/replicate runs are heavily recommended or enforced for the studies.

But on a general point of view, the Custom design will provide the optimal design related to your parameters, so if you don't have strong incentives to use replicate runs, I would let the algorithm choose the most informative runs.

Hope this answer will help you,

[#3]

Thanks for the response @Victor_G,

I work in the process development area of pharmaceuticals. I have used both types of design successfully (Where the model provides meaningful results that lead to process improvements). Because of this, I am unsure of what the "best" approach is. My personal perspective is that if we can trust JMP design eval, then I am inclined to go with option 2. @statman suggests (Rightfully) that this is indeed an assumption.

In general, many of my colleagues disagree completely with the concept that you can make statistical conclusions without run replicates. I believe this comes from the intuitive nature of what a replicate run provides and difficulty with the abstract ideas that JMP custom designer utilizes. To be clear, none of us are statisticians, and we have varying levels of understanding of statistical approaches (Ranging from significant interest and heavy use to a general fear/mistrust of statistics).

Out of curiosity, at the end of your statement, you suggest option 2 is the better option. Why is your gut reaction/habit to lean this way? 

[#4 · Accepted solution]

Hi @MetaLizard62080,

I completely understand your situation, and actually also feel in a similar situation (in terms of background too, I'm a chemical formulation engineer).

Some of my colleagues creating and recommending designs for laboratories also feel more confident recommending a design with replicates ; this way, it provides an easy and very visual way to assess pure error/ reproducibility of the experiments.

I do understand their points, but I'm not 100% confident with this situation, as even if we explain the rational behind replicates, some people do repeated measurements instead of replicate runs, to save some time and because they think replicating an experiment is a waste of time/materials. So at the end, what we might visualize/analyze is not pure error/total reproducibility of the experimental setup (with experimental error + measurement error), but only measurement error, which leads to false optimistic evaluation of pure error.

In the case of design without replicates (like design 2 in your case), using other combination runs for the extra runs enables to explore more combinations in the design space, increase power, reduce prediction variance in most of the design space, and if the experimental budget is sufficient for the topic and the model not too complex, these extra runs will enable to estimate approximately the error/variance. Concerning the evaluation of the model, not using replicate runs or replicates will prevent you from doing a Lack of Fit test, but there are enough tools and visualizations to assess the adequacy of a design. Also, in some design cases (like Definitive Screening Design), replicate runs can't be added, as it could lead to adding correlations between terms (like main effects and interactions) and some analysis methods won't be available (like Fit Definitive Screening), as the design structure is changed.

So at the end it's more a question of assumptions and habits, and a question of knowledge about the experimental setup and measurement system. I tend to prefer recommending a sequential approach with Scoping designs (preliminary tests) to assess the variability, curvature and reproducibility of the setup, before recommending a design fully exploring the topic (in the limit of the experimental budget).

No "good" answer on this topic, but I hope this answer will help you determine what's best for your use cases, 

[#5]

I have a couple of comments you might want to disregard.  First, be careful putting too much weight in the design evaluation statistics. These are all á priori for potentially unknown reality.  Second, there is more than one way to run replicates.  If you use RCBD or BIB, you can simultaneously increase inference space while improving design precision.