How do I know if my DOE Custom Design is balanced?

[#1]

Hi!

I am using Custom Design to generate a DOE design. I was wondering how can I make sure that my design is balanced? The course I took on Custom Design of Experiments mentioned that this type of design is by default balanced, but I want to make sure I didn't mess anything up in my design and it is balanced. I didn't quite understand the "how to know if its balanced" part. Does design balance have anything to do with the final number of runs?

Thank you in advance!!

[#2]

I may not understand your question, but here are my thoughts/questions:

Balanced with respect to what (factor levels, blocks,)? Based on what criterion? Do you mean orthogonal?  

The number of runs will not answer the question in and of itself.

You can evaluate the design multiple ways:

https://www.jmp.com/support/help/en/17.0/?os=mac&source=application#page/jmp/design.shtml#ww286656

You might want to look at the color map on correlations.

[#3 · Accepted solution]

The design principle of balance may or may not be invoked when using the custom design platform...if you use the custom design platform to generate, say a 2**k full or 2**(k-p) fractional factorial design then the design will be balanced. But say you use the custom design platform to generate a d - optimal design, including replication, with disallowed treatment combinations and a non power of 2 number of runs, chances are very good the design will not be balanced. So the answer is 'it depends'. The main intent of the custom design platform is to offer a model driven, optimality criteria centric, flexible tool to generate designs that fit a variety of often encountered issues, like disallowed treatment combinations, specific number of runs, blocking (split plot and otherwise), replication, etc. So the property of 'balance' may not be achievable given these issues. But all is not lost...your design now fits your problem and constraints...not a design condition, balance. I hope this make sense?

Your last question? Yes in classic DOE methods. In optimal DOE methods? No.

[#4 · Accepted solution]

Hi @ADouyon,

Just to summarize and complete the excellent answers above :

Your DoE is balanced when for any factors, you have the same number of runs done at high and low levels.

This was the classical way to do DoE (before the rise and progress made on computer calculations), because of :

1. The simplicity of the factorial design created : it could be generated by hand, and calculations are also realized without difficulties (by hand or with Excel/calculator),
2. Nice orthogonality properties, meaning factors are studied and analyzed independently from each others (no correlations),
3. Their practical efficiency with a long history of success stories, which is why people tend to stick to it, because balanced designs are here for a long time, in opposition to custom optimal designs which are more recent.

This "classical" approach of balanced design creation can be found for classical designs like Plackett-Burman screening designs (or Hadamard matrices), full or fractional factorial designs, and response surface designs (Box Behnken or Central Composite Design).

But as stated by @P_Bartell, I wouldn't attach too much importance now to the evaluation of balance when creating a design, particularly in custom designs : the designs calculated for optimal designs are flexible, handle constraints, disallowed combinations and a large combination of different factor types, and are (as the name suggest) optimal for a certain criterion (D, I or A).
So as long as your topic/problem is clear, the factors and ranges well identified, constraints (if any) entered, and model terms entered, you won't do any mistake with an optimal design (from the "Custom Design" platform).

Don't hesitate to generate several designs with different runs number, center points, replicate runs, ... in order to compare them and choose the most appropriate one depending on your goal, expected precision and experimental budget.

[#5]

Thank you all @statman, @P_Bartell and @Victor_G for your input! Much appreciated!!

Following up on that same topic, does it matter then if I have an even or an odd number of runs in my Custom DoE? Will Custom DoE always try to have an even total number of runs by default? if one adds, lets say 5 replicate runs (odd number) to the design, is this not ideal or it depends? if so, what does it depend on?

Thank you all!
Best,

[#6 · Accepted solution]

Hi @ADouyon,

I wouldn't care too much about knowing if the number of runs is odd or even, but I would be more interested to know how adding runs (to the minimum number of runs or the default one proposed by JMP in the Custom Design platform) to a design improve some of its properties (power, aliasing, predictive variance, depending on which properties are the most relevant to my topic and goal).

So to answer your question(s), I'm not sure there is a general theory for "optimizing" custom/optimal designs, because of the diversity of constraints, types of models, factors possible... Adding 5 replicate runs to the runs recommended by the design could only improve the precision of parameters estimates (and probably decrease variance in the experimental space), but if the 5 replicate runs are required in the platform instead of other runs that could be generated by JMP, it might not always help (for example, the 5 replicate runs could "take the place" of 5 other runs that could have better explored the experimental space, depending on the design and optimality criterion). The easiest and most practical way is to generate a small amount of designs, and to compare them to know which one is the most relevant for your specific topic.

Hope this answer will help you, despite being a bit "vague",

[#7 · Accepted solution]

To illustrate the absence of big theory behind optimal designs on specific case, here is a simulation I have done with the Design Explorer (the new feature coming in JMP 17).
It's a trial I have done on a concrete case with 4 factors and specific factor types, for optimal designs (D, A and I-optimality criterion chosen), with the number of runs from 19 to 21 (reference is 20 D-optimal designs with no center points and no replicates, marker with a grey star in the graph), different center points (0 to 3) and different replicate runs (0 to 1). Evaluation is done on D and I efficiencies relative to the reference design.

Here, run size might be interesting to look at, but perhaps more importantly replicate runs and centre points number. There seems to be no clear indication that odd or even number of runs would be preferable.
Depending on the experimental budget and goal (D, I and/or A efficiency), running different designs and visualizing them may help identify most promising designs, and evaluate how much you may increase or decrease the performance of your design, depending on the number of runs and other parameters chosen.

[#8]

Your line of questioning leads me to think you might need to develop a set of design selection criteria for yourself.  Perhaps the following is a place to start:

1. Constraints: Time, money, material availability, measurement/equipment capability, etc. How many treatments canbe made (this will likely need to be negotiated)?
2. How many factors are to be manipulated (the number of hypotheses to be compared)?
3. How will noise be managed or partitioned?
4. Are some factors harder to change than others?  Are there other restrictions on randomization?
5. What are the prioritized effects you want to estimate?

   EFFECT

   Noise

   Main Effects

   Two-factor Interactions

   Noise-by-factor Interactions

   Stability (special/common)

   Simple Curvature (quadratic)

   Complex non-linear (≥cubic)

   ≥3rd order linear effects

   Leverage (sets of x’s)

   Measurement Uncertainty

   Mean/Variation

6. Are higher order effects suspected/predicted (e.g., interactions, curvature)?

• What is the desired resolution? (What effects do you want to estimate/separate?)
• What order polynomial is necessary?

The question of adding 5 replicates or any additional runs is what will you be estimating with those additional runs.  What x's vary between and within replicates?  How likely will the additional runs improve how the experiment results will be useful in the future (how representative of future conditions is the experiment?). How well will those replicates estimate the true random errors in the process?  Could you use those resources more efficiently?

[#9]

Balance in a design was important 100 years ago when the factorial design was first presented as a method to design an experiment based on statistical theory. Why? Because 'computers' were people. 'Computer' was a job title. Manual computation was feasible when the design was balanced. It simplified to the difference between the means using the estimating column in the model matrix. We don't compute estimates by hand anymore. We use modern regression techniques. The custom design maximizes the information to be extracted from the regression model based on one of several choices for the criterion. Balance is not a consideration. @statman provides a much better way to think about it than asking if the design is balanced.

(BTW, achieving balance is more complicated than described by @Victor_G above. It requires column-wise balance.)

[#10]

Thank you all @Victor_G, @statman and @Mark_Bailey for the very helpful clarifications and various perspectives!! Much, much appreciated! thank you!