Doe and power

[#1]

Hi,
When planing an experiment and looking at the design and power for each parameter (at the initial design step), there are is an option to change the RMSE. By changing it, obviously the power is changed. The question is how we can predict or assume this parameter ahead (in screening experiment for example)

Thanks

[#2]

@statman you raise a really good point here. Probably in most cases we would want the noise (RMSE) in the experiment to reflect typical process variation because we really do want to understand if the treatment effects are large relative to what we would expect typically in production.  If we are running say, 3 replicates over the duration of a 15 or 20 run DOE -- one near the beginning, one near the middle and one near the end -- and if on average we are getting "similar" results with little spread, then we have some reasonable indication that the measurement process is stable.

But how much spread is a "little"?  A Gauge R&R should help us consider a way to baseline the precision of our measurement method for Y, perhaps even before we do a DOE (for process characterization and discovery).  If we can characterize the repeatability and reproducibility (precision) of the measurement system in Gauge R&R, at least we have a baseline for what we determine as 'acceptable precision' in the measurement of Y before running that DOE. Of course we could do this concurrently with a DOE. It all really depends -- in my mind on things like the 'robustness' of the measurement system (which depends on things like: how well we've characterized it before, how complex we believe the system is, how much we understand about the interaction of the measurement system with the part of interest which may or may not be in scope the Gauge R&R study... & that's another one up for debate!)

[#3]

No debate on questioning the measurement system always.  You can either understand it á priori or during an experiment with nested components within treatments.  I do believe there are better methods than the "typical" gage R&R (see Wheeler).

But my point has less to do with measurement errors and more to do with how experimental error is estimated and what it represents during an experiment.

[#4 · Accepted solution]

I completely agree with @ih . I would also say that sometimes the answer is that you just can't. Power (and other design diagnostics) are often only really useful as relative measures to compare one design to another. A lot of times we don't have enough information for reliable estimates of absolute power. Here is an answer to a previous question like this that you might find useful.

[#5]

Thanks Phil, appreciate your support also here, as always

[#6]

Happy to help, @YanivD. My concern about power analysis is that it might stop people from using DOE. Often people hear that power has to be >0.8. So if they look at the power analysis in their DOE software and they see something less than 0.8 they just think that they might as well not bother with a designed experiment. But you need to understand that the absolute number is only meaningful if you use good estimates of the noise and the signal that you need to detect. Many times we don't have good estimates of one or both of these. So the power estimate is not meaningful. But that does not mean there is anything wrong with the designed experiment.

[#7]

thank you, absolutely agree with you - the DOE approach itself is much more analytical and professional than any other (usually intuition driven).. 

[#8]

Here is another approach. The initial values for RMSE and coefficients might seem unrealistic. How often would the anticipated values be 1? Very unlikely in practice. On the other hand, together, these values also represent the case where you expect the effect (twice the coefficient) contributed by each term to be twice the RMSE. That is, it is a relative measure of the response variance. This approach can be helpful when it is difficult to determine absolute values for the RMSE and coefficients.

[#9]

agree, thank you for your helpful inputs @Mark_Bailey