topic Doe and power in Discussions

Doe and power

YanivD — Thu, 08 Jun 2023 21:11:59 GMT

Hi,
When planing an experiment and looking at the design and power for each parameter (at the initial design step), there are is an option to change the RMSE. By changing it, obviously the power is changed. The question is how we can predict or assume this parameter ahead (in screening experiment for example)

Thanks

Re: Doe and power

ih — Wed, 28 Sep 2022 12:23:02 GMT

An understanding of the system being studied helps here. How good of a job do you think you will be able to do when modeling the system? If you have any historical data, measure the variation in your responses when the system is at steady state. If you really have no idea you might need to take some measurements first and use the standard deviation of those responses as a rough estimate the lowest possible RMSE of your final model.

Re: Doe and power

Phil_Kay — Wed, 28 Sep 2022 13:46:26 GMT

I completely agree with @ih . I would also say that sometimes the answer is that you just can't. Power (and other design diagnostics) are often only really useful as relative measures to compare one design to another. A lot of times we don't have enough information for reliable estimates of absolute power. Here is an answer to a previous question like this that you might find useful.

Re: Doe and power

YanivD — Wed, 28 Sep 2022 16:08:35 GMT

Thanks Phil, appreciate your support also here, as always :folded_hands::smiling_face_with_smiling_eyes:

Re: Doe and power

YanivD — Wed, 28 Sep 2022 16:08:58 GMT

Thanks, will try it out

Re: Doe and power

PatrickGiuliano — Thu, 29 Sep 2022 05:06:02 GMT

@ih , @Phil_Kay I would think that the total estimate of the RMSE for the response could just be the sum of the squared variances attributed to each of the predictors. Does that seem like a reasonable point-estimate for the RMSE? Or would you necessarily have to run a preliminary multifactorial experiment (or collect data observationally) and calculate the RMSE on the response on the basis of a fitted model to the response vs the predictors?

Would it make sense to potentially consider using the worst-case (highest possible) estimate of the RMSE (vs the lowest possible)? I would think that, for example, considering an upper 95% confidence limit on the average sigma estimate might be more conservative (result in lower estimates of Power under the assumption that the total average sigma (RMSE) is in-fact larger in the population than what was observed in the sample).

Re: Doe and power

Phil_Kay — Thu, 29 Sep 2022 08:41:54 GMT

Happy to help, @YanivD. My concern about power analysis is that it might stop people from using DOE. Often people hear that power has to be >0.8. So if they look at the power analysis in their DOE software and they see something less than 0.8 they just think that they might as well not bother with a designed experiment. But you need to understand that the absolute number is only meaningful if you use good estimates of the noise and the signal that you need to detect. Many times we don't have good estimates of one or both of these. So the power estimate is not meaningful. But that does not mean there is anything wrong with the designed experiment.

Re: Doe and power

Phil_Kay — Thu, 29 Sep 2022 08:50:30 GMT

@PatrickGiuliano - You could use estimates of the variance in the predictors (factors) but you would also need to know how those factors affect the response in order to calculate how the variance propagates into the response. That means you would need a model of the response versus the factors, which you only have after you have run the experiment. (And you would also need to add an estimate of measurement variation for the response.)

Quite simply, you just need the standard deviation of the response for repeated runs at constant factor settings. Looking at the CI for this estimate will be useful, as you say, because unless you have a large sample for estimating the standard deviation, you will have large uncertainty. And this will have a big impact on the power estimate. You also need to worry about whether this estimate reflects the variation for all of the factor space or just for the settings where you have taken repeated runs.

So back to my point that the absolute estimate of power is very often not useful in industrial experiments.

Re: Doe and power

Mark_Bailey — Thu, 29 Sep 2022 13:24:39 GMT

Here is another approach. The initial values for RMSE and coefficients might seem unrealistic. How often would the anticipated values be 1? Very unlikely in practice. On the other hand, together, these values also represent the case where you expect the effect (twice the coefficient) contributed by each term to be twice the RMSE. That is, it is a relative measure of the response variance. This approach can be helpful when it is difficult to determine absolute values for the RMSE and coefficients.

Re: Doe and power

YanivD — Thu, 29 Sep 2022 14:06:39 GMT

thank you, absolutely agree with you - the DOE approach itself is much more analytical and professional than any other (usually intuition driven)..

Re: Doe and power

YanivD — Thu, 29 Sep 2022 14:08:33 GMT

agree, thank you for your helpful inputs @Mark_Bailey

Re: Doe and power

PatrickGiuliano — Fri, 07 Oct 2022 06:23:37 GMT

@Phil_Kay all of this makes a lot of sense the way you have explained it thank you for answering and clarifying my thinking in a few respects. The RMSE reflects the variance in Y as a function of the X's. So we need to at least measure Y at fixed levels across the X's to get a reasonably accurate 'historical' estimate of the RMSE, that much is quite clear.

The other thing you mentioned about how the variance might not be constant at different levels of X is a good consideration. If we didn't test (repeatedly) at those levels, then we don't know for sure and we are therefore making an assumption that sometimes doesn't hold. It's very palpable that the variance in Y might differ at different levels of X, e.g. for a number of reasons not the least of which could include problems with the measurement system. I remember cases in industry where we saw 'parallax' in optical metrology measurements, e.g. where measuring near the edge of a cylindrical part caused more variation in the visual field than at the center of the the part because of back-lighting reflectance causing edge-detection problems.

Re: Doe and power

Phil_Kay — Fri, 07 Oct 2022 08:35:46 GMT

Yes, @PatrickGiuliano , I think there will be many cases where the "noise" is not a constant across the factor space. Intuitively, if you are setting factors to the extremes of their ranges, experimental variation will be higher. This is often brought up as an argument against using repeated centre-point runs for post-experiment estimation of RMSE; variation in the centre of the factor space is probably not a good estimate of the variation elsewhere in the factor space.

Re: Doe and power

rsomankar — Fri, 07 Oct 2022 09:08:55 GMT

Hi Phil,

This is good point and agree about the Noise across factor space. When we actual perform experiments and measure outcome for extreme settings may be variable due to limitations of equipment's/machines/raw material properties. In such case what I believe is to consider the average value for noise and signal to initiate DoE design.

Re: Doe and power

PatrickGiuliano — Fri, 07 Oct 2022 17:32:56 GMT

Great discussion; as one of my dear mentors in industry, @charles_chen likes to say: "Gauge R&R and DOE together."

Re: Doe and power

statman — Fri, 07 Oct 2022 17:46:53 GMT

Perhaps, but the question is: Are the treatment effects (factors and interactions) significant when compared to the "typical" variation in the process? If the center point runs are the current conditions, then replicates of these (randomized over the entire design order) might provide a good estimate of current variation of the process and if any of the treatments are significant in comparison to this variation. The objective is not to determine the significance of treatment effects in the design space. It is to determine if treatment effects will be significant and useful in the future.

Re: Doe and power

PatrickGiuliano — Fri, 07 Oct 2022 20:07:44 GMT

@statman you raise a really good point here. Probably in most cases we would want the noise (RMSE) in the experiment to reflect typical process variation because we really do want to understand if the treatment effects are large relative to what we would expect typically in production. If we are running say, 3 replicates over the duration of a 15 or 20 run DOE -- one near the beginning, one near the middle and one near the end -- and if on average we are getting "similar" results with little spread, then we have some reasonable indication that the measurement process is stable.

But how much spread is a "little"? A Gauge R&R should help us consider a way to baseline the precision of our measurement method for Y, perhaps even before we do a DOE (for process characterization and discovery). If we can characterize the repeatability and reproducibility (precision) of the measurement system in Gauge R&R, at least we have a baseline for what we determine as 'acceptable precision' in the measurement of Y before running that DOE. Of course we could do this concurrently with a DOE. It all really depends -- in my mind on things like the 'robustness' of the measurement system (which depends on things like: how well we've characterized it before, how complex we believe the system is, how much we understand about the interaction of the measurement system with the part of interest which may or may not be in scope the Gauge R&R study... & that's another one up for debate!)

Re: Doe and power

statman — Sat, 08 Oct 2022 14:22:33 GMT

No debate on questioning the measurement system always. You can either understand it á priori or during an experiment with nested components within treatments. I do believe there are better methods than the "typical" gage R&R (see Wheeler).

But my point has less to do with measurement errors and more to do with how experimental error is estimated and what it represents during an experiment.