Doe and power

[#1]

Hi,
When planing an experiment and looking at the design and power for each parameter (at the initial design step), there are is an option to change the RMSE. By changing it, obviously the power is changed. The question is how we can predict or assume this parameter ahead (in screening experiment for example)

Thanks

[#2]

An understanding of the system being studied helps here.  How good of a job do you think you will be able to do when modeling the system?  If you have any historical data, measure the variation in your responses when the system is at steady state.  If you really have no idea you might need to take some measurements first and use the standard deviation of those responses as a rough estimate the lowest possible RMSE of your final model.

[#3]

Thanks, will try it out

[#4]

@ih , @Phil_Kay I would think that the total estimate of the RMSE for the response could just be the sum of the squared variances attributed to each of the predictors.  Does that seem like a reasonable point-estimate for the RMSE?  Or would you necessarily have to run a preliminary multifactorial experiment (or collect data observationally) and calculate the RMSE on the response on the basis of a fitted model to the response vs the predictors?

Would it make sense to potentially consider using the worst-case (highest possible) estimate of the  RMSE (vs the lowest possible)?  I would think that, for example, considering an upper 95% confidence limit on the average sigma estimate might be more conservative (result in lower estimates of Power under the assumption that the total average sigma (RMSE) is in-fact larger in the population than what was observed in the sample). 

[#5]

@PatrickGiuliano - You could use estimates of the variance in the predictors (factors) but you would also need to know how those factors affect the response in order to calculate how the variance propagates into the response. That means you would need a model of the response versus the factors, which you only have after you have run the experiment. (And you would also need to add an estimate of measurement variation for the response.)

Quite simply, you just need the standard deviation of the response for repeated runs at constant factor settings. Looking at the CI for this estimate will be useful, as you say, because unless you have a large sample for estimating the standard deviation, you will have large uncertainty. And this will have a big impact on the power estimate. You also need to worry about whether this estimate reflects the variation for all of the factor space or just for the settings where you have taken repeated runs.

So back to my point that the absolute estimate of power is very often not useful in industrial experiments.

[#6]

@Phil_Kay  all of this makes a lot of sense the way you have explained it thank you for answering and clarifying my thinking in a few respects.  The RMSE reflects the variance in Y as a function of the X's.  So we need to at least measure Y at fixed levels across the X's to get a reasonably accurate 'historical' estimate of the RMSE, that much is quite clear. 

The other thing you mentioned about how the variance might not be constant at different levels of X is a good consideration.  If we didn't test (repeatedly) at those levels, then we don't know for sure and we are therefore making an assumption that sometimes doesn't hold. It's very palpable that the variance in Y might differ at different levels of X, e.g. for a number of reasons not the least of which could include problems with the measurement system. I remember cases in industry where we saw 'parallax' in optical metrology measurements, e.g. where measuring near the edge of a cylindrical part caused more variation in the visual field than at the center of the the part because of back-lighting reflectance causing edge-detection problems.

[#7]

Yes, @PatrickGiuliano , I think there will be many cases where the "noise" is not a constant across the factor space. Intuitively, if you are setting factors to the extremes of their ranges, experimental variation will be higher. This is often brought up as an argument against using repeated centre-point runs for post-experiment estimation of RMSE; variation in the centre of the factor space is probably not a good estimate of the variation elsewhere in the factor space.

[#8]

Hi Phil, 

This is good point and agree about the Noise across factor space. When we actual perform experiments and measure outcome for extreme settings may be variable due to limitations of equipment's/machines/raw material properties. In such case what I believe is to consider the average value for noise and signal to initiate DoE design. 

[#9]

Great discussion; as one of my dear mentors in industry, @charles_chen likes to say: "Gauge R&R and DOE together." 

[#10]

Perhaps, but the question is:  Are the treatment effects (factors and interactions) significant when compared to the "typical" variation in the process?  If the center point runs are the current conditions, then replicates of these (randomized over the entire design order) might provide a good estimate of current variation of the process and if any of the treatments are significant in comparison to this variation.  The objective is not to determine the significance of treatment effects in the design space.  It is to determine if treatment effects will be significant and useful in the future.