topic Re: How to get RMSE of the Power analysis for DOE in Discussions

How to get RMSE of the Power analysis for DOE

DendrogramSteer — Mon, 26 Jun 2023 20:32:35 GMT

Hello! I think this question was asked several times before. But I am still confused about how to get the RMSE for the power analysis for DOE with multiple factors. Say I have 7 factors and 10 responses, do I have to first conduct 10 different pilot studies for each response to construct a fitting model, and then get the RMSE (that will be a lot of work!)? How to set up the pilot study? Can someone provide more detailed procedures to obtain the RMSE? Thanks!

Re: How to get RMSE of the Power analysis for DOE

Phil_Kay — Tue, 27 Jun 2023 09:33:32 GMT

If you need power analysis for all 10 responses, you need RMSE for all 10 responses. First of all I would question why you need power analysis for all 10 responses. Are they all that important? Perhaps they are.

This does not mean that you need to do a separate pilot study for each response. That would be very strange. You are not going to carry out separate experiments for each response, after all.

In fact, you may not need to run any pilot study at all.

You need an estimate of the variability in the response when the factors are held constant. You might have this from existing data on the process/system. If not, then measure all responses for 6 runs (more is better if you can afford it) with all factors set to constant. The std dev of each response over those 6 runs will be a useful (not perfect) estimate of the variability in the response that you can use for power analysis.

I hope that helps,

Phil

Re: How to get RMSE of the Power analysis for DOE

Victor_G — Tue, 27 Jun 2023 09:01:10 GMT

Hi @DendrogramSteer,

Your question around the Power analysis assessment is indeed a very frequently asked topic on this forum.

There are some interesting discussions you may want to look at here (not an exhaustive list), about how using this analysis and how to trust the results from the model :

https://community.jmp.com/t5/Discussions/Should-I-consider-power-analysis-in-DOE/m-p/501063

https://community.jmp.com/t5/Discussions/Comparing-DoEs-Why-D-G-A-I-efficiencies-are-all-the-SAME-and/m-p/559266

https://community.jmp.com/t5/Discussions/Losing-Power-and-Prediction-Variance-in-Custom-DOE-constraints/m-p/544828

Power is the ability to detect significant effect if they are effectively present. I guess based on the characteristics of your study that you may be in a screening (or beginning of optimization) phase, hence your need to evaluate and assess power of your design, to be sure not to miss significant effects.

In order to use Power analysis efficiently, you need to specify :

The size of the signal you need to detect (through "Anticipated Coefficients" values)
Estimates of the experimental and response measurement noise (through "Anticipated RMSE" value) (to be determined for each response, or use the worst case scenario (bigger value))
Significance level threshold (by default 0,05).

You can find more info on the Power Analysis platform here : Power Analysis (jmp.com)

You may not have these informations at the beginning of your study or in a screening phase if you don't have historical data (and create these pilot studies may represent a lot of work as you mention, without having a lot of added value compared to runs that could be done in the context of your DoE).

You can however use this Power analysis platform to compare different designs and/or models, and assess how your experimental budget/constraint may affect the possibility to detect effectively significant effects.

I hope this first answer will help you, I'm sure other DoE experts like @Phil_Kay can also provide new perspectives or enrich this discussion,