Why is the RMSE so high in this DSD analysis

[#1]

Hi,

I am new to DOE and Definitive Screening Design, that is why I am having some questions regarding the analysis of DSD. I am using JMP 13.

I want to find significant factors in the process of spherical crystallization. I did Definitive Screening Design with 6 factors on three levels, with four additional runs (2 fake factors). I am measuring several different responses. The design evaluation shows me, that I have good statistical power. However, when I try to analyse the results using Fit Definitive Screening, I get none or one at best significant main effects. As a consequence, I also get none significant second order effects (see the added photo below, right). I think the reason for that is very high RMSE. I was learning how to do Fit Defintive Screening on the example shown in the JMP tutorial (file Extraction 3, see added photo below, left). When I analysed the example I got the same results as shown in the tutorial. I also noticed, that in analysing extraction 3 RMSE was much lower, although responses in Extraction 3 and in my experiment were approximately the same size, since both of responses are Yield. 

I don't understand how is it possible that the difference in the RMSE is so big. If you look at the distributon in main effect plot, you can see, that the distribution in my experiment is similar, if not a bit narrower that the distribution in Extraction 3. I also calculated RMSE using Fit Y by X tool, where I got bigger RMSE in Extraction 3 (around 20) than in my experiment (around 18). That's why it looks like the analysis with Fit Definitive Screening makes the difference.

Am I missing something?

Should I do some preanalysis processes with my data prior to Fit Definitive Screening?

Or do I really have an experiment with big RMSE. If so, is it possible to fix that?

Thank you.

Danijel

[#2]

Yes, we are aware that measurements could also be the problem. Each time we did at least 3 repetitions of measurement (which showed a relatively low coefficient of variance) for each sample. We had some problems with the lasser diffraction analysis, which probably raised RMSE in some responses, but RMSE in the responses which do not include laser diffraction analysis is also a bit high, but not as much. 

[#3]

Thank you for very good ideas. We decided to repeat the second run twice, so we will see if variation is high also in one run, or just in the whole model. Then we will add the new runs to the model and see if RMSE is any lower.

Does that makes sense to you?

[#4]

This is a start. A couple of cautions on these repeat runs: you would be assuming that the variability is constant over the design space (in other words, run #2 is representative of all other experimental conditions). Does the variability on your repeat runs capture all of the variability sources encountered when running the initial design (did operators change? instrumentation changes? setup changes?). How comfortable are you in estimating a variance based on only 2 or 3 data points? That is essentially what you are doing.

[#5]

One correction from my previous post: I meant we will repeat the central run twice, not the second run (I apologise for a little confusion).

Anyway, the central run should also cover all of the points (cautions) you mentioned. 

Since we will be repeating central runs, we are hoping, that the variability of the central runs would be the best (the best of the 17 runs we can repeat) approximation of the variability of the whole design . Furthermore, we are hoping, that RMSE of our design will drop as well, since there will be more runs in the model. If the drop of  the RMSE will be big enough to detect some new effects, we won't have to estimate a variance based on only 3 data points.

[#6]

Just to tell you how we solved the problem in the end. We tryed different data transformations, so we could get more normal distribution, with more constant variance over the entire range of response. When we analysed transformed response, we could detect many more effects (main and second order effects). 

[#7]

You mean a normal distribution of the residuals, right?

Glad you found a solution!

[#8]

I mean the distribution of response.

[#9]

Why would you expect the response to be normally distributed if there are active effects?