Definitive Screen Design (DSD) Model confirmation

[#1]

Hi all,

I designed a DSD with the following info:

- Design: 7 continuous and 1 categorical factors, 22 run

- Design Evaluation on power and correlation: very good

- Fit model via Fit Definitive Screening: excellent

• RMSE = 0.2803, RSq =0.997, PValue<0.001)
• Lack of Fit: no (F Ration 191, Prob > F = 0.0561)
• Residual and Studentized Residuals are all good

- Model structure: 

• Contains 5 main, 4 quadratic and 1 interaction factors

- Model confirmation: try a few runs (points) around to Max desirability

• some agree with prediction very well
• some show large differences

Due to resource constrain, we can run 5 to 10 confirmation runs.

Please advise:

- Potential causes of the difference say underfit or overfit?

- Method or procedure to improved/optimized the model

Thanks.

[#2]

Mark,

There are many factors can affect response we want to control. We want to identify which are the main contributors. Through engineering experience, we were able to narrow it down to eight factors. The purpose of the DOE is to further narrow it down and get practical direction on how to improve the response. For this purpose, I think DSD does an excellent job.

I have tried two things:

1) Within the DSD platform, I removed a few not as significant terms and trimmed the model from 5 MEs, 4 quadratics and 1 interaction down to 4 MEs, 1 quadratic and 1 interaction. I was able to get more desirable confirmation results.

2) I simply add 4 of the 5 confirmation runs to the original DSD table, and use JMP Analyze: Fit Model : Standard Least Square to the same set up as 1) (4 MEs, 1 quadratic and 1 interaction). I was also able to get more desirable confirmation results with the remaining confirmation run data.

Question: Can I say screening effective factors and generating predictive model are two things, i.e., the later may be due to other reasons such as not having enough data (runs) to fit the model, however, the screening results are still sound and applicable?

Your point about space-filling design is very interesting. I will look into it and circle back in a few days.

Thanks and Happy Holidays!

[#3]

OK...I can't resist. I've been sitting on the sidelines on this thread...but now I can't help myself. @ZenCar Your question in your last post is at the heart of this narrative. DSD are first and foremost screening designs. DOE for predictive purposes is something completely different. One of my pet peeves of DOE practitioners is they start with a practical problem that is fundamentally an optimization goal for a set of responses...and then somehow forget the tried and true method of sequential experimental through inductive reasoning. George Box and others have discussed this for years. It's the way to proceed. That's at the heart of the difference between all screening designs and other designs more adept at prediction.

Also to @statman 's earlier points about 'noise'. Noise in future operation of a system comes in all manner of forms shapes and sizes. Many can't be built into designed experiments. For example, when I worked in industry, often production systems would be shut down for major maintenance and capital upgrades. The rule of thumb was 'You never want to be first up on the coating machine after a capital shutdown. The magic of physics and chemistry you thought you understood may not now apply based on the process changes that have occurred.' How you gonna build  a major capital systems upgrade into a DOE before the upgrade happens? Not practical and can't be done.

So if predictive power of DOE based models falls apart in the future...why is anybody surprised? Just means more work is needed to fully understand the process.

Rant over. Thanks for reading this far if you have. 

[#4]

Hi @P_Bartell ,

Your comments are quite enlightening. I really appreciate you pointing out that DSDs are first and foremost screening designs. Yes, that was the primary goal before people want to get more out of it.

Coupling with @Mark_Bailey point about the determination of simulation software, I am going to question about the need of accurate predictive model while we now know the direction from DSD and can get exactly answer from the simulation software.

Thanks for such a pleasant writing. Enjoy your holidays.