Share your ideas for the JMP Scripting Unsession at Discovery Summit by September 17th. We hope to see you there!
Choose Language Hide Translation Bar
Highlighted
Danijel_V
Level III

K fold Cross-Validation in DSD

Hello,

 

I have a quick question. If you analyse DSD the literature (instructions in JMP) proposes two methods. The first is Fit defintive screening and the second is Forward stepwise regression, with Stopping rule: min AICc and rules: Combine. I was thinking about using k fold cross validation, if Forward stepwise regression turns out to give the better model than the Fit definitive screening. But does that even make sense with DSD? DSD is a foldover design and I think that doing model on only, for example, 3/4 of the runs, wouldn't make sense. Am I right or wrong?

 

Thank you for your answer.

Danijel

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: K fold Cross-Validation in DSD

First of all, there is no method that I know of that is guaranteed to find the best model. (That is, if you can even define "best.")

 

Second of all, this case is a screening experiment in which economy is one of the most desirable characteristics of the design. That economy means that a DSD is a small design, compared to the minimum number of runs for the linear model with only main effects. Holding out any data will affect the estimation and testing (power). You might be able to compensate by add extra runs, but then you might just as well use custom design.

 

Third of all, K-fold cross-validation is useful with small data sets but the analysis and unique set of benefits of the DSD depend on the unique structure of this design. The omission of each fold in turn would destroy this structure and compromise the fitting.

 

Would it work? Most likely. Would it work well? Not likely.

 

It can't hurt to try it, instead of thinking about it. Simulation would be a valuable approach. JMP makes that approach easy.

Learn it once, use it forever!

View solution in original post

2 REPLIES 2
Highlighted

Re: K fold Cross-Validation in DSD

First of all, there is no method that I know of that is guaranteed to find the best model. (That is, if you can even define "best.")

 

Second of all, this case is a screening experiment in which economy is one of the most desirable characteristics of the design. That economy means that a DSD is a small design, compared to the minimum number of runs for the linear model with only main effects. Holding out any data will affect the estimation and testing (power). You might be able to compensate by add extra runs, but then you might just as well use custom design.

 

Third of all, K-fold cross-validation is useful with small data sets but the analysis and unique set of benefits of the DSD depend on the unique structure of this design. The omission of each fold in turn would destroy this structure and compromise the fitting.

 

Would it work? Most likely. Would it work well? Not likely.

 

It can't hurt to try it, instead of thinking about it. Simulation would be a valuable approach. JMP makes that approach easy.

Learn it once, use it forever!

View solution in original post

Highlighted
Danijel_V
Level III

Re: K fold Cross-Validation in DSD

Thank you, I was also thinking similarly.
Article Labels

    There are no labels assigned to this post.