cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Check out the JMP® Marketplace featured Capability Explorer add-in
Choose Language Hide Translation Bar
bbrown1
Level I

k-fold r2

I am performing a stepwise procedure and evaluating the performance using a k-fold r-squared. Currently, the only k-fold r-square that is provided is averaged across all 5 folds. Is it possible to secure the r-squared for each individual k-fold in a model? 

1 REPLY 1
Victor_G
Super User

Re: k-fold r2

Hello @bbrown1,

In order to see the individual results for each fold, you can try to use instead the "Model Screening" platform ("Analyze", "Predictive Modeling", "Model Screening").
You'll enter your factors and response(s), check the "K-fold cross-validation", specify the number of folds, and simply check the "Fit Stepwise" in the menu "Methods".
Then, you should have your individual folds results for training and validation like in this screenshot : 

 

Victor_G_1-1651735401115.png

Be careful not to do "cherry picking" and choose to use the fold with the best results, as it will mean you are choosing the training and validation set in order to get the best results, which is not the target of this validation technique (and not the objective of a validation at all). It's more for assessing the robustness of a model in the context of small dataset, where no data is "lost" or excluded in the validation set.
You can find more information on cross-validation and model assessment in this discussion : Solved: How good is K-fold cross validation for small datasets? - JMP User Community 

I hope it will help you,

Victor GUILLER
L'Oréal Data & Analytics

"It is not unusual for a well-designed experiment to analyze itself" (Box, Hunter and Hunter)