k-fold cross-validation with bootstrap forests model and standard least squares regression

[#1]

Hi,
Is it possible to use k-fold cross-validation with bootstrap forests model and standard least squares regression in JMP?
I have done the k-fold cross-validation with neural networks model, and I am looking for some methods which could use the same cross-validation, so that I can do model comparisons.
Thank you.

[#2 · Accepted solution]

I found the answers to my question on the website: https://learning.oreilly.com/library/view/Building+Better+Models+with+JMP+Pro/9781629599564/xhtml/Ch....

[#3 · Accepted solution]

Would like to add we are experimenting with a new automated way to perform k-fold cv in the new XGBoost add-in for JMP Pro.   If you have access to JMP Pro 15 and would like to try an early adopter version, please let me know.    In addition, we are tentatively planning to have the add-in available in the JMP lab at Discovery Munich with JMP Pro 16 Early Adopter.

[#4]

Dear Mr. Wolfinger,

This issue also concerns me, as I am a JMP Pro15 user and see some things to be adjusted in K-fold crossvalidation sections for all models. In particular, JMP does not have an important option to visualize the folds yet, but shows just its average accuracy (R2) without details in each fold. For instance, I would run the 5-fold validation and would like to see the list of 4 folds + their average with the metrics (R2, AUC, AIC, RASE etc). If there is an add-in with this option, please, let me know, since this very important while handling small datasets. Thanks a lot

[#5]

Hi @Nazarkovsky ,  The new Model Screening platform offers breakdowns by fold for both k-fold and nested k-fold, and as you might guess the output can become voluminous.    In the XGBoost add-in the results are still summarized over all folds, but for any model Red Triangle > Save Prediction Formula produces columns in the table for fold membership and corresponding in-fold or out-of-fold predictions.  Recommend obtaining the latest JMP 16 Pro Early Adopter to try the new functionality, and we really appreciate your feedback as we continue making enhancements along these lines.  

[#6]

Thanks, @russ_wolfinger . Today I have played with XGBoost for a continious response. Many thanks for your invitation sent. Later on I see how to handle cathegorical responses and give the final feedback in the field provided.

Actually, talking about K-fold cross-validation, I meant a weak place in JMP Pro: it does not demonstrate content of folds. Just like this: https://en.wikipedia.org/wiki/Cross-validation_(statistics). Scripts writing is not my strong skill, so I can only rely on developers in JMP that they could provide such an option. 

And I might have not got your idea right why the output must be enormous. Normally, it is represented as a list of the folds with the average of the metrics (R2, RASE, AIC etc) in the end. Please, correct me, if I am wrong. 

Thank you very much!

Mike

[#7]

As I mentioned in your other discussion about the same topic, you might consider using AICc across all the candidate models. JMP Pro also provides a model comparison platform to assist you.

[#8]

Thanks for summary. So k-fold cross validation is not necessary or not possible with bootstrap forest in JMPpro?
When creating a Validation colum when you have a binary outcome, would you go for a stratified Random, statify by Group or Grouped random method?