Is it possible to use k-fold cross-validation with bootstrap forests model and standard least squares regression in JMP?
I have done the k-fold cross-validation with neural networks model, and I am looking for some methods which could use the same cross-validation, so that I can do model comparisons.
Would like to add we are experimenting with a new automated way to perform k-fold cv in the new XGBoost add-in for JMP Pro. If you have access to JMP Pro 15 and would like to try an early adopter version, please let me know. In addition, we are tentatively planning to have the add-in available in the JMP lab at Discovery Munich with JMP Pro 16 Early Adopter.
Hi @Nazarkovsky , The new Model Screening platform offers breakdowns by fold for both k-fold and nested k-fold, and as you might guess the output can become voluminous. In the XGBoost add-in the results are still summarized over all folds, but for any model Red Triangle > Save Prediction Formula produces columns in the table for fold membership and corresponding in-fold or out-of-fold predictions. Recommend obtaining the latest JMP 16 Pro Early Adopter to try the new functionality, and we really appreciate your feedback as we continue making enhancements along these lines.
Thanks, @russ_wolfinger . Today I have played with XGBoost for a continious response. Many thanks for your invitation sent. Later on I see how to handle cathegorical responses and give the final feedback in the field provided.
Actually, talking about K-fold cross-validation, I meant a weak place in JMP Pro: it does not demonstrate content of folds. Just like this: https://en.wikipedia.org/wiki/Cross-validation_(statistics). Scripts writing is not my strong skill, so I can only rely on developers in JMP that they could provide such an option.
And I might have not got your idea right why the output must be enormous. Normally, it is represented as a list of the folds with the average of the metrics (R2, RASE, AIC etc) in the end. Please, correct me, if I am wrong.
Thank you very much!
As I mentioned in your other discussion about the same topic, you might consider using AICc across all the candidate models. JMP Pro also provides a model comparison platform to assist you.
There are no labels assigned to this post.