How to evaluate model accuracy on holdout set

Danijel_V — Sat, 21 Sep 2019 06:39:12 GMT

Hello,

I am using JMP 13 and I can not seem to find a way to automaticaly evaluate my model's accuracy on the holdout set. I am buliding a linear regression model.

So I would like to preserve the model (with the contained effects and estimates sizes), which was bulit on the training set, but calculate accuracy predictors (R^2, R^2 adjusted and RMSE) on the holdout set. In the moment the only solution I found is to calculate RMSE "by hand", since you can calculate the residuals. Is there any way to do this automaticaly?

This seems such a basic operation for model buliding process so I am quite sure there should be some good option.

Thank you for your help,

Danijel

Re: How to evaluate model accuracy on holdout set

Mark_Bailey — Sat, 21 Sep 2019 11:34:01 GMT

The ability to automatically perform honest assessment of predictive models is a JMP Pro feature. The Fit Model launch dialog in JMP Pro has a Validation analysis role that is not available in JMP.

The Tabulate platform might make the estimation of the RMSE easier for you. Use your validation column for grouping and tabulate the standard deviation of the saved residuals.

topic How to evaluate model accuracy on holdout set in Discussions

How to evaluate model accuracy on holdout set

Re: How to evaluate model accuracy on holdout set