Model Comparison – Continuous Response
Example: We use the Body Fat.jmp data to predict Percent body fat. Formulas for several models, saved to the data table, are grouped under Prediction Formulas in the column panel. The models were chosen based on the model’s performance on validation data.
- From the data table,select Analyze > Predictive Modeling > Model Comparison.
- Select the prediction formula columns from Select Columns, and click Y,Predictors. If no columns are selected JMP will use all saved prediction formulas in the data table.
- Select Validation from the Select Columns list, and click By. Then, click OK. JMP provides the following statistics for comparing the performance of the different models (statistics for the Validation set are displayed):
- RSquare (higher is better)
- RASE: Root Average Squared Error (like RMSE, but can be compared for competing models)
- AAE: Average Absolute Error (a measure of the magnitude of the errors)
Model Comparison – Categorical Response
Example: In the Equity.jmp data, we predict Bad as a function of several predictors. In this example, the data was partitioned into Training, Validation and Test sets.
Create the predictive models of your choice (with Validation), then follow steps 1-3 above to generate model statistics for each value of the Validation column.
- The misclassification rates and other measures of performance are provided.
- Select Decision Threshold under the red triangle to launch an interactive tool to compare correct/incorrect classification rates using different cutoff criteria.
- ROC curves, lift curves, model averaging and other options are available from the red triangles.
- The profiler can also be shown for each model to provide a detailed view of the effect each of the factors has on the outcome and compare the predictions for various factor settings.
Model Comparison is also available from the Formula Depot.
Visit Predictive and Specialized Models > Model Comparison in JMP Help to learn more.