turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Set Probability Threshold / Confusion Matrix

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

May 8, 2017 12:41 AM
(890 views)

The default probability threshold is set to 0.5. That can make it difficult to compare models when one has higher specificity and lower sensitivity, but both have similar accuracies and positive predictive values. How do we change the probability threshold to compare model performance?

Thanks!

Aaron

1 ACCEPTED SOLUTION

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

May 9, 2017 1:14 PM
(1713 views)

Solution

Aaron,

It is unclear which platform you are working in. If you happen to have JMP Pro and are fitting models using generalized regression then one of my favorite JMP13 features is the ability to interactively change the probability threshold and watch my sensitivity and specificity estimates change. But I digress. To explore various cut-offs on a model I might save the model formula to my data table so that I can set up a column with the "model call" (use the formula: score> cut-off to get a column of 0s and 1s) and use that to evaluate model performance (thinking in the diagnostic model world). If you are in the logistic platform and have ROC curves for your models then the ROC Table will provide "2x2" tables for each possible cut-off in your data set. Finally there is an add-in in the file exchange that will calculate a series of performance measures and confidence intervals for diagnostic type models.

https://community.jmp.com/t5/JMP-Add-Ins/Performance-Summary-for-Diagnostic-Tests/ta-p/22524

Hopefully something here is helpful to what you are trying to do.

It is unclear which platform you are working in. If you happen to have JMP Pro and are fitting models using generalized regression then one of my favorite JMP13 features is the ability to interactively change the probability threshold and watch my sensitivity and specificity estimates change. But I digress. To explore various cut-offs on a model I might save the model formula to my data table so that I can set up a column with the "model call" (use the formula: score> cut-off to get a column of 0s and 1s) and use that to evaluate model performance (thinking in the diagnostic model world). If you are in the logistic platform and have ROC curves for your models then the ROC Table will provide "2x2" tables for each possible cut-off in your data set. Finally there is an add-in in the file exchange that will calculate a series of performance measures and confidence intervals for diagnostic type models.

https://community.jmp.com/t5/JMP-Add-Ins/Performance-Summary-for-Diagnostic-Tests/ta-p/22524

Hopefully something here is helpful to what you are trying to do.

2 REPLIES

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

May 9, 2017 1:14 PM
(1714 views)

It is unclear which platform you are working in. If you happen to have JMP Pro and are fitting models using generalized regression then one of my favorite JMP13 features is the ability to interactively change the probability threshold and watch my sensitivity and specificity estimates change. But I digress. To explore various cut-offs on a model I might save the model formula to my data table so that I can set up a column with the "model call" (use the formula: score> cut-off to get a column of 0s and 1s) and use that to evaluate model performance (thinking in the diagnostic model world). If you are in the logistic platform and have ROC curves for your models then the ROC Table will provide "2x2" tables for each possible cut-off in your data set. Finally there is an add-in in the file exchange that will calculate a series of performance measures and confidence intervals for diagnostic type models.

https://community.jmp.com/t5/JMP-Add-Ins/Performance-Summary-for-Diagnostic-Tests/ta-p/22524

Hopefully something here is helpful to what you are trying to do.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

May 9, 2017 2:34 PM
(854 views)