Can I check F 1 score in JMP

[#1]

Hi! I'm wondering if I can check F1 score in JMP pro when I'm comparing different models?

Also does JMP provides precision or recall value for the models?

Many thanks!!

[#2]

I assume you have looked at the Model Comparison platform in JMP Pro. Unfortunately, I'm not familiar with the 'F1 score', so can you give further details or a link, please? Regarding you second question, are you thinking about predicted values, and the uncertainties in these?

[#3]

Hi! Thank you so much for your reply. F1 score is a measure of test accuray. Here's link of wilipedia on F1 score. https://en.wikipedia.org/wiki/F1_score

And for the second question, yes, I'm thinking about the predicted values. 

Precision=True Positives/(True Postives +False Postives)

and recall=True Positives/(True Positives+False Negatives) 

I didn't find precision and recall under model comparison. I'm wondering if JMP have these values. From my understanding, we sometimes may care more about precison or recall than misclassification rate in different situations, right?

[#4]

Hi Ian,

I am a bit surprised with your reply. It is not too much effort (especially, for a JMP expert) to web-search "F1 score", self-educate, and reply to the user with a meaningfull answer. Moreover, the inquirer provided additional info, but never received a meaningful support. That surprises too.

Would you or anyone else with the appropriate expertise help the JMP community with this question. F1 score is a key measure in logistic regression (or any model with nominal response) and imbalanced response levels (significantly more true negatives than true positives). Greatly appreciated. As noted, Wikipedia describes these concepts in details.

The Model Comparison link you provided is elaborate, but seems to be missing the F1 score computation. I'm sure this can be done by hand, but, if missing in JMP, should be strongly considered as a feature. Thanks in advance.

[#5]

A wise man once said that 'the only scarce commodity is time', and I think it's understood that the Community relies on crowdsourcing knowledge (from users and JMP staff alike), and could not operate effectively if it's reliant on a small group.

In addition to the Commmunity, there is also the official support channel, which anyone who licenses a SAS product may use. One key difference is that there are service level agreements for response times, and internal processes for escalation to achieve satisfactory resolution. This is definitely the best mechanism for surfacing suspected bugs, but also for making new feature requests. In addition, 'How do I?' questions, or 'I'm used to software X but can't do the same thing in JMP' questions will also be answered, but within this more rigorous and predictable framework.

So, for bugs and features, or if the Community appears unresponsive, I would certainly consider this channel too. And, stating the obvious perhaps, it doesn't have to be 'one or the other'. Many Community threads have lead to support tracks.

[#6 · Accepted solution]

This type of question can be directed to JMP Technical Support so we can get involved. 

As for the question, JMP gives measures of Sensitivity, and 1-Specificity in the ROC Table when the "ROC Curve" menu item is selected, along with the True Positives, True Negatives, False Positives, and False Negatives.  F1 is not specifically listed, but, from the description, it can be calculated with these measures.

[#7]

Hi Ian, you asked for a formula. Jenny provided one. Still no solution, yet the discussion was viewed 500+ times. So, the question has weight.

[#8 · Accepted solution]

The F1 scores (and other popular measures of accuracy) have been deprecated in some corners of the Machine Learning community for their performance under bias.  Please reference the attached file, which suggests the utility of Receiver Operating Curves and other measures.

JMP does include ROC (and Lift Curves) in some platforms.

[#9]

Can anyone provide a clear and succinct answer to Jenny's answer, instead of dumping an opinion in this discussion? This is not a blog. Question asked, question answered. Thanks in advance.

[#10 · Accepted solution]

Hello @Oleg,  

---

@Jenny wrote:

...didn't find precision and recall under model comparison. I'm wondering if JMP have these values.

JMP does not have these values natively in most platforms (as of the release of JMP 16.2), and as @Duane_Hayes answered in 2017, but they can be calculated from the output currently provided.

@Jenny wrote:

From my understanding, we sometimes may care more about precision or recall than misclassification rate in different situations, right?.

---

Yes this is my understanding.

To this end,

I am attaching an example with calculations in column formulas in JMP where Accuracy, F1-Score, Precision and Recall are calculated, using the formulas that I was readily able to find. Those for Precision and Recall are consistent with what @Jenny provided (also see p. 144 of "The StatQuest Illustrated Guide to Machine Learning!!!" by Josh Starmer, Ph.D, © 2022).  Formulas for Accuracy and F1-Score can be found here among other places (Wikipedia):  https://neptune.ai/blog/f1-score-accuracy-roc-auc-pr-auc.  This example is from the VSSTeamData.jmp sample data file available as part of JMP's free online applied statistics course, STIPS. The calculations are consistent with what's reported by JMP in the Confusion Matrix output associated with the Partition model generated and saved herein as a saved data table script. 

PatrickGiuliano_1-1659431181161.png

Click the green play button next to the script name to regenerate the model after opening the .jmp data table.

PatrickGiuliano_0-1659431007024.png

I hope this information is helpful!   - Patrick Giuliano (@JMP Technical Support)