topic How do I auto-calculate precision and F1 score for models on imbalanced data? in Discussions

How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Fri, 09 Jun 2023 23:24:23 GMT

I like that JMP's prints a confusion matrix and ROC for the logistic (Generalized) regression, but some important metrics seem to be missing from the results: precision and F1 score. Is there a way to auto-include these in the output? Or, have it be computed as "automatically" as possible when I re-run a new classificaiton model. I am trying to avoid manually recomputing these with each new model.

UPDATE: Please post answers to the questions only. I am not looking for an opinion about whether I shall or shall not use Precision and F1. I am confidently looking for a way to compute these automatically in JMP. If you are not aware of such technique, there is no reason to contemplate whether this is doable or not, or post anything below. Thank you :)

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

txnelson — Thu, 28 Sep 2017 14:00:20 GMT

If JMP does not provide a specific statistic you want for a given analyses, my experience has been, the developers at JMP have a good reason for having not included it. This Discussion Group, while monitored by JMP, is not the best way to get your statistic included into a future release. You need to send an enhancement request to

support@jmp.com

They have a formal tracking system for such items.

But now to the beauty of JMP. Many times, the Discussion Community will have a solution to what you need. But, since this topic has been discussed before, with no response on F1 etc. I would assume the solution is not available. However, many, if not most, of these requests can be added to your analysis through JSL. So in response to your question, I am sure you could write a script to add these to your analysis.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Kevin_Anderson — Thu, 28 Sep 2017 14:54:45 GMT

Hi, Oleg!

The F1 scores (and other "important" popular measures of accuracy) have been deprecated in some corners of the Machine Learning community for their performance under bias. Please reference the attached file, which suggests the utility of the Receiver Operating Curves and some other measures.

And like Jim notes, you can script anything your heart desires.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:20:23 GMT

Hi Jim, your answer implies that JMP is perfect and contains just what it needs to have (up to a good reason). That is, anything missing in JMP does not have a good reason to be in JMP. An interesting claim ;) The discussions themselves generate plenty of "good reasons" for new JMP features. A question was posted about F1 (not precision) and still remains unanswered, yet, these measures are widely discussed and used in scientifica publications.

I like your suggestion of emailing the support, but they ask to launch a discussion. My hope is that the community will provide the answer, even if not right away.

If you feel my question can be improved, I welcome your suggestions.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 16:54:51 GMT

Hi Kevin, Thanks for the reply. I do not understand how others' preference for the use of F1 score shall affect my need for it in this discussion? :) Each and every metric has a bias. There are thousands (more recent) publications praising the value of F1 and precision metric.

The bottom line, in this discussion I seek assistance in auto-copmuting F1 and precision scores. Yes, it can be scripted; and I would greatly appreciate an assistance from someone who has already implemeted it, so that this wheel is not reinvented.

Indeed, I am alreday using ROC, but I also need F1 and precision. At the very least, the answer (whenever discovered) to the question I asked shall help others to auto-compute not just F1 or precision score, but any metric. Please keep the bigger picture in mind :)

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

txnelson — Thu, 28 Sep 2017 17:00:04 GMT

My previous response was intended to be much simpler than it appears you have interpreted it.

1. JMP is not perfect, but it is pretty good, and the developers really keep up with the areas they are working on, and therefore what they are presenting deserves a good look.

2. Best way to make sure your request for an enhancement has made it to JMP is to send it to

support@jmp.com

3. The request for F1 was already covered in a recent Discussion track, thus I assume there is not a user with a ready made solution. Therefore, my encouragement that it can still be done in JMP, where you could code the solution using JSL

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:22:39 GMT

Jim, here is how a discussion spawning "good reasons" for new features to be added to JMP.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Kevin_Anderson — Thu, 28 Sep 2017 17:29:52 GMT

Hi, Oleg!

Here's the big picture: Jim Nelson has solved many, many problems in this community. Please don't force your opinions on him.

And here's my opinion: you may get more help with your scripting if you act like you've given it a try and ask for help with a little less arrogance. :)

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:23:27 GMT

Hi Kevin. I appreciate Jim's help and have given him Kudos and accepted his solutions whenever it was appropriate. However, I must not agree with him on every point. In this thread, I expect answers to the questions asked, not opinions on what's fashionable in the ML world.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:04:15 GMT

Thanks Jim.

1. I agree

2. Support asks to use the discussion group, which is what I am trying.

3. Sure, JSL is a good direction. I was hoping that someone has already implemented it and can lead to an example.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:02:30 GMT

Kevin, you may have misread my original question. I was hoping for a built-in computation of both F1 and precision. JSL is a reasonable alternative.

Do you have an example I can follow?

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Kevin_Anderson — Thu, 28 Sep 2017 18:22:01 GMT

Hi, Oleg!

Perhaps it is an unfamiliarity with the language, but the word Discussions (the name of this site in the Community) mean much more than just questions and answers. It can rightfully include opinions, admonitions, pleas, rants, and many other forms of dialog, irrespective of your desires.

One of the many things I love about statistics (and JMP!) is that the really great, inventive work seems to happen at the interface between the science of the applications and the art. See? My opinion.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 18:34:02 GMT

Hi Kevin. :)

If this "Discussion" thread is anything like StackOverflow, then I would expect clear answers to clear questions. If I am mistaken, feel free to guide me to the guidelines for posting to discussions. I can read other people's opinioins in news papers. However, in the context of science and in the spirit of "get it done", the clear and succinct answers are value-added.

Btw, did you just admit that I recieved opinions to my question? LOL.

Anyhow, it would be more effective to seek you help on this one.

Have you come across any JSL examples that would help me script F1 and Precision, and, preferrably, added it to the regression output page?

Cheers!
O.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Kevin_Anderson — Thu, 28 Sep 2017 20:08:53 GMT

Hi, Oleg!

I don't think I misread your original question...but that's just my opinion...

Here's an example for you to try:

Take Big Class.jmp. Run some ridiculously-overfit Neural Network analysis with Sex as the Response and Age, Height, and Weight as the factors. Consider Female as the Positive case. :)

I got a Contingency Table that looks like this:

Choose Make Into Data Table.

TP= 16. FP=2. FN=2.

F1 can be calculated by 2*TP/(2*TP+FP+FN). I get F1=0.89.

This is a very easy thing to script (in my opinion :), and you can even automatically include this metric (and any other) on any of your output with a script. Give it a shot, and I'll try to help you get it if you fail.

Good luck.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 20:57:30 GMT

Kevin, thanks for the example.

I don't have a problem running a model and computing F1 and Precision scores by hand. But, is there an example to auto-compute these in JMP (which is what I asked originally)?

In the meantime, I coded these Python, which took less time than posting a question. If someone provides an answer in the future, I'll gladly give it another try.

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

txnelson — Thu, 28 Sep 2017 22:31:51 GMT

Examples are available in the File Exchange, the script examples installed when you install JMP, directly in the Scripting Guide book, also installed when you install JMP, etc. And on top of that, here is a sample script that reads from an output and displays it in a different form as part of an output.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA\Big Class.jmp" );

// Run the Fit Y by X to get the XY plot
Biv = dt << Bivariate( Y( :height ), X( :weight ) );

// Run the distributions to get the stats
Dis = dt << Distribution( Continuous Distribution( Column( :height ) ), Continuous Distribution( Column( :weight ) ) );

// Extract the stats
heightMean = Report( dis )["Height"]["Summary Statistics"][Number Col Box( 1 )][1];
weightMean = Report( dis )["Weight"]["Summary Statistics"][Number Col Box( 1 )][1];

// Create the new display object
hlb = H List Box(
	Text Box( "Mean of Height = " ),
	Number Edit Box( heightMean ),
	Spacer Box( size( 30, 0 ) ),
	Text Box( "Mean of Weight = " ),
	Number Edit Box( weightMean )

);

// Add the new object to the plot
Report(Biv)[Outline Box(1)] << Append( hlb );

Dis << close window;

Re: How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg — Thu, 28 Sep 2017 22:40:51 GMT

Thanks Jim. A cocnrete example like this is what I was looking for. I've already reverted to Python, but I will give your JSL example a try later.