How do I auto-calculate precision and F1 score for models on imbalanced data?

[#1]

I like that JMP's prints a confusion matrix and ROC for the logistic (Generalized) regression, but some important metrics seem to be missing from the results: precision and F1 score. Is there a way to auto-include these in the output? Or, have it be computed as "automatically" as possible when I re-run a new classificaiton model. I am trying to avoid manually recomputing these with each new model. 

UPDATE: Please post answers to the questions only. I am not looking for an opinion about whether I shall or shall not use Precision and F1. I am confidently looking for a way to compute these automatically in JMP. If you are not aware of such technique, there is no reason to contemplate whether this is doable or not, or post anything below. Thank you :)

[#2]

Hi, Oleg!

Perhaps it is an unfamiliarity with the language, but the word Discussions (the name of this site in the Community) mean much more than just questions and answers.  It can rightfully include opinions, admonitions, pleas, rants, and many other forms of dialog, irrespective of your desires.

One of the many things I love about statistics (and JMP!) is that the really great, inventive work seems to happen at the interface between the science of the applications and the art.  See?  My opinion.

[#3]

Hi Kevin. :)

If this "Discussion" thread is anything like StackOverflow, then I would expect clear answers to clear questions. If I am mistaken, feel free to guide me to the guidelines for posting to discussions. I can read other people's opinioins in news papers. However, in the context of science and in the spirit of "get it done", the clear and succinct answers are value-added.

Btw, did you just admit that I recieved opinions to my question? LOL.

Anyhow, it would be more effective to seek you help on this one. 

Have you come across any JSL examples that would help me script F1 and Precision, and, preferrably, added it to the regression output page?

Cheers!
O.

[#4]

Kevin, you may have misread my original question. I was hoping for a built-in computation of both F1 and precision. JSL is a reasonable alternative. 

Do you have an example I can follow?

[#5]

Hi, Oleg!

I don't think I misread your original question...but that's just my opinion...

Here's an example for you to try:

Take Big Class.jmp.  Run some ridiculously-overfit Neural Network analysis with Sex as the Response and Age, Height, and Weight as the factors.  Consider Female as the Positive case. :)

I got a Contingency Table that looks like this:

Choose Make Into Data Table.

TP= 16.  FP=2.  FN=2.

F1 can be calculated by 2*TP/(2*TP+FP+FN).  I get F1=0.89.

This is a very easy thing to script (in my opinion :), and you can even automatically include this metric (and any other) on any of your output with a script.  Give it a shot, and I'll try to help you get it if you fail.

Good luck.

[#6]

Kevin, thanks for the example.

I don't have a problem running a model and computing F1 and Precision scores by hand. But, is there an example to auto-compute these in JMP (which is what I asked originally)?

In the meantime, I coded these Python, which took less time than posting a question. If someone provides an answer in the future, I'll gladly give it another try.

[#7]

Examples are available in the File Exchange, the script examples installed when you install JMP, directly in the Scripting Guide book, also  installed when you install JMP, etc.  And on top of that, here is a sample script that reads from an output and displays it in a different form as part of an output.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA\Big Class.jmp" );

// Run the Fit Y by X to get the XY plot
Biv = dt << Bivariate( Y( :height ), X( :weight ) );

// Run the distributions to get the stats
Dis = dt << Distribution( Continuous Distribution( Column( :height ) ), Continuous Distribution( Column( :weight ) ) );

// Extract the stats
heightMean = Report( dis )["Height"]["Summary Statistics"][Number Col Box( 1 )][1];
weightMean = Report( dis )["Weight"]["Summary Statistics"][Number Col Box( 1 )][1];

// Create the new display object
hlb = H List Box(
	Text Box( "Mean of Height = " ),
	Number Edit Box( heightMean ),
	Spacer Box( size( 30, 0 ) ),
	Text Box( "Mean of Weight = " ),
	Number Edit Box( weightMean )

);

// Add the new object to the plot
Report(Biv)[Outline Box(1)] << Append( hlb );

Dis << close window;

[#8]

Thanks Jim. A cocnrete example like this is what I was looking for. I've already reverted to Python, but I will give your JSL example a try later.