topic Re: Weighted Kappas in Discussions

Weighted Kappas

bky — Sat, 05 Jan 2019 21:48:51 GMT

Hi, we have a study where two raters are rating a 50 items for quality on a scale of 1-4. We wanted to do a weighted kappa in JMP; I entered the data, set the data type as ordinal, but JMP seems to only produce a simple Kappa when I Fit Y by X and look at the Agreement Statistic.

Is JMP capable of calculating a weighted kappa?

Thanks in advance!

Re: Weighted Kappas

gzmorgan0 — Tue, 08 Jan 2019 10:50:20 GMT

I did not find a buit-in solution to your question. Of course JMP is capable, but it needs a script (unless one of the JMP developers has an alternate solution).

You did not report which version of JMP and which operating system you have, so I am not sure this will work for you. Attached is a script that I have tested on JMP 12,13 and 14. It uses methods to be backward compatible with JMP12. It :

Runs on the current data table and calls up a Contingency analysis. In other words, it uses the JMP UI.
The script is written for one contingency report, that is, one Y and one X ordinal variable.
Specifies Expected values to be included in the contingency table to reduce the complexity (and risk) of scripting errors.
Makes a table from the Contingency table.
From that, it creates the observed, and expected probabilities.
It calculates the standard kappa where the weight matrix is the Identity matrix; a Linear weight 1- abs(i - j)/(k-1); and a quadratic weight which is 1 - square( (i-j)/(k-1) ). See the references reported in the script.
The value of k, kL and kQ are appended to the JMP report (see below).
The calculated kappa is reported as a check since it should match the kappa reported by JMP
Weighted errors and Confidence Intervals are not calculated in this script.

This might be a wish list item for future JMP versions. Since some references for weighted kappa include more than Linear and Quadratic weights for ordinal categories, you should specify if this wish list item should allow for custom weights.

The script is attached and a sample data set where column n should be used as Freq. slide58.jmp is a summary table from the third reported reference (page 58). It is one of 10 tables used to test the script. I hope this helps.

Re: Weighted Kappas

ron_horne — Sun, 21 Feb 2021 18:15:16 GMT

thank you @gzmorgan0 for the script,

For some reason i get an error when running the script. i am using JMP 14.2.

any clue how to fix it?

Re: Weighted Kappas

gzmorgan0 — Tue, 23 Feb 2021 09:34:16 GMT

@ron_horne, hello.

I am on JMP 14.3. Could you send an example, an anonymized data table. Before declaring it a version issue, I'd like to check it out. My script had a few checks included, however, I could have missed a corner case.

Re: Weighted Kappas

gzmorgan0 — Tue, 23 Feb 2021 09:40:13 GMT

@ron_horne,

To be specific, I use JMP PRO. I tested on JMP PRO 13.2.1 and got the same results.

Re: Weighted Kappas

ron_horne — Tue, 23 Feb 2021 11:45:46 GMT

@gzmorgan0 ,

this is what i get using the example file Slide58, not my own data. it also produces the same error on JMP11 and JMP 15.

all i did was open the file and run the script.

thank you,

Ron

Re: Weighted Kappas

ron_horne — Tue, 23 Feb 2021 15:50:18 GMT

@gzmorgan0

for example this data set. each column has a different assessment of the same scale.

please do let me know if you mange to run it using the script.

many thanks,

ron

Re: Weighted Kappas

gzmorgan0 — Fri, 26 Feb 2021 10:25:26 GMT

Hi Ron, @ron_horne .

Sorry for the late reply. I ran the weigthed Kappa script while your subset table was the current data table. I specified BP1 as Y and BP2 as X. the results are attached as a .jrn file. Below is a snippet.

When you run the script:

does the contingency table contain Total %, Col %, Row % and Expected?
I suggest you comment out line 88 of the script that deletes the table made from the contingency table. Then run the script. Does a table get created and does it look like this?

The error you reported appears that it cannot execute cmat = dt[idx,ccols]; idx is an index of rows where Cell == "Count" or later "Expected" and ccols are continuous columns which should be the list {"0", "0.25", "0.5", "0.75", "1"}. If the Cell names are not the same, maybe an extra space or something it could throw off the script, maybe a more robust solutions is to use where Contains(:Cell, "Count")

Just guessing here.

Re: Weighted Kappas

ron_horne — Fri, 26 Feb 2021 11:42:17 GMT

Thank you @gzmorgan0 ,

I do manage to run it row by row but not as a function. That is the big surprise. in any case i do manage to produce the actual numbers and i do not need to do this many times so i am fine.

Many thanks,

ron

Re: Weighted Kappas

ghartel — Thu, 05 Aug 2021 22:50:59 GMT

Hi All,

I couldn't get this script to run as is and anyway I wanted a function to use within scripts rather than something interactive, so I adapted this code to create a function. I also added calculation of standard errors and asymptotic confidence intervals and dealing with BY variables.

To use it just include it in your script, run a contingency analysis and assign it to a variable and then pass it to the function. The function will add an outline item to each Kappa table giving the linear and quadratic weighted Kappas. Let me know if you discover any errors.

Include( Udir || "\Weighted Kappa Function.jsl" );

CT = Contingency( Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );
CT2 = Contingency( by( :Rater), Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );

WeightedKappa(CT);
WeightedKappa(CT2);s

Re: Weighted Kappas

lcmaley — Tue, 05 Mar 2024 02:27:13 GMT

I am trying to use your script, I put this on the end,

dt0 = Current Data Table();

CT = dt0 << Contingency(
Contingency Table( Expected( 1 )), Agreement Statistic( 1 ), Tests(0) );

WeightedKappa( CT );

And it is not working, nothing is being appended. What am I doing wrong?

I get an error saying it is not a contingency table, then nothing is appended. Suggestions?

Re: Weighted Kappas

gzmorgan0 — Tue, 05 Mar 2024 06:16:34 GMT

@lcmaley ,

I had to brush off some rust, since I have not looked at my script since for multiple years.

You should always mentions which version of JMP you are using. That said, I suggest you use @ghartel 's solution posted in 2021.

Download the Weighted Kappa Function.jsl that was provided in the @ghartel post, and save it to a drive for your scripts.

Note, the Weighted Kappa Function solution includes the Std Err and confidence limits, as well as, a By variable; my script did not. @ghartel 's solution is a much better solution.

Below is the JSL I used to run the Weighted Kappa Function, for the sample table provided with my original script. See the attached screenshot of the result for the JMP table slide58.jmp.

Note:

Udir can be any drive you wish and the data table and function can be in different directories.
You do not need to call out each option for the Contingency, likely, you have defaults set up. However, Counts and Expected need to be included in your Contingency Platform options or called out in your script.

Kudos to @ghartel for the terrific function!

Names Default to Here(1);
Udir ="C:\temp";

dt = open(Udir || "\slide58.jmp");

Include( Udir || "\Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	Y( :Y ),
	X( :X ),
	Freq( :n ),
	Contingency Table(
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col Cum( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col Cum %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row Cum( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row Cum %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa(CT);

Re: Weighted Kappas

gzmorgan0 — Tue, 05 Mar 2024 06:17:31 GMT

See attachment

Re: Weighted Kappas

lcmaley — Tue, 05 Mar 2024 14:50:55 GMT

Sorry, I am using JMP Pro 17.0.0 on a Mac. I tried what you had but it kept duplicating the path. So I changed it to:

dt = open("Subset of Data for Kappa index.jmp");

Include("Weighted Kappa.jsl" );

Which does open the file (I am trying the file on here first, mine is a little more complicated and I wanted to make sure it worked with the file on here), but then I get an error like:

Weighted Kappa.jsl includes itself through a closed path.:

Weighted Kappa.jsl in access or evaluation of 'Include' ,

Include/*###*/("Weighted Kappa.jsl")

Re: Weighted Kappas

ghartel — Wed, 06 Mar 2024 10:55:10 GMT

you need to say

include("Weighted Kappa Function.jsl");

Re: Weighted Kappas

lcmaley — Thu, 07 Mar 2024 16:07:41 GMT

I did try that at first and it did not work. I changed it to without function because that was the name of my script. I changed the script to match Weighted Kappa Function.jsl and it still does not work. I am getting the same error

Weighted Kappa.jsl includes itself through a closed path.:

Weighted Kappa.jsl in access or evaluation of 'Include' ,

Include/*###*/("Weighted Kappa Function.jsl")

Here is my code I am putting with the function:

Names Default to Here(1);

dt = open("Subset of Data for Kappa index.jmp");

Include("Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	
	Contingency Table(
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col cumulative %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row cumulative %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa(CT);

Re: Weighted Kappas

ghartel — Thu, 07 Mar 2024 23:25:25 GMT

are you actually getting a contingency table? I don't see you specifying an X and a Y in your table. The following code works on my system (V17.2.0, Windows 10)

Names Default To Here( 1 );
Include("Weighted Kappa Function.jsl" );
dt = Open( "$SAMPLE_DATA/Prime Minister Ratings.jmp" );
CT = dt << Contingency(
	Y( :Second Survey ),
	X( :First Survey ),
	Freq( :Count ),
	Tests( 0 ),
	Agreement Statistic( 1 )
);

WeightedKappa(CT);

Re: Weighted Kappas

lcmaley — Tue, 12 Mar 2024 12:45:05 GMT

Sorry, I was traveling. This way I do not get a contingency table. I tried it with the X and Y specified and did the same thing. I just took the X and Y out so I could use it with any table. It just brings up the dialog box for me to enter the X and Y.

dt = Open( "Subset of Data for Kappa index.jmp" );

Include( "Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	
	Contingency Table(
		Y( :BP1),
		X( :BP2 ),
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col cumulative %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row cumulative %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa( CT );

I do get a contingency table when I run this, but nothing is appended:

dt0 = Current Data Table();

tsub = Subscribe to Data Table List( , On Open( WeightedKappa ) );

reportc = Contingency(
	Contingency Table( Expected( 1 ) ),
	Agreement Statistic( 1 ),
	<<Make Into Data Table
);


Unsubscribe to Data Table List( tsub );

Re: Weighted Kappas

MeanBasilisk816 — Wed, 26 Jun 2024 00:05:15 GMT

I tried the exact same code and it worked up to reporting the contingency table, but it showed "Wrong dimensionnally" and did not calculate the weighted kappa. (JMP18.01, Win11)

Re: Weighted Kappas

MeanBasilisk816 — Wed, 26 Jun 2024 00:44:47 GMT

It seems to be a language problem with JMP. I modified "Cell" in the script to Japanese and it works. Thanks for the great job.