Weighted Kappas

[#1]

Hi, we have a study where two raters are rating a 50 items for quality on a scale of 1-4. We wanted to do a weighted kappa in JMP; I entered the data, set the data type as ordinal, but JMP seems to only produce a simple Kappa when I Fit Y by X and look at the Agreement Statistic. 

Is JMP capable of calculating a weighted kappa?

Thanks in advance!

[#2]

I am trying to use your script, I put this on the end,

dt0 = Current Data Table();

CT = dt0 << Contingency(
Contingency Table( Expected( 1 )), Agreement Statistic( 1 ), Tests(0) );

WeightedKappa( CT );

And it is not working, nothing is being appended.   What am I doing wrong?

I get an error saying it is not a contingency table, then nothing is appended.  Suggestions?  

[#3]

@lcmaley ,

I had to brush off some rust, since I have not looked at my script since for multiple years.

You should always mentions which version of JMP you are using. That said, I suggest you use @ghartel 's solution posted in 2021.

Download the Weighted Kappa Function.jsl that was provided in the @ghartel  post, and save it to a drive for your scripts.

Note, the Weighted Kappa Function solution includes the Std Err and confidence limits, as well as, a By variable; my script did not. @ghartel 's solution is a much better solution.

Below is the JSL I used to run the Weighted Kappa Function, for the sample table provided with my original script.   See the attached screenshot of the result for the JMP table slide58.jmp.

Note:

• Udir can be any drive you wish and the data table and function can be in different directories.
• You do not need to call out each option for the Contingency, likely, you have defaults set up. However, Counts and Expected need to be included in your Contingency Platform options or called out in your script.

Kudos to @ghartel for the terrific function!

Names Default to Here(1);
Udir ="C:\temp";

dt = open(Udir || "\slide58.jmp");

Include( Udir || "\Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	Y( :Y ),
	X( :X ),
	Freq( :n ),
	Contingency Table(
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col Cum( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col Cum %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row Cum( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row Cum %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa(CT);

[#4]

See attachment

[#5]

Sorry, I am using JMP Pro 17.0.0 on a Mac.   I tried what you had but it kept duplicating the path.  So I changed it to:

dt = open("Subset of Data for Kappa index.jmp");

Include("Weighted Kappa.jsl" );

Which does open the file (I am trying the file on here first, mine is a little more complicated and I wanted to make sure it worked with the file on here), but then I get an error like:

Weighted Kappa.jsl includes itself through a closed path.:

 Weighted Kappa.jsl in access or evaluation of 'Include' , 

Include/*###*/("Weighted Kappa.jsl")

[#6]

you need to say

include("Weighted Kappa Function.jsl");

[#7]

I did try that at first and it did not work.  I changed it to without function because that was the name of my script.   I changed the script to match Weighted Kappa Function.jsl and it still does not work.  I am getting the same error

Weighted Kappa.jsl includes itself through a closed path.:

 Weighted Kappa.jsl in access or evaluation of 'Include' , 

Include/*###*/("Weighted Kappa Function.jsl")

  Here is my code I am putting with the function:

Names Default to Here(1);

dt = open("Subset of Data for Kappa index.jmp");

Include("Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	
	Contingency Table(
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col cumulative %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row cumulative %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa(CT);

[#8]

are you actually getting a contingency table? I don't see you specifying an X and a Y in your table. The following code works on my system (V17.2.0, Windows 10)

Names Default To Here( 1 );
Include("Weighted Kappa Function.jsl" );
dt = Open( "$SAMPLE_DATA/Prime Minister Ratings.jmp" );
CT = dt << Contingency(
	Y( :Second Survey ),
	X( :First Survey ),
	Freq( :Count ),
	Tests( 0 ),
	Agreement Statistic( 1 )
);

WeightedKappa(CT);

[#9]

Sorry, I was traveling.  This way I do not get a contingency table.  I tried it with the X and Y specified and did the same thing.  I just took the X and Y out so I could use it with any table.  It just brings up the dialog box for me to enter the X and Y.

dt = Open( "Subset of Data for Kappa index.jmp" );

Include( "Weighted Kappa Function.jsl" );

CT = dt << Contingency(
	
	Contingency Table(
		Y( :BP1),
		X( :BP2 ),
		Count( 1, Format( "Fixed Dec", 8, 0 ) ),
		Total %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Col %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Row %( 1, Format( "Fixed Dec", 8, 2 ) ),
		Expected( 1, Format( "Best", 8 ) ),
		Deviation( 0, Format( "Best", 8 ) ),
		Cell Chi Square( 0, Format( "Fixed Dec", 8, 4 ) ),
		Col cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Col cumulative %( 0, Format( "Fixed Dec", 8, 2 ) ),
		Row cumulative( 0, Format( "Fixed Dec", 8, 0 ) ),
		Row cumulative %( 0, Format( "Fixed Dec", 8, 2 ) )
	),
	Agreement Statistic( 1 )
);


WeightedKappa( CT );

I do get a contingency table when I run this, but nothing is appended:

dt0 = Current Data Table();

tsub = Subscribe to Data Table List( , On Open( WeightedKappa ) );

reportc = Contingency(
	Contingency Table( Expected( 1 ) ),
	Agreement Statistic( 1 ),
	<<Make Into Data Table
);


Unsubscribe to Data Table List( tsub );