Weighted Kappas

[#1]

Hi, we have a study where two raters are rating a 50 items for quality on a scale of 1-4. We wanted to do a weighted kappa in JMP; I entered the data, set the data type as ordinal, but JMP seems to only produce a simple Kappa when I Fit Y by X and look at the Agreement Statistic. 

Is JMP capable of calculating a weighted kappa?

Thanks in advance!

[#2]

I did not find a buit-in solution to your question.  Of course JMP is capable, but it needs a script (unless one of the JMP developers has an alternate solution).

You did not report which version of JMP and which operating system you have, so I am not sure this will work for you. Attached is a script that I have tested on JMP 12,13 and 14.  It uses methods to be backward compatible with JMP12.  It :

• Runs on the current data table and calls up a Contingency analysis.  In other words, it uses the JMP UI. 
• The script is written for one contingency report, that is, one Y and one X ordinal variable.
• Specifies Expected values to be included in the contingency table to reduce the complexity (and risk) of scripting errors.
• Makes a table from the Contingency table.
• From that, it creates the observed, and expected probabilities.
• It calculates the standard kappa where the weight matrix is the Identity matrix; a Linear weight 1-  abs(i - j)/(k-1);  and a quadratic weight which is 1 - square( (i-j)/(k-1) ). See the references reported in the script.
• The value of k, kL and kQ are appended to the JMP report (see below).
• The calculated kappa is reported as a check since it should match the kappa reported by JMP
• Weighted errors and Confidence Intervals are not calculated in this script. 

This might be a wish list item for future JMP versions. Since some references for weighted kappa include more than Linear and Quadratic weights for ordinal categories, you should specify if this wish list item should allow for custom weights.

The script is attached and a sample data set where column n should be used as Freq. slide58.jmp is a summary table from the third reported reference (page 58). It is one of 10 tables used to test the script.  I hope this helps.

  

[#3]

thank you @gzmorgan0 for the script,

For some reason i get an error when running the script. i am using JMP 14.2.

any clue how to fix it?

[#4]

@ron_horne, hello.

I am on JMP 14.3. Could you send an example, an anonymized data table.  Before declaring it a version issue, I'd like to check it out. My script had a few checks included, however, I could have missed a corner case.

[#5]

@ron_horne,

To be specific, I use JMP PRO. I tested on JMP PRO 13.2.1 and got the same results.

[#6]

@gzmorgan0 ,

this is what i get using the example file Slide58, not my own data. it also produces the same error on JMP11 and JMP 15.

all i did was open the file and run the script.

thank you,

Ron

[#7]

@gzmorgan0 

for example this data set. each column has a different assessment of the same scale.

please do let me know if you mange to run it using the script.

many thanks,

ron

[#8]

Hi Ron, @ron_horne .

Sorry for the late reply. I ran the weigthed Kappa script while your subset table was the current data table. I specified BP1 as Y and BP2 as X. the results are attached as a .jrn file. Below is a snippet.

When you run the script:

• does the contingency table contain Total %, Col %,  Row % and Expected? 
• I suggest you comment out line 88 of the script that deletes the table made from the contingency table. Then run the script.  Does a table get created and does it look like this?

The error you reported appears that it cannot execute  cmat = dt[idx,ccols];  idx is an index of rows where Cell == "Count"  or later "Expected" and ccols are continuous columns which should be the list {"0", "0.25", "0.5", "0.75", "1"}. If the Cell names are not the same, maybe an extra space or something it could throw off the script, maybe a more robust solutions is to use where Contains(:Cell, "Count")

Just guessing here.

[#9]

Thank you @gzmorgan0 ,

I do manage to run it row by row but not as a function. That is the big surprise. in any case i do manage to produce the actual numbers and i do not need to do this many times so i am fine.

Many thanks,

ron

[#10]

Hi All,

I couldn't get this script to run as is and anyway I wanted a function to use within scripts rather than something interactive, so I adapted this code to create a function. I also added calculation of standard errors and asymptotic confidence intervals and dealing with BY variables.

To use it just include it in your script, run a contingency analysis and assign it to a variable and then pass it to the function. The function will add an outline item to each Kappa table giving the linear and quadratic weighted Kappas. Let me know if you discover any errors.

Include( Udir || "\Weighted Kappa Function.jsl" );

CT = Contingency( Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );
CT2 = Contingency( by( :Rater), Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );

WeightedKappa(CT);
WeightedKappa(CT2);s