cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Browse apps to extend the software in the new JMP Marketplace
Choose Language Hide Translation Bar
bky
bky
Level I

Weighted Kappas

Hi, we have a study where two raters are rating a 50 items for quality on a scale of 1-4. We wanted to do a weighted kappa in JMP; I entered the data, set the data type as ordinal, but JMP seems to only produce a simple Kappa when I Fit Y by X and look at the Agreement Statistic. 

 

Is JMP capable of calculating a weighted kappa?

 

Thanks in advance!

19 REPLIES 19
gzmorgan0
Super User (Alumni)

Re: Weighted Kappas

I did not find a buit-in solution to your question.  Of course JMP is capable, but it needs a script (unless one of the JMP developers has an alternate solution).

 

You did not report which version of JMP and which operating system you have, so I am not sure this will work for you. Attached is a script that I have tested on JMP 12,13 and 14.  It uses methods to be backward compatible with JMP12.  It :

  • Runs on the current data table and calls up a Contingency analysis.  In other words, it uses the JMP UI. 
  • The script is written for one contingency report, that is, one Y and one X ordinal variable.
  • Specifies Expected values to be included in the contingency table to reduce the complexity (and risk) of scripting errors.
  • Makes a table from the Contingency table.
  • From that, it creates the observed, and expected probabilities.
  • It calculates the standard kappa where the weight matrix is the Identity matrix; a Linear weight 1-  abs(i - j)/(k-1);  and a quadratic weight which is 1 - square( (i-j)/(k-1) ). See the references reported in the script.
  • The value of k, kL and kQ are appended to the JMP report (see below).
  • The calculated kappa is reported as a check since it should match the kappa reported by JMP
  • Weighted errors and Confidence Intervals are not calculated in this script. 

This might be a wish list item for future JMP versions. Since some references for weighted kappa include more than Linear and Quadratic weights for ordinal categories, you should specify if this wish list item should allow for custom weights.

 

The script is attached and a sample data set where column n should be used as Freq. slide58.jmp is a summary table from the third reported reference (page 58). It is one of 10 tables used to test the script.  I hope this helps.

 

image.png  

 

ron_horne
Super User (Alumni)

Re: Weighted Kappas

thank you @gzmorgan0 for the script,

For some reason i get an error when running the script. i am using JMP 14.2.

any clue how to fix it?

ron_horne_0-1613931253805.png

 

gzmorgan0
Super User (Alumni)

Re: Weighted Kappas

@ron_horne, hello.

 

I am on JMP 14.3. Could you send an example, an anonymized data table.  Before declaring it a version issue, I'd like to check it out. My script had a few checks included, however, I could have missed a corner case.

 

 

 

gzmorgan0
Super User (Alumni)

Re: Weighted Kappas

@ron_horne,

 

To be specific, I use JMP PRO. I tested on JMP PRO 13.2.1 and got the same results.

ron_horne
Super User (Alumni)

Re: Weighted Kappas

@gzmorgan0 ,

this is what i get using the example file Slide58, not my own data. it also produces the same error on JMP11 and JMP 15.

all i did was open the file and run the script.

 

thank you,

Ron

 

ron_horne
Super User (Alumni)

Re: Weighted Kappas

@gzmorgan0 

for example this data set. each column has a different assessment of the same scale.

please do let me know if you mange to run it using the script.

many thanks,

ron

 

 

gzmorgan0
Super User (Alumni)

Re: Weighted Kappas

Hi Ron, @ron_horne .

 

Sorry for the late reply. I ran the weigthed Kappa script while your subset table was the current data table. I specified BP1 as Y and BP2 as X. the results are attached as a .jrn file. Below is a snippet.

image.png

When you run the script:

  • does the contingency table contain Total %, Col %,  Row % and Expected? 
  • I suggest you comment out line 88 of the script that deletes the table made from the contingency table. Then run the script.  Does a table get created and does it look like this?

image.png

The error you reported appears that it cannot execute  cmat = dt[idx,ccols];  idx is an index of rows where Cell == "Count"  or later "Expected" and ccols are continuous columns which should be the list {"0", "0.25", "0.5", "0.75", "1"}. If the Cell names are not the same, maybe an extra space or something it could throw off the script, maybe a more robust solutions is to use where Contains(:Cell, "Count")

 

Just guessing here.

 

ron_horne
Super User (Alumni)

Re: Weighted Kappas

Thank you @gzmorgan0 ,

I do manage to run it row by row but not as a function. That is the big surprise. in any case i do manage to produce the actual numbers and i do not need to do this many times so i am fine.

 

Many thanks,

ron

 

 

ghartel
Level III

Re: Weighted Kappas

Hi All,

I couldn't get this script to run as is and anyway I wanted a function to use within scripts rather than something interactive, so I adapted this code to create a function. I also added calculation of standard errors and asymptotic confidence intervals and dealing with BY variables.

To use it just include it in your script, run a contingency analysis and assign it to a variable and then pass it to the function. The function will add an outline item to each Kappa table giving the linear and quadratic weighted Kappas. Let me know if you discover any errors.

Include( Udir || "\Weighted Kappa Function.jsl" );

CT = Contingency( Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );
CT2 = Contingency( by( :Rater), Y( :Comparison ), X( :Reference ), Contingency Table(), Tests( 0 ) );

WeightedKappa(CT);
WeightedKappa(CT2);s