BookmarkSubscribeRSS Feed
Choose Language Hide Translation Bar
bky

Community Member

Joined:

Jan 5, 2019

Weighted Kappas

Hi, we have a study where two raters are rating a 50 items for quality on a scale of 1-4. We wanted to do a weighted kappa in JMP; I entered the data, set the data type as ordinal, but JMP seems to only produce a simple Kappa when I Fit Y by X and look at the Agreement Statistic. 

 

Is JMP capable of calculating a weighted kappa?

 

Thanks in advance!

1 REPLY 1
Highlighted
gzmorgan0

Super User

Joined:

Jul 25, 2016

Re: Weighted Kappas

I did not find a buit-in solution to your question.  Of course JMP is capable, but it needs a script (unless one of the JMP developers has an alternate solution).

 

You did not report which version of JMP and which operating system you have, so I am not sure this will work for you. Attached is a script that I have tested on JMP 12,13 and 14.  It uses methods to be backward compatible with JMP12.  It :

  • Runs on the current data table and calls up a Contingency analysis.  In other words, it uses the JMP UI. 
  • The script is written for one contingency report, that is, one Y and one X ordinal variable.
  • Specifies Expected values to be included in the contingency table to reduce the complexity (and risk) of scripting errors.
  • Makes a table from the Contingency table.
  • From that, it creates the observed, and expected probabilities.
  • It calculates the standard kappa where the weight matrix is the Identity matrix; a Linear weight 1-  abs(i - j)/(k-1);  and a quadratic weight which is 1 - square( (i-j)/(k-1) ). See the references reported in the script.
  • The value of k, kL and kQ are appended to the JMP report (see below).
  • The calculated kappa is reported as a check since it should match the kappa reported by JMP
  • Weighted errors and Confidence Intervals are not calculated in this script. 

This might be a wish list item for future JMP versions. Since some references for weighted kappa include more than Linear and Quadratic weights for ordinal categories, you should specify if this wish list item should allow for custom weights.

 

The script is attached and a sample data set where column n should be used as Freq. slide58.jmp is a summary table from the third reported reference (page 58). It is one of 10 tables used to test the script.  I hope this helps.

 

image.png