Intraclass Correlation of 1 in JMP?

[#1]

Hello.  I am trying to calculate intraclass correlation to estimate interrater reliability for some continuous variables.  I have two reviewers that coded the numbers of different types of comments in 20 of the same survey papers.  Under Measurement Systems Analysis (JMP 11.0.0 Mac), I entered the continuous variable (number of comments) under Y, Response.  I placed Reviewer Name under X, Grouping, and I placed Survey Number under Part, Sample ID. 

When I click "OK" I get a message saying "Not enough data to compute the process standard deviation. Disabling Options that require standard deviation" and when I check EMP Results, I see an ICC (no bias) of 1, ICC (with bias) of 0.9958, and ICC (with bias and interactions) of 0.8539.  I can't figure out why the first ICC value would be 1.  The only thing I have inferred is that perhaps that ICC is in fact telling me intrarater reliability, and since each rater only reviewed each paper one time, there is no variation in the way each individual rater coded each paper.  From what I read, though, I did not see anything that explicitly explained the first ICC value as strictly an intrarater/intraoperator value.

Thanks for any tips on how to interpret these numbers and which, if any, is appropriate for my question of interrater reliability!

[#2]

Since each reviewer evaluated each paper only once, the MSA platform is probably not the best place to try and evaluate concordance between reviewers. I suggest using the Multivariate platform and do a pairwise comparison on each paper's 'code'. Maybe using Kendall's tau as a measure of association for the two reviewers in the nonparametric hot spot?

[#3 · Accepted solution]

Hi jsjmp,

Let me expand a little on what peter.bartell said. Intraclass correlation for interrater reliability is a tricky topic -- there are several different models, depending on the assumptions you are willing to make. The key reference is by Shrout and Fliess (1979) -- Intraclass correlations: uses in assessing rate... [Psychol Bull. 1979] - PubMed - NCBI .

The basic idea is to build an appropriate ANOVA model in the Fit Model platform, with fixed and/or random effects as necessary. Based on the information you provided, it sounds like you probably want N of comments as Y, and judges and papers both modeled as random effects. The Intraclass correlation coefficient is easily calculated from the variance components output.