Variability in Responses

[#1]

Hi All,

I am trying to analyze a data set in which 15 doctors all ranked images of a disease on a scale of 0-9. There were 110 images. I want to look at the variability of the doctors/determine how good they are at coming to a similar ranking. Any idea on how to go about this?

[#2 · Accepted solution]

The Distribution platform is the 'go to' platform for looking at each response one at a time. Fit Y by X could be very useful for looking across the evaluators. Much can also be done in Graph Builder with things like data filters. All good analysis starts with a clear articulation  of the practical problems you would like to address...then find the appropriate analysis methods to answer those questions.

[#3 · Accepted solution]

Adding to Peter's advice, you can analyze your data using the Contingency platform. First be sure that the Rating response is set to use the Ordinal modeling type. The Rater data column that identifies the doctor should use the Nominal modeling type. Select Analyze > Fit Y by X. Assign Rating to the Y role and Rater to the X role and click OK. Click the red triangle next to Contingency and select Agreement Statistic. The kappa statistic indicates the strength of the agreement. Kappa close to zero indicates poor agreement and kappa close to one indicates good agreement.

[#4]

Thanks for the advice! So I was able to get contingency tables, but do not see agreement statistics. I posted a file of the two options that come up when clicking the red triangle. One is for the red triangle for contingency analysis, and one is for contigency table. 

Hope i'm not being confusing! I appreciate the help i've received. 

[#5]

Nevermind. For some reason, when I ran it again, it worked as you said. Thanks!