topic Re: Help with Reading a Connecting Letters Report in Discussions

Help with Reading a Connecting Letters Report

ataylor — Tue, 14 Jan 2020 19:49:09 GMT

For pairwise comparisons, I understand the core idea in the Connecting Letter Report, is that if two groups don't share a letter they are significantly different. In the example posted here, I don't understand why we need both of the two first columns --- the second column appears to make the first one redundant since the second column tells us that all pairs between grp_0, grp_1, grp_3, grp_4, grp_5 are not significant and the first column is just the subset of groups 3, 4, and 5, which is covered by the list above.

Also to confirm, is it a correct interpretation to say this report tells us that the only pairing that are significant are grp_2/grp_3 and grp_2/grp_5?

Re: Help with Reading a Connecting Letters Report

Dan_Obermiller — Tue, 14 Jan 2020 20:10:47 GMT

Most of the time when you see a connecting letters report that seems "nonsensical", the groups typically have different sample sizes and/or the assumption of equal variances has been violated. Are either of these the case for your data? Also, the type of multiple comparison test that you used can play a part on this, too. Which test were you using?

Re: Help with Reading a Connecting Letters Report

ataylor — Tue, 14 Jan 2020 21:04:48 GMT

This is actually coming from the Games-Howell plugin (so the unequal sample sizes and unequal variance should be handled). We have some proprietary data where we have evidence the letters report in that plugin has bugs/errors. I generated these data specifically to illustrate that and wanted to make sure that my understanding of this being wrong is correct before I post about there being errors in the plugin.

Re: Help with Reading a Connecting Letters Report

ataylor — Tue, 14 Jan 2020 21:20:06 GMT

And interestingly, if I run Tukey in the main program on the same data, the letters report matches the p-values returned and does not have these oddities, so it is definitely something going on in the Games-Howell plugin.

Re: Help with Reading a Connecting Letters Report

ataylor — Wed, 15 Jan 2020 00:37:14 GMT

I decided my interpretation was correct that this table shows there is some sort of bug/error in the Games-Howell plugin and I've posted in the comments for that plugin about this issue.