How to Interpret Overlapping Letter Tukey's HSD?

[#1]

I am having a hard time explaining to myself (thus others) how to interpret a Tukey's HSD test between 3 treatment groups, when the letter assignment goes like this:

I understand how there isn't enough of a difference between Level 1 subgroup and Level 3 subgroup means to say they are different when viewed pairwise, same with Level 2 and Level 3, while the difference between Level 1 and Level 2 is great enough they are considered different distributions. 

However, considering the situation 'as a whole', is it correct to say the Level 3 subgroup distribution is it's own entity because it appears to contain members both from the Level 1 distribution, and Level 2? In other words I can conclude each treatment level has done something 'unique' to each subgroup, because Level 3's distribution is a 'hybrid' of the other 2? 

I am trying to resolve a contradiction I see where if either Level 1 or Level 2 is excluded from the analysis, which could have happened in real life if the experiment budget was smaller (and those samples were never made), then Level 3 is judged 'insignificantly different' from either of the other two levels. It is only analyzing all 3 subgroups together Level 3's 'uniqueness' is determined. What 'is' the Level 3 distribution, with respect to the Level 1 and Level 2 distributions? 

[#2]

Hi,

In your example, you have 3 populations that could be ordered in ascending or descending order as Level 1 > Level 3 > Level 2

Levels 1 and 3 are not significantly different, and Levels 3 and 2 are not significantly different. However, Level 1 is significantly different from Level 2. 

Does it make sense?

Best,

TS

[#3]

Hey Thierry, 

Yes, that makes sense, but doesn't exactly reconcile what I was asking.

If I were to do this analysis with only either Level's 1 and 3 present, or Level's 2 and 3 present, ANOVA doesn't detect a difference between the populations. The differences associated between their distributions is attributed to noise in the measurement. I would then conclude that members of the Level 3 group are from the same population as either Level 1 or 2, depending on which group was included in the analysis. 

Seeing the significant difference between Level's 1 and 2 though, and concluding that they are from different populations, how then should I interpret what Level 3 is? I suppose it is sufficient to say, "upon learning more information about the system, I can now determine Level 3 is neither Level 1 or Level 2 but a 'combination' of both. It is a distinct population." Is that the correct interpretation? Without testing all 3 levels, I would have mistakenly concluded that the Level 3 large-N-limit distribution would equal either the Level 1 or Level 2 large-N-limit distributions when it wouldn't have? 

[#4 · Accepted solution]

I believe you are falling into the trap of accepting the null hypothesis. You stated that 

---

I would then conclude that members of the Level 3 group are from the same population as either Level 1 or 2, depending on which group was included in the analysis. 

No, that would only be the case if the null hypothesis were true.  Tukey's test, and indeed, most statistical tests will not declare equality. All you can say is that the difference between level 1 and level 3 is not significant at the 95% confidence level. So we should NOT declare them to have the same mean. (A minor point here is that the tests are only on the means under the assumption of equal variances and normality. It is not really a test on the population distribution). You can use the Ordered Differences report to see a confidence interval for the difference of the means. That will show you the range of possibilities on how far apart level 3 is from level 1 and level 2.

In this situation there are a number of ways to state the result. One way that has worked well for me in most situations:  You could say that we did not detect a significant difference between level 3 and level 1 at the 95% confidence level. But we did detect a significant difference between levels 1 and 2. Notice that careful wording. Not a significant difference between levels 1 and 3. There may still be a difference, but it is not significant. From your last paragraph, I would not say that level 3 is a blend of levels 1 and 2. It MIGHT be, but it could be it's own population yet. It is just too close to level 1 for us to detect the difference. It is too close to level 2 to be declared different. The noise is too large to see that small of a difference.

If you want to test for equality, you can use the Equivalence Test option that JMP has available. When setting that up you will need to specify what you mean by "equality".

Finally, I believe you know this, but I want to state that you should not subset the data into just the levels you want to test. If you want to compare all three levels, use the ANOVA approach as you did. If you subset and only compare level 1 to level 3, you are reducing the power of the test since you are using a smaller sample size.