Why don't i get significant letters even though my P value is <0.05?

[#1]

Hello everyone!

I am running a regression analysis and even though my P-value is less than 0.05 (0.0385 in my case), the letters are all A, showing no significance. How can i get JMP to recognize the significance? Any experience here?

Best regards

Haris

[#2]

Hi Haris: Can you provide more details about the model, the p-value (for what null hypothesis?), and what you mean by "letters are all A"?

kind regards,

Mark

[#3]

Please see below

[#4]

Hi @xaris19801001,

Looking at your Leverage Plots (jmp.com) for treatment, you can see that confidence curve is asymptotic to horizontal line, so it's borderline in terms of statistical significance (hence the p-value 0.0385 close to threshold 0.05 you have chosen). This "borderline" situation can also be seen when looking on the left at your "Effect tests", with "Treatment" in red showing its value close to 0.05.

Statistics is rarely a black or white situation, and p-values are helps to find best compromise in your decision/conclusion. Here the situation is not well defined, which might be due to data itself, or the choice of the test (Tukey-Kramer):

• Have you checked the assumptions are valid for parametric test (like Tukey-Kramer) : independence of observations, data are (quasi)-normally distributed, and homogeneity of variances for the groups ?
• Are you interested in all comparisons, or do you have a "control" (in this case a Dunnett's test might be more appropriate) or "best" group you want to compare to ? 

Depending on the choice of the test, the significativity threshold (fixed BEFORE the experiment, not during/after to have the conclusion you would like to have), its assumptions and the number of comparisons, you may have different results.

Hope this first answer will help you, looking forward to other responses from members

[#5]

Remember that each of these tests is independent so significance or lack of it in one test does not guarantee a result in another test. I will use your stated level of significance, alpha = 0.05, for my reply. The effect test is based on a term in the model. The comparison of means is based on individual means and an adjusted quantile to account for the number of multiple comparisons to keep the overall coverage of the test at 95%. The adjustment produces a larger quantile (multiplier) than would be used for a single two-sample t-test. So the confidence intervals for each of the comparisons is wider, and they all include zero.

You might consider using a contrast for your post hoc testing. I notice that the first two treatments produce similar means and the last two treatments produce similar means but they are different from the first two means. So you could test the difference between the first two and the last two.

[#6]

t would help if you post a picture of what you see so we can see it, too.

[#7]

Please see here. P value is 0.0385, it is red, but all the letters below are A.