topic Re: Why don't i get significant letters even though my P value is <0.05? in Discussions

Why don't i get significant letters even though my P value is <0.05?

xaris19801001 — Fri, 09 Jun 2023 00:57:00 GMT

Hello everyone!

I am running a regression analysis and even though my P-value is less than 0.05 (0.0385 in my case), the letters are all A, showing no significance. How can i get JMP to recognize the significance? Any experience here?

Best regards

Haris

Re: Why don't i get significant letters even though my P value is <0.05?

MRB3855 — Thu, 10 Nov 2022 11:19:29 GMT

Hi Haris: Can you provide more details about the model, the p-value (for what null hypothesis?), and what you mean by "letters are all A"?

kind regards,

Mark

Re: Why don't i get significant letters even though my P value is <0.05?

Mark_Bailey — Thu, 10 Nov 2022 14:52:03 GMT

t would help if you post a picture of what you see so we can see it, too.

Re: Why don't i get significant letters even though my P value is <0.05?

xaris19801001 — Mon, 14 Nov 2022 08:23:37 GMT

Please see here. P value is 0.0385, it is red, but all the letters below are A.

Re: Why don't i get significant letters even though my P value is <0.05?

xaris19801001 — Mon, 14 Nov 2022 08:24:34 GMT

Please see below

Re: Why don't i get significant letters even though my P value is <0.05?

Victor_G — Mon, 14 Nov 2022 12:06:54 GMT

Hi @xaris19801001,

Looking at your Leverage Plots (jmp.com) for treatment, you can see that confidence curve is asymptotic to horizontal line, so it's borderline in terms of statistical significance (hence the p-value 0.0385 close to threshold 0.05 you have chosen). This "borderline" situation can also be seen when looking on the left at your "Effect tests", with "Treatment" in red showing its value close to 0.05.

Statistics is rarely a black or white situation, and p-values are helps to find best compromise in your decision/conclusion. Here the situation is not well defined, which might be due to data itself, or the choice of the test (Tukey-Kramer):

Have you checked the assumptions are valid for parametric test (like Tukey-Kramer) : independence of observations, data are (quasi)-normally distributed, and homogeneity of variances for the groups ?
Are you interested in all comparisons, or do you have a "control" (in this case a Dunnett's test might be more appropriate) or "best" group you want to compare to ?

Depending on the choice of the test, the significativity threshold (fixed BEFORE the experiment, not during/after to have the conclusion you would like to have), its assumptions and the number of comparisons, you may have different results.

Hope this first answer will help you, looking forward to other responses from members :)

Re: Why don't i get significant letters even though my P value is <0.05?

Mark_Bailey — Mon, 14 Nov 2022 20:00:13 GMT

Remember that each of these tests is independent so significance or lack of it in one test does not guarantee a result in another test. I will use your stated level of significance, alpha = 0.05, for my reply. The effect test is based on a term in the model. The comparison of means is based on individual means and an adjusted quantile to account for the number of multiple comparisons to keep the overall coverage of the test at 95%. The adjustment produces a larger quantile (multiplier) than would be used for a single two-sample t-test. So the confidence intervals for each of the comparisons is wider, and they all include zero.

You might consider using a contrast for your post hoc testing. I notice that the first two treatments produce similar means and the last two treatments produce similar means but they are different from the first two means. So you could test the difference between the first two and the last two.