One-tail vs. Two-tail tests

[#1]

Hi,

I am doing the means comparison using one-way ANOVA in Jmp. When I look at the detailed report, I see the p-values for both one-tail and two-tail t-tests.
However, the comparison circles seem to display according only to the two-tail confidence levels. The same about LSD (least significant difference) - it is calculated only for the two-tailed test.

Question: can I change it somewhere so that one-tail test will be used as default? If not, can I get the circles and lsd values for one-tail test somehow?

Thanks,
Mike.

[#2]

Yes, you have to make a change in your mental process, not in JMP.

If you want to do the one-sided test to see if the mean of group 1 is less than the mean of group 2, you go to comparison circles, and if they indicate a difference and the mean of group 1 is less than group 2 ... you have found a difference. If you go to the comparison circles and they indicate a difference but the mean of group 1 is greater than the mean of group 2, you have found no significant difference. You might have to adjust the alpha level of the test to get exactly what you want.

[#3]

Thanks for your response.

I am certainly ready to change whatever is needed to get the right result. But first, let me clarify my point.

The radii of the circles are T(alpha)*Sigma, where
T(alpha) is an alpha-quantile of the Student t-distribution,
Sigma is the group standard deviation.

Alpha is the type I error that you suggest to change if I want. I understand this. Now, suppose I have default Alpha = 0.5 and I performed the t-test on my 2 data groups and see that "Prob > t" is equal to 0.49.

If I know in advance which group is higher, and I am not interested in the situation when this group is significantly lower, I operate in one-tail test realm, and the "Prob >t" is exactly what I need. So, if this value is less than 0.5, then my groups are significantly different.

But if I click on the circles, they show no significant difference. This is because both the LSD (least significant difference) and the circles radii are calculated from the two-tailed error level equal 2*Alpha.

In my example 2*Alpha = 0.98 > 0.5

The 2-tail approach makes sense in certain situations, but my question is - can I change it to one-tail? It so happened, I do know in advance that one dataset can be only higher, and I want to test ONLY this for my error level.

[#4]

Since you have specified alpha=0.5 in several places, I assume this isn't a typo, but it sure is an unusual way to do things.

The default in JMP is to do a two-sided test, so when alpha = 0.5, there is 0.25 in each tail. So, when you set alpha = 0.5 in JMP, the one-sided test as I described it, will be at alpha=0.25. That doesn't sound like what you want.

You want to do a one-sided test with alpha = 0.5, and that is impossible to do in JMP. You would need to set alpha = 1, and I'm pretty sure JMP will give you an error if you try it.

Pencil and paper, you could do the math and come up with a one-sided test at alpha=0.5. JMP doesn't allow you to do this for alpha=0.5, but does allow you to do this for any alpha less than 0.5, using the method I described.

Performing hypothesis tests at alpha = 0.5 is not only unusual, but seems to me to be a crazy idea. Flipping coins will give you the same results. I recommend you not perform hypothesis tests at alpha = 0.5.

[#5]

Paige -

I am sorry. Of course it's a typo.

Alpha = 0.05, it's a default value.
The test shows " Prob > t = 0.049", meaning 1-tail difference is significant.
Circles show the difference is unsignificant, because it uses 2-tali test numbers, and 2*Alpha = 0.098 > 0.05
LSD table title reads "ABS(Dif) - LSD" suggesting also 2-tail test (absolute value of difference).

You said JMP can't do one-tailed test circles - I got it, thanks for the clarification.

Now, this your statement:
>>The default in JMP is to do a two-sided test, so when alpha = 0.5, there is 0.25 in each tail.

is wrong. It looks like JMP is really doing two-sided test, but if I set Alpha = 0.05, there is 0.05 in EACH tail, so the real error level for the mean difference is 0.1 !

This is the second part of my question - is this is a bug or is it by design?

[#6]

> Now, this your statement:
> >>The default in JMP is to do a two-sided test, so
> when alpha = 0.5, there is 0.25 in each tail.
>
> is wrong.
> It looks like JMP is really doing two-sided
> test, but if I set Alpha = 0.05, there is 0.05 in
> EACH tail, so the real error level for the mean
> difference is 0.1 !

Really? How do you know this?

I ask because when I look in JMP Help, in a chapter entitled "Details of Comparison Circles", it sure looks to me like they are claiming that the comparison circles are based upon alpha/2 in each tail.

> This is the second part of my question - is this is a
> bug or is it by design?

Since I don't work for JMP, I cannot answer this.

Message was edited by: Paige

[#7]

Well, looks like the Help could be misleading. It is rather easy to check:

1. Take any 2 datasets with reasonably different means - or take any dataset and split it in two parts.

2. Run "one-way ANOVA" t-test, and note the value of "Prob > t". Lets suppose it is equal to X.

3. Go to "Set Alpha level" -> "Other" and set Alpha equal to the value X+0.01

4. Go to "Compare means" -> "Each pair, Student's t" and click on the circles to see that they are claimed not significantly different.

5. Change Alpha level to 2*X-0.01 and repeat (4) with the same result

6. Change Alpha level to 2*X+0.01 and repeat (4) to see that the difference becomes significant.

7. You may also see this on the t-test report in the value "Prob > Abs(t)" - it is equal to 2*X, and this seems to be used in the significance test.

[#8]

Sorry, I don't buy this.

I created a very small data set, so I can do the calculations by hand. The prob > |t| is very clearly the two-tailed probability.

If you look at the comparison circles in my specific data set, the means are not different at alpha = 0.05. Matches my hand calculations. For this data set, if you set the alpha to exactly equal the prob > |t|, the circles still show no difference. If you increase the alpha to be slightly greater than the prob > |t|, the circles now show a statistically significant difference. This is exactly what you would expect.

So, the comparison circles use two-tailed tests.

[#9]

But this is exactly what I wrote. What do you not agree with? We already established that Jmp uses two-tailed test for the mean comparison. It's a pity, but I can live with this. No problem.

Problem is different: if you set the error level = Alpha in Jmp GUI, then Jmp uses two-tail test with error level 2*Alpha!

[#10]

Disagree.

I do not agree that "Problem is different: if you set the error level = Alpha in Jmp GUI, then Jmp uses two-tail test with error level 2*Alpha!"

I say ... JMP uses two-tail test with error level alpha.