topic Re: One-tail vs. Two-tail tests in Discussions

One-tail vs. Two-tail tests

Fri, 27 Mar 2009 23:17:23 GMT

Hi,

I am doing the means comparison using one-way ANOVA in Jmp. When I look at the detailed report, I see the p-values for both one-tail and two-tail t-tests.
However, the comparison circles seem to display according only to the two-tail confidence levels. The same about LSD (least significant difference) - it is calculated only for the two-tailed test.

Question: can I change it somewhere so that one-tail test will be used as default? If not, can I get the circles and lsd values for one-tail test somehow?

Thanks,
Mike.

Re: One-tail vs. Two-tail tests

Tue, 31 Mar 2009 14:15:33 GMT

Yes, you have to make a change in your mental process, not in JMP.

If you want to do the one-sided test to see if the mean of group 1 is less than the mean of group 2, you go to comparison circles, and if they indicate a difference and the mean of group 1 is less than group 2 ... you have found a difference. If you go to the comparison circles and they indicate a difference but the mean of group 1 is greater than the mean of group 2, you have found no significant difference. You might have to adjust the alpha level of the test to get exactly what you want.

Re: One-tail vs. Two-tail tests

Tue, 31 Mar 2009 18:10:56 GMT

Thanks for your response.

I am certainly ready to change whatever is needed to get the right result. But first, let me clarify my point.

The radii of the circles are T(alpha)*Sigma, where
T(alpha) is an alpha-quantile of the Student t-distribution,
Sigma is the group standard deviation.

Alpha is the type I error that you suggest to change if I want. I understand this. Now, suppose I have default Alpha = 0.5 and I performed the t-test on my 2 data groups and see that "Prob > t" is equal to 0.49.

If I know in advance which group is higher, and I am not interested in the situation when this group is significantly lower, I operate in one-tail test realm, and the "Prob >t" is exactly what I need. So, if this value is less than 0.5, then my groups are significantly different.

But if I click on the circles, they show no significant difference. This is because both the LSD (least significant difference) and the circles radii are calculated from the two-tailed error level equal 2*Alpha.

In my example 2*Alpha = 0.98 > 0.5

The 2-tail approach makes sense in certain situations, but my question is - can I change it to one-tail? It so happened, I do know in advance that one dataset can be only higher, and I want to test ONLY this for my error level.

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 12:19:36 GMT

Since you have specified alpha=0.5 in several places, I assume this isn't a typo, but it sure is an unusual way to do things.

The default in JMP is to do a two-sided test, so when alpha = 0.5, there is 0.25 in each tail. So, when you set alpha = 0.5 in JMP, the one-sided test as I described it, will be at alpha=0.25. That doesn't sound like what you want.

You want to do a one-sided test with alpha = 0.5, and that is impossible to do in JMP. You would need to set alpha = 1, and I'm pretty sure JMP will give you an error if you try it.

Pencil and paper, you could do the math and come up with a one-sided test at alpha=0.5. JMP doesn't allow you to do this for alpha=0.5, but does allow you to do this for any alpha less than 0.5, using the method I described.

Performing hypothesis tests at alpha = 0.5 is not only unusual, but seems to me to be a crazy idea. Flipping coins will give you the same results. I recommend you not perform hypothesis tests at alpha = 0.5.

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 17:18:14 GMT

Paige -

I am sorry. Of course it's a typo.

Alpha = 0.05, it's a default value.
The test shows " Prob > t = 0.049", meaning 1-tail difference is significant.
Circles show the difference is unsignificant, because it uses 2-tali test numbers, and 2*Alpha = 0.098 > 0.05
LSD table title reads "ABS(Dif) - LSD" suggesting also 2-tail test (absolute value of difference).

You said JMP can't do one-tailed test circles - I got it, thanks for the clarification.

Now, this your statement:
>>The default in JMP is to do a two-sided test, so when alpha = 0.5, there is 0.25 in each tail.

is wrong. It looks like JMP is really doing two-sided test, but if I set Alpha = 0.05, there is 0.05 in EACH tail, so the real error level for the mean difference is 0.1 !

This is the second part of my question - is this is a bug or is it by design?

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 18:10:40 GMT

> Now, this your statement:
> >>The default in JMP is to do a two-sided test, so
> when alpha = 0.5, there is 0.25 in each tail.
>
> is wrong.
> It looks like JMP is really doing two-sided
> test, but if I set Alpha = 0.05, there is 0.05 in
> EACH tail, so the real error level for the mean
> difference is 0.1 !

Really? How do you know this?

I ask because when I look in JMP Help, in a chapter entitled "Details of Comparison Circles", it sure looks to me like they are claiming that the comparison circles are based upon alpha/2 in each tail.

> This is the second part of my question - is this is a
> bug or is it by design?

Since I don't work for JMP, I cannot answer this.

Message was edited by: Paige

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 18:28:58 GMT

Well, looks like the Help could be misleading. It is rather easy to check:

1. Take any 2 datasets with reasonably different means - or take any dataset and split it in two parts.

2. Run "one-way ANOVA" t-test, and note the value of "Prob > t". Lets suppose it is equal to X.

3. Go to "Set Alpha level" -> "Other" and set Alpha equal to the value X+0.01

4. Go to "Compare means" -> "Each pair, Student's t" and click on the circles to see that they are claimed not significantly different.

5. Change Alpha level to 2*X-0.01 and repeat (4) with the same result

6. Change Alpha level to 2*X+0.01 and repeat (4) to see that the difference becomes significant.

7. You may also see this on the t-test report in the value "Prob > Abs(t)" - it is equal to 2*X, and this seems to be used in the significance test.

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 18:44:57 GMT

Sorry, I don't buy this.

I created a very small data set, so I can do the calculations by hand. The prob > |t| is very clearly the two-tailed probability.

If you look at the comparison circles in my specific data set, the means are not different at alpha = 0.05. Matches my hand calculations. For this data set, if you set the alpha to exactly equal the prob > |t|, the circles still show no difference. If you increase the alpha to be slightly greater than the prob > |t|, the circles now show a statistically significant difference. This is exactly what you would expect.

So, the comparison circles use two-tailed tests.

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 18:54:53 GMT

But this is exactly what I wrote. What do you not agree with? We already established that Jmp uses two-tailed test for the mean comparison. It's a pity, but I can live with this. No problem.

Problem is different: if you set the error level = Alpha in Jmp GUI, then Jmp uses two-tail test with error level 2*Alpha!

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 20:01:35 GMT

Disagree.

I do not agree that "Problem is different: if you set the error level = Alpha in Jmp GUI, then Jmp uses two-tail test with error level 2*Alpha!"

I say ... JMP uses two-tail test with error level alpha.

Re: One-tail vs. Two-tail tests

Wed, 01 Apr 2009 20:22:04 GMT

OK. You're right.
Thanks.