testing p-value agains alpha level in t-test

[#1]

Hi,

I'm doing some homework for a stats class and running a t-test at different alpha levels. 

For alpha = 0.05 the Prob > |t| 0.0416 turns red which I expected as it is less than the alpha value.

When I changed alpha to 0.0056 it stayed the same value and color remained red. Shouldn't it turn black as the value of 0.0416 > 0.0056 (the set alpha level). Or am interpreting Prob > |t| incorrectly?

?

Thanks

Ole

[#2]

The p-value is calculated independently of your alpha level. It will not change no matter how many times you run the test. That is the beauty of the p-value. It leads to a straight-forward decision rule: If p-value is less than alpha, reject the null hypothesis.The alpha level is "in your head" meaning it is the threshold that YOU establish. JMP does not really care what it is, but JMP will give you the information you need to make your decision.

The changing of the color (and adding an asterisk) of the p-value is a different issue. That is just a formatting of the p-value that JMP provides. That formatting is hard-coded for alpha = 0.05 (the typical alpha value most people use). If you are using a different alpha, the formatting will not be correct. But the decision rule that I mentioned above will always be correct.

Note that there are ways in JMP to change the formatting that is used for the p-values, but that is beyond the scope of this post.

[#3]

Hi Dan,

Thanks for the quick reply.

One follow-up question: is there a lot of scripting involved to change the color according to the alpha level used? If its straight forward I would update my script accordingly so the teacher can't tell me that R is a more powerful solution (as he recommended it …).
Otherwise I use a trend chart that compares the different |t| values by variable against the 2 different alpha levels (with/and without Bonferroni correction) as y-reference lines.

Thanks

Ole

[#4]

You can do this interactively:

Double-click the p-value. You will see a dialog that says the format is PValue. You will also see Conditional Highlighting is PValue. Click on either one of those to get a dropdown of the various formats or conditional highlighting options. The last one is Custom or Add New. Choose those and define as you will.

You can also do this through scripting. Details are in the "Create Custom Functions, Transforms, and Formats" section of the Scripting Guide documentation.

[#5 · Accepted solution]

The p-Value being displayed is using the P-Value Conditional Formatting.  This Conditional Format is globally set in the JMP Preferences for Report, and by default is set to <.05 being set to Red. The conditions are user changeable, and if changed with the Oneway Platform being displayed, the display will be immediately changed upon closing the Preferences window.  In the picture below, it shows that I have changed the PValue conditional formatting to different settings.  Here, I going to color only values < .02 to red. And I added another condition that controls any value found that has a value >= .1 will be green.  You don't see those setting since only one condition can be displayed at one time.  However, in the Oneway analysis on the right hand side of the picture, you can see that only the p-value less that .2 is red, and a p-value greater than .1 is green.

I hope this clears up the issue.

BTW, I do think it would be nice if the formatting could follow the Confidence (alfa) setting.  You need to add that to the JMP Wish List 

[#6]

Jim's approach is better!

[#7]

Many thanks to both of you.