Welch's T-test with multiple comparisons correction

[#1]

I know how to do Steel-Dwass with control for comparing multiple population means to a single control population in JMP, but how do I use JMP to do a Welch's t-test with Benjamini-Hochberg correction (for example)?  Also, I know that the core difference between these approaches is that the former does not make the assumption of normality but the latter does.  I've not found good discussions of why you might choose one over the other assuming the normality assumption is satisfied (which really it never is, but sometimes is reasonable)?  

[#2]

Steel-Dwass has very little power with small group sizes.  If you have group sizes around n=15 or greater, it will probably be a decent option.  Years ago, I was looking for a Tukey HSD analysis with an unequal variances correction and came across Games-Howell, which is not available in JMP.  I added it to the JMP Wish List, and is one of the most up-voted proposals on there right now.  In the last year, I created an add-in to do Games-Howell on a single X and Y which you can find here:

https://community.jmp.com/t5/JMP-Add-Ins/Games-Howell-Test-Tukey-HSD-with-Welch-s-correction-for-Une...

I know you are wanting to do the multiple comparisons relative to a control, which I don't have an exact solution for, but if you are willing to do a Tukey-style analysis and just look at which levels are not connected to your control, you may get what you need from that add-in.

[#3]

Chris, 

Thanks for getting back to me.  The info about Steel-Dwass and small group sizes that is super helpful!  I agree that when comparing all pairs Games-Howell is the way to go, but I have simulations that show it can be overly conservative when you only want to compare the multiple populations to a single control (so not all pairwise comparisons). 

It would still be helpful to know if it is possible to do Welch's T-test with multiple comparison correction in JMP.  

[#4 · Accepted solution]

It's not currently possible.  Best you can probably do with what is currently available in JMP is a Bonferroni correction, and I doubt you would want to go that direction.  If you are an R user, the DTK package may give you what you need.

[#5]

Thanks Chris!

[#6]

Hi,

for a stats assignment I have to do a Bonferroni correction on my data. I'm using a two sided t-test to compare several t distributed datasets to each other and will write a jsl script to do so. How do I incorporate the Bonferroni correction in JMP?

Thanks

Ole

[#7]

Bonferroni is just an adjustment on alpha, which you decide. You just take your desired alpha for the family of tests and divide that by the number of tests. For example, if I want a family-wise error rate of 0.05 and do 10 pairs of comparisons, I would compare the p-value of each individual comparison to 0.05/10 = 0.005.