Games-Howell Test (Tukey HSD with Welch's correction for Unequal Variances)

[#1]

Enhancements to Version 1.1:

• Revamped connecting letters report generation using the insert-absorb algorithm and sweep post-processing algorithm described in Piepho, H.-P., 2004. An algorithm for a letter-based representation of allpairwise comparisons. Journal of Computational and Graphical Statistics 13, 456–466.
• Improved handling of excluded groups and missing values.

Version 1.2 and 1.3 both address bugs encountered by users.

This add-in allows you to performs the Games-Howell test for one nominal/ordinal group variable and one continuous response.  Games-Howell is an adjustment to the Tukey HSD procedure to adjust the error degrees of freedom to account for unequal variances.

Output is similar to Fit Y by X output for the Tukey HSD procedure, which includes the Tukey HSD threshold matrix, Connecting Letters Report, and the Ordered Differences Report.  Sample output using Weight vs. Position from the Football sample data set is shown below:

Threshold Matrix

Connecting Letters

Ordered Differences

If you get a lot of value from the add-in, please let me know in the comments.  If there is sufficient interest, I will work on adding support for multiple X and Y variables, By variables, Weight, etc.

[cistoladp]

This add-in is most helpful and valuble - thank you!

[Lyanne]

I am running a oneway analysis with a block and need to use Welch's Test as the variances are not equal. This Games-Howell  add-in does not seem to interact directly with outputs, is that correct?

I am wondering if I need to center the data around the block and then run this add-in when I am running a oneway with a block.

Thanks in advance.

[cwillden]

Hi @Lyanne,

To answer your first question, no it does not use any information from the Fit Y by X report and it is not currently able to analyze blocked data.  You could normalize by the block effects, but I think you will be anti-conservative using this add-in since it will not adjust the degrees of freedom for your block effect.

[aharding]

I like the approach, but wonder how powerful it is (statistical power to detect differences) compared to the weighted least squares method uisng 1/variance of each group as the weight. Have you compared these approaches?

[ezorlo]

Hi thanks for creating this addin

i installed it but can't find the option to perform this test. How do i actually run it? thanks

[cwillden]

@ezorlo , you should see it available from the Add-Ins menu as "Games Howell Test." 

[elpulster]

Thanks for creating this add-in.  I receive the following error message:  "invalid subscript (must be number or list of numbers) in access or evaluation of 'thresh_ind[0,i]' , thresh_ind[/*###*/0, i]" when trying to look for annual differences in my data set.  Does this add-in not work with year (ordinal) as a factor?

[cwillden]

@elpulster Can you send a sample data set that generates the error?  If your data is sensitive, you could just anonymize the table.  I can see investigate the issue.

[elpulster]

@cwillden thanks for the quick response.  I just realized that I only receive that error when I try to hide or exclude certain rows.  Any way around this?

[cwillden]

Hmmm... I haven't looked at that code for quite some time so I'll need to dig in.  You can probably just create a copy of the table without the excluded rows and re-run on that.

[ataylor]

Hi.  The connecting letters report has a bug or error in it.  I have data that I generated to illustrate the problem, but csv and jmp are not supported file types for attaching to a comment like this, so I'm not sure how to share the data I used to generate this report. I can attach a screen shot of the report generated.  The "ordered differences report" is correct for these data.  The letters report shows a couple of issues. 

1. The first column is redundant given the second column and similarly the forth column is redundant given the third column.  
2. According to this report, only `grp_2` and `grp_5` are significantly different, but we can easily see in the differences report that there are 9 pairings that are different.  I think the report should be (modulo some useful ordering of the levels): 

   Level
   grp_0	A			D
   grp_1	A	B
   grp_2		B
   grp_3			C
   grp_4			C	D
   grp_5			C

[cwillden]

@ataylor , thank you for bringing that to my attention and I think I know how that is happening.  I'll determine a fix and upload soon.  Though I think the correct table would actually be

Level
grp_3	A
grp_5	A
grp_4	A	B
grp_0		B	C
grp_1		B	C
grp_2			C

[cwillden]

New version just uploaded!  @elpulster, I believe any issues with excluded and missing values has been resolved.  @ataylor, the new algorithm is now returning what should be the correct connecting letter report.  Thank you for your comments and suggestions.

[ataylor]

Thanks!  Yes, we agree that is the correct table above and glad to hear you resolved it!

[STAT]

Hi, I am trying to undertake the Games-Howell test with the Add-in, however always receive the error notificiation: "Argumentwert ist ungültig bei Zugriff oder Auswertung von group1 , group1/*###*/" (sorry, it is in German - it says something like argumentation value is invalid or the evaluation of Group 1, Group 1/*###*/). What does that mean? What is the problem? The add-in is generally functioning because when I try it out with another column it functions (the only difference is that there is somewhat more data in the column that works, but basically it is the same kind of type of data, i.e. just numbers). I would be very grateful if someone could help me.

[STAT]

Hi,

Do you have any idea what I do wrong and how I can solve it when I get the following problem code while using the Games Howell Add-in?

"keine Zeichenkette bei Zugriff oder Auswertung von String Col Box, Bad Argument(3795), String Col Box "###"/("Level", grps_sort)"

Translated in English it says something like "no string during evaluation of string col box"

I would be very grateful for any help!

Thanks and best wishes

[cwillden]

Without getting into the code (I’m away from my laptop right now), I would guess you have a numeric variable for the groups.  I thought I had addressed that in my last update, but maybe not.  Change the group to a character column and try again.  Make sure you have the most recent version

[abarabas]

Thank you for this!

The connecting letters report shows the group means. Are there plans to display group SE as well? Is it possible to see residuals?

[sonyaponzi]

Hi @cwillden ,

Do you know what this error means? I am able to run the add-in on the sample football data, but when I run it on my own data I get this message. 

[cwillden]

@sonyaponzi , I can run your data through the script to debug this.  Can you send it to me?  I'll send you a DM with my email.

[cwillden]

New add-in just uploaded (V1.2) to correct the bug encountered by @sonyaponzi.  Thanks for the tip!

[Raaed]

what the different between your add-in and (Fit Y by X output for the Tukey HSD procedure)

[ataylor]

Tukey HSD assumes equal sample sizes and equal variances for all groups, while, as the title notes, Games-Howell (what this plug-in does) is also for comparing groups, but does not require groups to have equal sample sizes nor equal variances.  If you use this plug-in when the groups have equal variance and equal sample sizes, the test will generally be more conservative than Tukey HSD, and you should not use Tukey when the assumptions are violated.  

[bucky]

@cwillden, thanks for creating this add in. I'm getting a similar error to @sonyaponzi, but using v1.2. It runs seemingly correctly for one column of data but not for another, and I can't figure out why. Thoughts?

[cwillden]

@bucky - sorry the community notifications were going to my junk folder.  If you still would like help with this, I can take a look at your data set and see what the hangup is.

[CentralMustang5]

@cwillden Could you please also post an add-in for Dunnett's T3 multiple comparisons test (Tukey HSD with Welch's correction for Unequal Variances)?