Equivalence test

[#1]

Is there an equivalence test procedure that can be used with categorical data (i.e., both independent variables and dependent variable are categorical)?

A nominal logistic fit for fertility as influenced by hormonal procedure, lactation, and age category shows no stat. sig. influence. I was under the impression that it would be inappropriate to conclude there is no difference between the means unless we employ an equivalence test comparison that statistically validates such claim(?).  

Many thanks!  

[#2]

Well, the null hypothesis is that there is no difference, and that's what we assume until we can demonstrate convincing evidence to the contrary.  That's not the same as saying you've provided strong evidence for the null hypothesis, so you're right in that regard.  So, if it's sufficient to say that there is not sufficient evidence to show that hormonal procedure, lactation, and age category have an effect on fertility, then you're done. 

If you need to demonstrate the effects are so small that they are practically negligible, then something analaogous to an equivalence test for categorical data would be necessary.

You can effectively do this by looking at the confidence intervals for each of your model terms.  If they are all contained within some range you would deem practically negligible, then you can make the kind of conclusion you're looking for.  However, I think coming up with a range you would deem practically neglible might be kinda hard when you're talking about effect sizes in the logit space.

[#3 · Accepted solution]

Generally speaking it sounds like you may be confusing the two analytic concepts of equivalence testing and hypothesis testing. These are two very different types of decisions. Hypothesis testing never proves anything with certainty. All is allows one to do is either reject of fail to reject the null hypothesis under a certain prescribed set of conditions such as alpha risk, for a given sample size, etc. No where in either of those two conclusions is the notion of equivalence covered. Equivalence testing is a completely different analytics approach.

There are a few different types of equivalence testing out there that are used in different contexts. My colleague @Mark_Bailey for example has written a nice JMP script in this space:

https://community.jmp.com/t5/JMP-Scripts/Test-for-Parameter-Equivalence/ta-p/29375/jump-to/first-unr...

In addition, often in a biotech setting, fitting nonlinear models to data is common and there are different types of 'equivalence testing' used in this context. These are supported in JMP's Fit Curve platform:

http://www.jmp.com/support/help/13-2/Equivalence_Test_2.shtml

[#4 · Accepted solution]

As it turns out, JMP provides equivalence testing (under multiple comparisons) in the Generalized Regression platform, if you have JMP Pro. I just want to emphasize Peter's point that hypothesis testing for a difference or for equivalence are very different. Your choice depends entirely on what you are testing for (alternative hypothesis). If your were using nominal logistic regression to test for associations and found none, then you are finished. You don't need to show statistically significant equivalence.

If you are surprised by this result then perhaps it is a matter of the power in the study.