Cochran-Mantel-Haenszel Tests Interpretation: permuting X var and Blocking Var = different results

[#1]

I have the following mock data set that I created specifically to test association between two categorical variables (X = BIOMARKER, Y = RESPONSE, Freq = FREQ), grouped by a third categorical variable = ARM

ARM	BIOMARKER	RESPONSE	FREQ
PBO	BM-	NR	25
PBO	BM-	R	25
PBO	BM+	NR	21
PBO	BM+	R	29
ACTIVE	BM-	NR	38
ACTIVE	BM-	R	12
ACTIVE	BM+	NR	20
ACTIVE	BM+	R	30

When I perform the X by Y contigency analysis followed by CMH test, I get the following result:

Cochran-Mantel-Haenszel Tests
Stratified by
ARM
CMH Test	ChiSquare	DF	Prob>Chisq
Correlation of Scores	9.739	1	0.0018
Row Score by Col Categories	9.739	1	0.0018
Col Score by Row Categories	9.739	1	0.0018
General Assoc. of Categories	9.739	1	0.0018

But when I try the reciprocal analysis where X = ARM, Y = RESPONSE, and Freq = FREQ, grouped by BIOMARKER, I get a different result:

Cochran-Mantel-Haenszel Tests
Stratified by
BIOMARKER
CMH Test	ChiSquare	DF	Prob>Chisq
Correlation of Scores	3.0013	1	0.0832
Row Score by Col Categories	3.0013	1	0.0832
Col Score by Row Categories	3.0013	1	0.0832
General Assoc. of Categories	3.0013	1	0.0832

Clearly, I'm not understanding the mechanic of the CMH analysis because I would have expected to get the same results either way.

Also, in this example, there is only contrast in BIOMARKER in the ACTIVE group (by design) but I unsure how to calculate the p value for each ARM (beside using the "By" option).

Thanks  

[#2 · Accepted solution]

Why would you expect the same results from two different analyses?

Pay attention to the analysis roles. The Y role is for the response or outcome. The X role is for the explanatory or predictive variable. Use these roles to define the association of interest. The CMH analysis uses a third role: stratum. Use this role when you want to see if the significance of the association between Y and X is greater or lesser across the strata (levels in the third variable).

Your first analysis explored/tested the association between Response and Biomarker stratified by ARM. Your second analysis explored/tested the association between Response and ARM stratified by Biomarker. These two analyses are not the same.

See: Agresti, Alan (1996) An Introduction to Categorical Data Analysis, John Wiley & Sons, New York, page 61.

[#3]

Dear Mark,

Thank you for your clarification. Sorry for my limited understanding of the CMH test but, as a follow up, I would like to know if there are any circumstances where the association between reponse and predictive variable would be non-significant but the CMH test would be significant across the strata?

So far, I have experimented with some mock data and I can't find a combination of data producing this type of output.

Thank you for your help.

Sincerely,

Thierry

[#4]

I was unable to create an example of such a case and I don't have access to Agresti's textbook to check the computation and determine under what circumstances, if any, it might be possible. So you will have to wait for a while for an answer from me or perhaps another expert might answer in the meantime.

[#5]

Your hunch is correct. A colleague of mine, Bob Lucas, replied with the following answer:

"I went to this link.

From looking at the test statistic, I do not think it is possible.

The numerator of the CMH test is just the sum over all the strata of the observed  minus predicted squared. The denominator is just the scaling factor so that the distribution is asymptotically Chi Square.

The numerator of the CMH test is similar to the numerator of the F in ANOVA.  Consequently,  I think it is a similar situation to overall F test in ANOVA vs. comparing two means in an ANOVA. 

I think it is possible that in one stratum, there may be a significant association but none in the others so by averaging over all the strata, the one association is hidden.  Similar to how two means in an ANOVA can be different but the overall F is not significant.

In ANOVA, if none of the pairwise mean comparison are significant, the overall F cannot be."