Subscribe Bookmark RSS Feed

Cochran-Mantel-Haenszel Tests Interpretation: permuting X var and Blocking Var = different results

sornasst

Community Trekker

Joined:

Feb 2, 2016

I have the following mock data set that I created specifically to test association between two categorical variables (X = BIOMARKER, Y = RESPONSE, Freq = FREQ), grouped by a third categorical variable = ARM

ARMBIOMARKERRESPONSEFREQ
PBOBM-NR25
PBOBM-R25
PBOBM+NR21
PBOBM+R29
ACTIVEBM-NR38
ACTIVEBM-R12
ACTIVEBM+NR20
ACTIVEBM+R30

 

When I perform the X by Y contigency analysis followed by CMH test, I get the following result:

Cochran-Mantel-Haenszel Tests
Stratified by   
ARM   
    
    
CMH TestChiSquareDFProb>Chisq
Correlation of Scores9.73910.0018
Row Score by Col Categories9.73910.0018
Col Score by Row Categories9.73910.0018
General Assoc. of Categories9.73910.0018

 

But when I try the reciprocal analysis where X = ARM, Y = RESPONSE, and Freq = FREQ, grouped by BIOMARKER, I get a different result:

    
    
Cochran-Mantel-Haenszel Tests
Stratified by   
BIOMARKER  
    
    
CMH TestChiSquareDFProb>Chisq
Correlation of Scores3.001310.0832
Row Score by Col Categories3.001310.0832
Col Score by Row Categories3.001310.0832
General Assoc. of Categories3.001310.0832
    

 

Clearly, I'm not understanding the mechanic of the CMH analysis because I would have expected to get the same results either way.

Also, in this example, there is only contrast in BIOMARKER in the ACTIVE group (by design) but I unsure how to calculate the p value for each ARM (beside using the "By" option).

 

Thanks  

1 ACCEPTED SOLUTION

Accepted Solutions
markbailey

Staff

Joined:

Jun 23, 2011

Solution

Why would you expect the same results from two different analyses?

Pay attention to the analysis roles. The Y role is for the response or outcome. The X role is for the explanatory or predictive variable. Use these roles to define the association of interest. The CMH analysis uses a third role: stratum. Use this role when you want to see if the significance of the association between Y and X is greater or lesser across the strata (levels in the third variable).

Your first analysis explored/tested the association between Response and Biomarker stratified by ARM. Your second analysis explored/tested the association between Response and ARM stratified by Biomarker. These two analyses are not the same.

See: Agresti, Alan (1996) An Introduction to Categorical Data Analysis, John Wiley & Sons, New York, page 61.

Learn it once, use it forever!
4 REPLIES
markbailey

Staff

Joined:

Jun 23, 2011

Solution

Why would you expect the same results from two different analyses?

Pay attention to the analysis roles. The Y role is for the response or outcome. The X role is for the explanatory or predictive variable. Use these roles to define the association of interest. The CMH analysis uses a third role: stratum. Use this role when you want to see if the significance of the association between Y and X is greater or lesser across the strata (levels in the third variable).

Your first analysis explored/tested the association between Response and Biomarker stratified by ARM. Your second analysis explored/tested the association between Response and ARM stratified by Biomarker. These two analyses are not the same.

See: Agresti, Alan (1996) An Introduction to Categorical Data Analysis, John Wiley & Sons, New York, page 61.

Learn it once, use it forever!
sornasst

Community Trekker

Joined:

Feb 2, 2016

Dear Mark,

 

Thank you for your clarification. Sorry for my limited understanding of the CMH test but, as a follow up, I would like to know if there are any circumstances where the association between reponse and predictive variable would be non-significant but the CMH test would be significant across the strata?

 

So far, I have experimented with some mock data and I can't find a combination of data producing this type of output.

 

Thank you for your help.

 

Sincerely,

 

Thierry

markbailey

Staff

Joined:

Jun 23, 2011

I was unable to create an example of such a case and I don't have access to Agresti's textbook to check the computation and determine under what circumstances, if any, it might be possible. So you will have to wait for a while for an answer from me or perhaps another expert might answer in the meantime.

Learn it once, use it forever!
markbailey

Staff

Joined:

Jun 23, 2011

Your hunch is correct. A colleague of mine, Bob Lucas, replied with the following answer:

"I went to this link.

From looking at the test statistic, I do not think it is possible.

The numerator of the CMH test is just the sum over all the strata of the observed  minus predicted squared. The denominator is just the scaling factor so that the distribution is asymptotically Chi Square.

The numerator of the CMH test is similar to the numerator of the F in ANOVA.  Consequently,  I think it is a similar situation to overall F test in ANOVA vs. comparing two means in an ANOVA. 

I think it is possible that in one stratum, there may be a significant association but none in the others so by averaging over all the strata, the one association is hidden.  Similar to how two means in an ANOVA can be different but the overall F is not significant.

In ANOVA, if none of the pairwise mean comparison are significant, the overall F cannot be."

Learn it once, use it forever!