Correctly interpreting multiple significant interactions and main effects- JMP

[#1]

My analysis produced the below p-value table. I have made some conclusions based upon the results, but am unsure whether I am on the right track given the multiple significant interactions and main effects, specifically if I have reached too far.

Preliminary results of this study indicate that soluble solids content and specific gravity differed across years, cultivars, and locations, but for SSC the difference in location depended on year. Titratable acidity differed across the cultivars and locations, and the difference in locations also was dependent on year. Juice pH and total tannins differed across years and cultivars. Significant variation in annual factors could explain the interactions observed for SSC and TA (Table 3; Figure 1).

TO MAKE IT CLEAR- I AM NOT LOOKING FOR ADVICE ON THE P-VALUES OR INTERPRETATION, RATHER MY ABILITY IN MAKING CONCLUSIONS GIVEN THE SIGNIFICANT OR LACK THEREOF INTERACTIONS AND MAIN EFFECTS.

1)  For SSC, I reran the model eliminating the Cultivar*Location interaction as it was insignificant ( p = .149, a = 0.05), and the Year*Cultivar interaction became insignificant. So, was it correct to make the conclusion that the main effects (Year, Cultivar, and Location) were significant, but that the effect of Location was dependent on Year? I believe I am.

2) For TA, I followed the same steps as SSC and came up with the same conclusion in that the effect of Location was dependent on Year. However, in this case the main effect of Year was actually never significant as it was for SSC (p = 0.567 vs. p = 0.0004). Is my conclusion still fine?

Thank you ahead of time for any input!!!

[#2]

Hi, Travis!

I hope no self-respecting statistician would look at your table of p values and tell you (one way or another) about your conclusions!  There is a "crisis of confidence" surrounding p values in all of Science right now, and many preeminent iconic teachers have stopped teaching p values altogether.

FWIW, I think all metrics (including p values) exist to enhance decisionmaking, not replace it.

Some of the great things about JMP are the relative ease with which one can review many of the model diagnostics, and the size of the effects, and the power of the study, etc.  One shouldn't opine about the p values without reviewing the details first, and a User Community discussion site isn't the most efficient place for anyone to be presenting or reviewing the details.

Probably not what you wanted to hear, but I recommend you retain the help of a statistician to review your design and analyses.

[#3]

First off, thank you for taking the time to reply.

Second, as you have made it impossible for anyone else to reply without feeling stupid, given your comment of "no self-respecting statistician would," I need to clarify my post. I would actually ask that you or this site's moderator re-word your post as it is not constructive but destructive.

I AM NOT LOOKING FOR INTERPRETATION OF MY SPECIFIC RESULTS, I AM NOT LOOKING FOR ADVICE ON P-VALUES. I am looking for advice on whether I followed the accepted protocol in interpreting significant interactions and main effects.

1) If a two-way interaction and both main effects are significant, can you make any conclusions on the main effects except for dependency?

2) If a two-way interaction and only one of the main effects is significant, what can you say about the non-significant main effect?

  I am asking whether my analysis was right in terms of ability, not asking in terms of content.

Again since no one else will probably reply to my post now, can you provide your advice.

Thank you

[#4]

Thank you for your clarification.  I did not intend my comment to be destructive; you may or may not be aware of the recent controversies swirling around the misinterpretation of p values and the irreplicability of many published scientific studies, but those controversies should give caution to us all.  I did not intend to hurt your feelings, but I still stand behind my statements.   Warning claxons should sound whenever we see a table of p values.   Sir Ronald Aylmer Fisher himself cautioned against using p values as a final arbiter of significance, and he really knew of whence he spoke.

If an interaction is significant, I consider it pointless (and even possibly misleading) to interpret the significance of the main effects involved in the interaction.

A significant interaction A*B (with acceptable designs, analysis approach, effect sizes, power, and diagnostics) implies that the effect of A is conditional on the level of B, and vice versa.  It generally makes little sense to interpret any effect that is conditional on the level of another as if it were not, at some fixed value.

I also generally consider it good practice to leave even insignificant main effects in the model when their interaction is significant.

And I still recommend you retain the help of a statistician to review your design and analyses prior to arriving at any other conclusion.

[#5]

Travis,

Agreed, I won't comment on p-values.  As Kevin noted that is a hot topic right now.  However, maybe this will help you with the overall understanding of interactions and main effect significances.

Main effect for NACL significant:

Significant interaction shows that NACL effect size depends on the level of a second factor. NACL is still important and even more important when the blue level of the second factor is in play.

You could also have a "crossing" interaction where the main effect shows no significance on its own but great significance (in opposite directions) depending on the level of the interacting factors.  So in that case the "main effect" factor while "not statistically significant" in the model is still very important. And hence I love JMP and hate tables of numbers, you have to put the data in context and explore what your table of numbers is pointing you to in terms of knowledge. In addition to p-values as Kevin noted you should consider effect sizes (how big of a difference are you seeing due to the various factors).  Hopefully this helps, use your p-values to help you sort out the important relationships from the unimportant.