topic Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test. in Discussions

Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

PeterD — Sat, 10 Jun 2023 23:31:44 GMT

We did a marketing study for our product that incidentally appears to show strong clinical treatment effect (participants had less pain at the end than the beginning) and I was volunteered to do the statistical analysis.

I did a paired T test on the baseline versus week 8 pain scores ( Specialized Modelling--> Matched Pairs -->) with very low p values.

However I got ambitious (because JMP makes it so easy) and compared baseline with 4 weeks then baseline with 8 weeks and 4 weeks with 8 weeks. All showed significance but the difference between 4 and 8 weeks was never clinically relevant.

So, when it came to publication the reviewer said I should have used "one way repeated measures ANOVA" because i had three timepoints. which seemed fair enough (sort of) ?

big problem---I cannot figure out how to run the ANOVA test. The histograms and box plots show visually that there is a difference (means 2.8 vs 2.76 S dev 1.6 , 1.1). The data are in three columns of pain scores (continuous (0-10)variables).

How can I get JMP to compare my columns? are they significantly different?

Apologies if this is a really dumb question but my reviewers need to see something...

Thanks for any help anyone can offer.

Peter

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

PeterD — Sat, 12 Jun 2021 22:12:30 GMT

The means in the post were 5.8 vs 2.76 (typo)

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

Kevin_Anderson — Sat, 12 Jun 2021 23:28:32 GMT

Hi, PeterD!

First, you should explore the Repeated Measures topic in the Statistics Index under Help. Following the Topic Help will suggest the data format you need and help understand the analysis. Help can be actually helpful.

Second, Repeated Measures are kind of statistical "heavy lifting" in the best of circumstances. Changing your analysis plan after the experiment has been designed and run is not the best of circumstances. From what I can tell, I fear your reviewer has correctly identified a problem with your analysis, but it's not necessarily easy to recover and whatever results you manage to salvage may not be technically useful. Ideally, you should've consulted a statistician before designing and running your experiment and they might've helped identify the problems with your plan earlier, but I would certainly advise you to consult one now and perhaps you can recover.

Good luck.

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

dale_lehman — Sun, 13 Jun 2021 12:39:24 GMT

I've not done a repeated measures ANOVA but that does sound like the correct thing to do. If I understand it correctly, it is analogous to the standard ANOVA in the same way that a standard t test relates to a matched pairs t test. But, aside from those statistical details, your description of your results sounds fairly clear to me. The treatment appears to have an impact from the baseline to either the 4 or 8 week point, but the difference from 4 to 8 weeks is not significant. Doesn't that mean there is evidence that the treatment has an impact, but that the impact is felt after 4 weeks and continuing to 8 weeks does not appear to add much? That story sounds consistent with what you have stated and would seem to make sense to me. The repeated measures ANOVA might yield somewhat different statistical results, but I think the emphasis on "significant" or "not significant" comparisons is not the most important thing here. The evidence you have is suggestive of a treatment that works, with the effect being felt within a 4 week period, and that longer treatment may not improve much from that point on.

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

Mark_Bailey — Mon, 14 Jun 2021 15:17:22 GMT

You might also consider a simpler approach: matched pairs. Each participant represents a block in the design. The before and after responses are correlated. A t-test or ANOVA might be appropriate, but they do not address the variation between participants.

Please see this page about the Matched Pairs platform to decide if it is appropriate.

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

Mark_Bailey — Mon, 14 Jun 2021 15:24:14 GMT

Additionally, the technical note might be useful.

Re: Paired T test vs ANOVA--and I cannot figure out how to run the ANOVA test.

PeterD — Wed, 07 Jul 2021 22:27:37 GMT

my apologies for the delay--all JMP emails went to an obscure junk mail folder and i only just recovered them--but i totally agree. it was NOT a satisfactory experimental design. As it happens, when we did do the repeated measures ANOVA, the results did not change--even the p values were close.