cancel
Showing results for
Show  only  | Search instead for
Did you mean:
Choose Language Hide Translation Bar
Level III

## How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

Hello,
I am analyzing an experiment where test variation had statistically significant results for some outcomes during 2nd and 3rd weeks with a difference of 20-30%, but had insignificant results for the other 2 weeks (1st and 4th) with a difference of ~0%.
How can I address the interpretation of these observations?
What is the appropriate statistical method to choose to check if the difference in observations among 2nd+3rd and 1st+4th are just within random variance?
Can we still make a valid conclusion for this experiment based on the total results?

We have checked any external factors which could influence the results and couldn't find any.

Table version:

 Trials Success step 1 Diff p-value Success step 1 which reports about step 3 Diff p-value Success step 3 Diff p-value Test Control Test Control Test Control Test Control Weeks March 8-14 88358 89019 2585 2590 -0.19% 0.5 1068 1038 +3.6% 0.2 187 191 -1.36 0.89 March 15-21 88407 88324 2656 2596 +2.2% 0.2 1102 1037 +6.17% 0.08 200 167 +20% 0.04 March 22-28 90718 92135 2835 2763 +4% 0.059 1277 1198 +8.26 0.02 210 165 +30% 0.006 March 29 - April 4 91496 91613 2506 2521 -0.4 0.57 1084 1139 -4.7% 0.25 137 137 0.13% 0.5 Total 358979 361091 10582 10470 1.6% 0.11 4531 4412 +3% 0.06 734 660 +11.8% 0.018 1st + 4th weeks 179854 180632 5091 5111 +0.04 0.5 2152 2177 -0.72% 0.8 324 328 -0.79% 0.92 2nd + 3rd weeks 179125 180459 5491 5359 +3.23% 0.04 2379 2235 +7.24% 0.017 410 332 +24% 0.003
Learning DOE
2 REPLIES 2
Super User

## Re: How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

I apologize, I don't have time right now to enter your data into JMP and take a look, but, in general, you seem to have either a Block effect or a Block-by-factor interaction effect.  Where you would want to investigate is: Why are the results different between weeks? Make sure the differences first pass the sensibility test.  That is, does the data vary enough for further analysis.  Is there enough variation in the Y's to be of practical importance?  Then, proceed with analysis of a Block design (this varies depending on whether you consider the Block effects to be random or fixed).

Blocks as used to partition the noise in an experiment.  The method is quite useful as you can reap the benefits of both increasing the inference space of your experiment without compromising precision (and, in some cases, assess the robustness of your model).

"All models are wrong, some are useful" G.E.P. Box
Level III

## Re: How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

Thank you! I will look into this from Block design standpoint.

I have added confidence intervals for absolute and relative effects of the last Y (success step 3).
Can we conclude from this data that because CIs of relative effects are overlapping, then the 2 other weeks of the same experiment are just a result of variation? If so perhaps we can just be ok with that 2 weeks showing different results, since this is a behavioral online experiment.

Learning DOE