topic How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks? in Discussions

How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

gustavjung — Sun, 11 Jun 2023 00:00:01 GMT

Hello,
I am analyzing an experiment where test variation had statistically significant results for some outcomes during 2nd and 3rd weeks with a difference of 20-30%, but had insignificant results for the other 2 weeks (1st and 4th) with a difference of ~0%.
How can I address the interpretation of these observations?
What is the appropriate statistical method to choose to check if the difference in observations among 2nd+3rd and 1st+4th are just within random variance?
Can we still make a valid conclusion for this experiment based on the total results?

We have checked any external factors which could influence the results and couldn't find any.

Table version:

	Trials	Success step 1	Diff	p-value	Success step 1 which reports about step 3	Diff	p-value	Success step 3	Diff	p-value
	Test	Control	Test	Control			Test	Control			Test	Control
Weeks
March 8-14	88358	89019	2585	2590	-0.19%	0.5	1068	1038	+3.6%	0.2	187	191	-1.36	0.89
March 15-21	88407	88324	2656	2596	+2.2%	0.2	1102	1037	+6.17%	0.08	200	167	+20%	0.04
March 22-28	90718	92135	2835	2763	+4%	0.059	1277	1198	+8.26	0.02	210	165	+30%	0.006
March 29 - April 4	91496	91613	2506	2521	-0.4	0.57	1084	1139	-4.7%	0.25	137	137	0.13%	0.5

Total	358979	361091	10582	10470	1.6%	0.11	4531	4412	+3%	0.06	734	660	+11.8%	0.018

1st + 4th weeks	179854	180632	5091	5111	+0.04	0.5	2152	2177	-0.72%	0.8	324	328	-0.79%	0.92
2nd + 3rd weeks	179125	180459	5491	5359	+3.23%	0.04	2379	2235	+7.24%	0.017	410	332	+24%	0.003

Re: How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

statman — Wed, 05 Apr 2023 23:15:46 GMT

I apologize, I don't have time right now to enter your data into JMP and take a look, but, in general, you seem to have either a Block effect or a Block-by-factor interaction effect. Where you would want to investigate is: Why are the results different between weeks? Make sure the differences first pass the sensibility test. That is, does the data vary enough for further analysis. Is there enough variation in the Y's to be of practical importance? Then, proceed with analysis of a Block design (this varies depending on whether you consider the Block effects to be random or fixed).

Blocks as used to partition the noise in an experiment. The method is quite useful as you can reap the benefits of both increasing the inference space of your experiment without compromising precision (and, in some cases, assess the robustness of your model).

Re: How to interpret results of an experiment where they are statistically significant during 2 weeks and are not during 2 other weeks?

gustavjung — Thu, 06 Apr 2023 12:19:59 GMT

Thank you! I will look into this from Block design standpoint.

I have added confidence intervals for absolute and relative effects of the last Y (success step 3).
Can we conclude from this data that because CIs of relative effects are overlapping, then the 2 other weeks of the same experiment are just a result of variation? If so perhaps we can just be ok with that 2 weeks showing different results, since this is a behavioral online experiment.