Sample size explorer - Power for two independent sample equivalence - Help needed

[#1]

Dear JMP community,

I would like to perform equivalence testing using TOST (comparing 2 samples). With JMP 17, I can see in the DOE section that we can perform some Power calculation with a power explorer. I assume that the appropriate power explorer before performing a TOST would be the “Power Explorer for Two Independent Sample Equivalence”.

I am trying to get it to work but I am a bit struggling. I am filling up the margins by calculating the Tolerance Intervals from the pooled values (samples 1 and 2). Is that correct?

I report the std dev from the individual samples.

 My sample sizes for sample 1 and 2 are locked.

I am a bit unsure about what value to put in the difference to detect.  

Thanks for you help!

Best,

Sébastien

I am using JMP v.17. @MarijaLind @spyros_megalou 

[#2]

Unlike hypothesis tests for a significant difference, the tests for significant similarity require to you specify what is practically the same. The difference to detect is the smallest one you would consider practically different. A smaller difference would be judged to be practically equivalent. For example, if you consider two populations practically equivalent if they are within +/-2, enter 2.

[#3]

Thanks Mark,

Let me ask a few more questions to make sure I do it right: 

- How to define the Margins then? I understand that the upper margin "Specifies the maximum value, above which the mean is considered different from the reference mean". How do I obtain this value? I tried with tolerance intervals. I also tried to fill up with (Upper TI-Lower TI)/2 and tick the symetric bounds. it doesn´t work still.

- Again for the "difference to detect", do you feed the profiler with (Max value of pooled data - Min Value of pooled data)/2? 

Best,

Sébastien

[#4]

Hi @Sebastien_ : Reading this entire thread, and perhaps reading between the lines some, perhaps you already have data and want to test for equivalence? Or are you planning an equivalence test and want to know sample size?   

[#5]

Hi @MRB3855 ,

Appreciate your help! I will have a look at the paper you sent. 

I am sorry if my goals were not clear.

I have now data from two independent experiments. With these data, I want to check what is the power value. And yes, equivalence testing will be perform. 

My goal is for the future experiments, to try to establish what will be the minimum sample size to use to get a power of 80%.

Best,

Sébastien 

[#6]

Hi @Sebastien_ : Ok, thanks for the clarification. The data you currently have may inform a few things.

- Difference to Detect: Assuming the data you have is representative of the two groups you wish to compare, the difference in the means (or the upper confidence bound) could perhaps be used.

- Std Dev (Group 1 and Group 2) : Assuming the data you have is representative of the two groups you are comparing, the

  sqrt(pooled variance) or its upper confidence bound could perhaps be used here (assuming the Std Dev is about the same for each group) 

This kind of power calculation can be thought of as sort of a "what if" analysis; i.e., if the inputs you provide (Diff to Detect, Std Dev) are true, and if your desired alpha, power, and margins, are as you input, then you can calculate power. And remember, Power is just another word for Probability (Probability of rejecting H0 when Ha is true), where:

H0: delta < Lower Margin OR delta > Upper Margin

Ha: Lower Margin <= delta <= Upper Margin 

i.e., Ha is what you want to show; it will be shown, once you collect the data, if the 100*(1-2*alpha)% confidence interval for the difference lies entirely within the Margins.       

[#7]

Determining the power in a previous analysis is a no-no. It is a retrospective power analysis, generally held to be invalid and a bad practice. Think about a lottery. Would you calculate the probability of a ticket winning a lottery that is already over and a winner chosen?

On the other hand, a prospective power analysis is valid and useful. The existing data might be used to estimate the means and variance that are part of the prospective power calculation.

[#8]

Hi @Sebastien_  as @Mark_Bailey  said, calculating power after you've collected the data is pointless.  Current data can inform a future study, but checking the power value from the data you have is useless.

Here is a good article with good references. https://library.virginia.edu/data/articles/post-hoc-power-calculations-are-not-useful 

One of the references here:  https://www.zoology.ubc.ca/~bio501/R/readings/hoenig%20&%20heisey%202001%20am%20stat%20-%20fallacy%2...

[#9]

The data do not determine the margins and difference. You determine them. They represent what you need for equivalence.

[#10]

Hi @Sebastien_ : To expand of  what @Mark_Bailey  said, you would not use a Tolerance interval in that way. Tolerance intervals tell you something about the range of individual values. Equivalence testing is about comparing means, and "proving", with some level of confidence, they (the population means) are close enough to each other (within the Margins, as defined by Lower Margin and Upper Margin) to be considered "equivalent". What should the lower Margin and Upper Margin be? That is not a statistical question. It will depend on the application. The Difference to Detect should be some value within the Margins (Difference to Detect  is what you believe the difference in population means actually is), and may be informed by past data. 

This link below may be helpful; it is a particularly readable explanation of Equivalence Testing.

 https://pubs.acs.org/doi/pdf/10.1021/ac053390m