topic Re: Sample Size and Power for Testing standard deviations in Discussions

Sample Size and Power for Testing standard deviations

JesperJohansen — Mon, 06 Mar 2017 07:59:27 GMT

I am wondering why the Sample Size and Power Calculator does not have an option to estimate sample size for test of standard deviations for two populations. I can find graphical methods in appendices of some statistics textbooks, but they must have come from some computation. How do I estimate a sample size to distinguish standard deviations with a certain ratio with a certain power? I realize that the solution to the problem must be numerical, but it should still exist, shouldn't it?

Re: Sample Size and Power for Testing standard deviations

Mark_Bailey — Mon, 06 Mar 2017 22:16:02 GMT

I wonder about a lot of things, too.

If you have JMP Pro 13, you might be able to use the new Simulate feature to assess power and determine appropriate sample sizes. I just confirmed that this method works nicely. In case you are unfamiliar with it, try the following:

Create a data table with a column for your grouping variable. I called it Group. I used the initial Column Info (or New Column) dialog to set the initial values to a Sequence of 50 replicates (rows) of 1 to 2 in steps of 1. So each group has a sample size of 25.
Create another column for the response. I created one called Y.
Create a third column for the Monte Carlo simulation. I created one called Y Formula. My formula for your purpose was Random Normal() * Choose( Group, 2, 3). This formula says that population 1 has a standard deviation of 2 and the other population has a sd of 3. Substitute the values of 2 and 3 for ones that makes sense for your alternative hypothesis test.
Perform one analysis (note that the names are the ones tha I used above).
1. Select Analyze > Fit Y by X.
2. Select Y and click Y.
3. Select Group and click X.
4. Click OK.
5. Click the red triangle and select Unequal Variances. This is the test you will use.
6. Now, right-click on the p-value column and select Simulate.
7. Select Y for replacement and Y Formula for the simulation.
8. Enter a larger number for the quantity of simulations if you need an answer with greater precision (narrower confidence intervals).
9. Click Go.
10. It takes a little while to complete all 2500 (default) simulations.
11. Click the green triangle next to Power table script.
12. Examine the information for the F-Ratio.

It is not likely that they initial guess of a sample size gives you the desired power. You can evaluate another sample size by simply change the rows with Group = 1 or 2 and repeating the simulation. For example, double the number of rows to see the improvement in power for doubling the sample size. (Hint: with now columns or rows selected, press CTRL-C and then CTRL+V)

How does this method work for you?

Re: Sample Size and Power for Testing standard deviations

Mark_Bailey — Mon, 06 Mar 2017 22:17:42 GMT

I am working on a calculator for you. It mimics the behavior of the Sample Size and Power calculators that are available through the JMP DOE menu. You can specify the sample size and compute the power. You can't go the other way yet. (It ain't fully cooked yet. If you try it, you break it.) It allows you to choose a two-sided test, lower one-sided test, or upper one-sided test. It assumes that you want a balanced or nearly balanced design. That is, if you enter 50 for a sample size, then it assumes that you will obtain a sample of 25 from each population.

By the way, the empirical power calculation available through the JMP Pro 13 Simulate feature that I mentioned in my previous reply and this exact closed form calculation agree very well. (But we don't always have a closed form to fall back on - love the new Simulate feature!)

Open and run the script. You must enter the alpha significance level of your test and the alternative hypothesis as the standard deviation for population 1 and 2. (Population 1 is the numerator of your F-test.) Then enter the candidate for your sample size and click Continue. Very fast way to evaluate the power with different sample sizes.

Re: Sample Size and Power for Testing standard deviations

JesperJohansen — Tue, 07 Mar 2017 06:37:02 GMT

Unfortunately I don't have JMP Pro, so the Sumulate feature is not avaliable to me.

Re: Sample Size and Power for Testing standard deviations

JesperJohansen — Tue, 07 Mar 2017 06:39:39 GMT

Hi Mark
This sounds great. You say that you are working on the script, but you also write "Open and run the script.". Should I have received it? I don't see it anywhere.
BR
Jesper

Re: Sample Size and Power for Testing standard deviations

JesperJohansen — Tue, 07 Mar 2017 06:43:28 GMT

I guess my question had two parts:

Can I fin the feature somewhere? The short answer is apparanetly no, but there are ways to work around this.
Why cant I fin this featrure anywhere? It seems like a peculiar omittance. I cant find the answer anywhere in litterrature either. Am I relly the one who might want to design a simple experiment to destinguish variances of two polulations?

Re: Sample Size and Power for Testing standard deviations

Mark_Bailey — Tue, 07 Mar 2017 16:21:06 GMT

I thought I attached it. Sorry about that. Again, it currently works in one direction only, Enter a candidate size and leave the power missing, then click Continue. I hope to get it working in the other direction but for now you can get useful information as it is.

Edit: DO NOT USE THIS SCRIPT. I removed it from the post. It does not compute the power correctly for the lower one-sided test. I corrected the error and finished the estimation of the sample size, so now it works in both directions. See the new attached add-in.

Re: Sample Size and Power for Testing standard deviations

JesperJohansen — Tue, 07 Mar 2017 11:15:46 GMT

Relly great. Thank you.

The sentence "it currently works in one direction only" suggests that an update is coming which will work to estimate sample sice from power too?

I see you are "staff". Any reason not to implemet this functionality in the Sample size an power calculator for all users to enjoy?

Re: Sample Size and Power for Testing standard deviations

Mark_Bailey — Tue, 07 Mar 2017 12:09:09 GMT

Answer to first question: I am working on getting it perform in both directions to help you and anyone else who might use the script but it is not a high priority because you can get an answer easily and quickly with it working in only one direction.

Answer to second question: I am staff but I work in SAS Education as a JMP instructor. It is up to JMP Development to decide if it should be implemented in the product. Of course, with a script or add-in to do the work, it is a lower priority for them, too.

Be patient.