Sample size & Statistical power

[#1]

Hi,

I have done a finite element model validation study by comparing the a certain quantity of interest to experimental data. I have conducted this study for 18 different samples in order to evaluate the mean prediction error of the model. The real & simulated samples are dependent.

I want to assess if 18 was a statistically significant number of samples.

How can i calculate the power?

Thanks for your help

[#2]

First, welcome to the community.  I'm not going to be of much help as I can't answer your question.  There is some missing information:

1. What do you mean by statistically significant?

2. You sampled 18 out of how many?

3. How were the samples acquired (randomly, systematically, et. al.)

4. What do you want to do with your model?  Predict future results? Explain variation present in the samples?

5. Is the process stable?

6. What alpha and beta risks are you willing to have?

7. Was your experiment replicated?  If so over what inference?  Is that inference representative of future conditions?

8. Do you know your measurement system error?

Statistical significance is a conditional statement.

[#3]

Hello Statman,

Thank you for your response. You can find my answers below :

1. What do you mean by statistically significant? Power > 0.8

2. You sampled 18 out of how many? Of all possible geometric possibilities. Goal was to include the widest design space possible inside those 18 samples to be able to demonstrate that the predictions obtained by the model were accurate and precise.

3. How were the samples acquired (randomly, systematically, et. al.) Randomly, but see response to #2

4. What do you want to do with your model?  Predict future results? Explain variation present in the samples? Predict future results

5. Is the process stable? I'm not sure what do you mean here?

6. What alpha and beta risks are you willing to have? 0.05 and 0.2 (power 0.8)

7. Was your experiment replicated?  If so over what inference?  Is that inference representative of future conditions?

8. Do you know your measurement system error? Mean absolute error prediction by the model is 8.1%. Average experimental variability was 5.0%

I hope that helps 

[#4]

Hi @TraceMatrices73 . For me, some clarity is needed here. You say power > 0.8 is what you are after. 
Power = Probability(reject H0, given Ha is true). What are the null and alternative hypotheses, H0 and Ha, respectively? Practically speaking, Ha is what you want to “prove”, and H0 is opposite of Ha.

[#5]

Hi MRB,

Thank you for your message.

What i want to prove is that the predictions obtained by the model correlate well with experimental results. For example, this could be demonstrated with an intraclass correlation coefficient greater than 0.9. Alternative hypothesis is that predictions by the model do not correlate well with experimental results.

[#6]

Hi @TraceMatrices73 : Excellent, that is something we can work with! To make sure we are all on the same page…for a one-way random effect ANOVA, the ICC is defined as follows.

Let V(b) = Variance between groups/classes/whatever.

Let V(w) = Variance within groups/classes/whatever.

Let V(t) = Total Variance = V(b) + V(w).

Then ICC = V(b)/V(t).

And your hypotheses are then:

H0: ICC <= 0.9

Ha: ICC > 0.9

Are we on the same page? If so, what are the groups/classes? I.e., what is the categorical factor/effect in the one-way ANOVA?