Discussions
Solve problems, and share tips and tricks with other JMP users.
When doing method validations, we use Variability to separate variance Between and Within occasions. Nowadays guidelines ask us to not only give the estimate, but also confidence intervals for the variances. I seem to to get the CIs directly in Variability (95% Upper), which I understand use Satterthwaite to calculate the degrees of freedom.
For my understanding, I am trying to do the same calculations with formulas as below to see if I get the same numbers. From the ANOVA I get DF for each level, and the variance components, but got confused on what to put in as n? I tried using DF or DF+1 to mimic n, but I always get numbers a few percent off from those given in the ANOVA table, for Total but also for Between. For Within, I get exactly the same numbers.
Have I underestimated the complexity of the calculations? I read some discussion with similar titles, but my knowledge was not enough to interpret the responses, so here goes nothing..
Accepted Solutions
Hi @Ake, As we discussed, the issue is not a bug, although it might appear strange . It's unfortunately an artifact of the Satterthwaite method with small sample sizes -- and it relates to numerical instability in this situation but also it relates to the inflation in uncertainty we get with these interval estimates when the sample size is particularly small. The calculations aren't trivial unfortunately and insofar as I understand, they cannot be reproduced using simple mathematical formulas (at least not in the form of JMP's internal implementation).
We can begin to understand the uncertainly inflation though in relation to the Wald approximation for these intervals (which we implement in the Fit Mixed Platform in JMP Pro, Confidence Intervals for Variance Components; to be clear, in Variability/Gauge we use the Satterthwaite method only, even if we use different methods to calculate the point estimates for the variance components themselves see: Variance Components).
Looking at the formula for the confidence interval on an observed standard deviation (which uses the chi-squared distribution), we can plot this for increasing sample sizes in Graph Builder, assuming e.g. an observed std dev of 1, at a 2-sided 95% confidence level. Here we can see how the error on the upper side for smaller samples is much higher, and is particularly high with a sample size of approximately 6 or less.
The result shown above is saved as a script in the attached JMP data table, and the formulas as referenced therein in the Columns for the Upper and Lower CI are adopted from the book Introduction to Probability & Statistics,5th edition, 1979, by W. Mendenhall (see p. 303-304)
Hi @Ake, As we discussed, the issue is not a bug, although it might appear strange . It's unfortunately an artifact of the Satterthwaite method with small sample sizes -- and it relates to numerical instability in this situation but also it relates to the inflation in uncertainty we get with these interval estimates when the sample size is particularly small. The calculations aren't trivial unfortunately and insofar as I understand, they cannot be reproduced using simple mathematical formulas (at least not in the form of JMP's internal implementation).
We can begin to understand the uncertainly inflation though in relation to the Wald approximation for these intervals (which we implement in the Fit Mixed Platform in JMP Pro, Confidence Intervals for Variance Components; to be clear, in Variability/Gauge we use the Satterthwaite method only, even if we use different methods to calculate the point estimates for the variance components themselves see: Variance Components).
Looking at the formula for the confidence interval on an observed standard deviation (which uses the chi-squared distribution), we can plot this for increasing sample sizes in Graph Builder, assuming e.g. an observed std dev of 1, at a 2-sided 95% confidence level. Here we can see how the error on the upper side for smaller samples is much higher, and is particularly high with a sample size of approximately 6 or less.
The result shown above is saved as a script in the attached JMP data table, and the formulas as referenced therein in the Columns for the Upper and Lower CI are adopted from the book Introduction to Probability & Statistics,5th edition, 1979, by W. Mendenhall (see p. 303-304)