Showing results for 
Show  only  | Search instead for 
Did you mean: 
Bootstrap Interval Calibration Using New “Save Simulation Formula” Feature in JMP® Pro 16 (2021-EU-30MP-744)

Level: Intermediate


Hadley Myers, JMP Systems Engineer, SAS
Chris Gotwalt, JMP Director of Statistical Research and Development, SAS


The calculation of confidence intervals of parameter estimates should be an essential part of any statistical analysis. Failure to understand and consider “worst-case” situations necessarily leads to a failure to budget or plan for these situations, resulting in potentially catastrophic consequences. This is true for any industry but particularly for pharmaceutical and life sciences. Previous work has explored various methods for generating these intervals: Satterthwaite, Parametric Bootstrap and Bias-Corrected (Myers and Gotwalt, 2020 Munich), and Bias-Corrected and Accelerated (Myers and Gotwalt, 2020 Cary), which were all seen to have error rates that were too high for the small samples typical in DOE situations. Therefore, we make use of the new “Save Simulation Formula” feature in JMP Pro 16 in an add-in that improves upon these by allowing users to perform a “Bootstrap Calibration” on the Satterthwaite estimates. The add-in also includes the ability to do this for linear combinations of random components, taking advantage of another addition to JMP Pro 16. Further, we investigate a new version of the fractionally weighted bootstrap that respects the randomization restrictions of variance component models, as an alternative to the parametric bootstrap, using the new “MSA Designer” debuted at this conference.



Auto-generated transcript...



Hello, my name is Chris Gotwalt. My co-presenter Hadley
Myers and I are presenting an add-in for obtaining improved
confidence intervals on sums or linear combinations of variance
components. This is part of a series of talks we have given as
we work on improving and evaluating several approaches.
Obtaining confidence intervals on sums of variance components
is important in quality because it provides an uncertainty
assessment on the repeatability plus the reproducibility of our
measurement system. The problem is that when we ask for a 95%
confidence interval, there are approximations involved and the
actual interval coverage can be as low as 80%.
In our previous studies, we found that two methods have
improved coverage rates, parametric bootstrapping and
Satterthwaite intervals, but it was still less than 95% in small
samples. The earlier version of the add-in implemented the
parametric bootstrap as a stopgap and Elizabeth Claassen
implemented the Satterthwaite intervals in the Fit Mixed
platform natively in JMP Pro 16. I want to stop here to give
Elizabeth Claassen credit for making interval estimation of
linear combinations of variance components so much easier and
JMP Pro 16. She also greatly extended the Mixed Model output,
which has made this presentation vastly easier. I'm also hoping
that this presentation will serve as an inspiration to
others to check these new Save commands out so they can get
more from JMP Pro’s Mixed Model capabilities. Now we're going to
combine the two approaches using a technique called Bootstrap
Interval Calibration that was introduced by Loh in a 1991
Statistica Sinica article. Bootstrap Calibration is a very
general procedure for improving the coverage of confidence
intervals that can be applied to almost any parametric
statistical model. I'm going to introduce the basic idea of
Bootstrap Interval Calibration in the simplest terms that I
can, and hand the mic over to Hadley, who's going to demo the
add-in and discuss our simulation results. To make this
simple, let's make it specific. Consider a very small nested
Gauge R&R-type study where we want to estimate the total
variation. We collect the data and run a nested variance
components model with an Operator effect, a Part within
Operator effect, and a residual effect. The software reports a
Satterthwaite-based interval on the total. It's well known that
this is an approximation that assumes a “large”
amount of data is present in order for the actual coverage of
the interval to be close to 95%.
In small samples, the actual coverage, the probability that the
interval procedure generates intervals that actually contain
the true value of the estimated quantity, will tend to be less
than 95%. Thing is the actual interval coverage is a
complicated function of the design, true values of
the functions, and a long list of other assumptions that are hard
or impossible to verify. What we can do though is used the fitted
model and their parameters to do a parametric bootstrap. When we
do this, we know the true value of the quantity we are
estimating because we were simulating using that value.
We can do the simulation thousands of times. We apply the
same model fitting process to all the simulated samples. We
can collect the intervals from JMP and calculate how often
they contain the generating value of the quantity that you
were interested in. In this case we were interested in the sum of all
the variance components, so the true value is 4.515.
Suppose we took our original data set, took the estimates,
use the Save Simulation Formula that is comes from Fit Mixed,
and generated a large number of new data sets, and applied the
same model fitting process that we applied here to each of them,
and we collected up all of the confidence intervals that were
reported around the total. After having done this, suppose
that that...the estimated coverage, the estimated number
of times that these intervals actually contained the truth,
turned out to be 88%.
So we wanted that 95% interval, but the Bootstrap
procedure is telling us that the actual coverage is closer to
88%. Now we can play a little game and we can repeat the
Parametric Bootstrap using a 99% interval this time. So we go
through that process, we redo all the bootstrap intervals and
when we did the 99% interval we get an actual coverage of
approximately 98%. Now suppose we did this game over and over
again until we found an alpha
with actual coverage approximately 95%. So in this
case, suppose we did that and we ended up with finding that 97.6%
when we asked for a 97.6% interval, we actually got
something like a 95% coverage.
Then what we can do is set 1 minus alpha to 0.976 using the Fit
Model launch dialogue, set alpha option and will get an
interval that has been Bootstrap Calibrated to have
approximate coverage 95%. This is still an approximation. There
is still a simulation component to it, as well as a deeper
underlying approximation that is extraordinarily hard to analyze,
but it can be made easy to use, and this is where Hadley comes
in. Now I'm going to hand it over to him and he will demo the
add-in and go over the simulations that he did that
show that we are able to get better coverage rates than
before by applying Bootstrap
Calibration to Satterthwaite intervals on linear combinations
of variance components. Take it away Hadley.
Thank you very much, Chris, and hello to everyone watching online
wherever you are. So I'm going to start out by showing you how the add-in
works and how you can use it to calculate Bootstrap Calibrated
confidence limits for random components in Mixed Models in
JMP Pro 16.
And from there we'll take a step back. We'll see how the add-in
makes these calculations and I'll highlight some of the additions
to Mixed Models in JMP Pro 16 that allow it to do that.
From there, I'll show you the results of some simulation
studies to give you an idea about how accurate this
interval estimation method is, the Bootstrap Calibration
method, and how it compares to some of the other methods for
calculating confidence limits, as well as the situations where it's
more or less accurate and some of the limitations and things you
should be aware of if you're going to be applying it.
We’ll discuss possibilities for improvements in future work
just briefly, and from there, I'll conclude by showing you the
new MSA Designer, Measurement Systems Analysis Designer,
available from the DOE menu in JMP Pro 16 so that you can quickly
and easily design and analyze your own MSA Gauge R&R
studies. So let's start out by looking at this data set.
This is one that I pulled from the sample data files.
I'm going to run this Fit Mix script here that I've saved. So
what we've got here are our random estimates, estimates
for a random components.
it could be that you want to, for some reason, calculate an
intermediate total, for example Operator and Part nested with
Operator, or the three of these,
you know, Operator and residual.
So to calculate those is very simple, we simply add these
estimates, but what's not so simple is to determine those
confidence limits.
There's a new feature in Mixed Models that's been added in 16.
The linear combination of
The linear combination of variance components feature
right here, and so what you can do is you can click that.
You can choose the combination of variance components that you're interested in,
and you can press done. So now we have an estimate for those.
Components as well as their
confidence limits. So,
what I'm going to do now is I'm going to take this one step
further and I'm going to calculate the Bootstrap
Calibrated Satterthwaite estimates and I'm going to do that by
going to my add-ins and clicking the Bootstrap Calibrated
confidence intervals there. So from here we can estimate the
number of simulations.
2500 is a
recommended number to the default number. It's also the
default number in some of the other simulation platforms and
in JMP Pro. I'm going to choose
this one. But one thing to note is that it takes some
time to be able to do this, and so in the interest of time
what I'm going to do is I'm going to stop it early.
And here we have our
calibrated intervals, calibrated upper and lower confidence
limits added to the report.
So let's take a step back and see what happened there.
I'm going to go ahead and add this again.
Now, one thing that the add-in
does, as soon as you run it,
is it adds
this simulation formula to the data table, so you can see the
simulation formula here.
When the add-in is closed,
the simulation formula disappears.
The simulation formula there takes advantage of another
feature that's been added to...
to the Mixed Models platform in JMP Pro, and that is the Save
Simulation Formula feature
here. So what this would allow you to do is to save the
simulation formula and then to
use that, for example, to simulate
these values here. So, we can swap out our “Y” with our
new simulation formula,
and go ahead and run that. So when you run the add-in, this is
all done in the background. But this is how the add-in goes
about calculating these intervals. So I'm going to stop
this early, once again in the interest of time.
And now we see here the samples
estimated for each.
simulation. And so how the add-in works is
it takes all of these.
And it calculates new estimates for the upper and
lower Satterthwaite intervals from this
estimate and this standard error, swapping out different
values for alpha. So what we're aiming for 0.05, right?
So that we get 95% upper and lower limits, and what it
does is it finds
an alpha value that results in 95% coverage, that is 95% hits
and 5% misses,
swaps that in, that's how you get your calibrated intervals.
So I hope you enjoyed seeing that. I hope you find it useful.
We've done some simulation studies and what we found out
is that
the intervals, which you can see here for four operators
and 12 days as our random components,
we've achieved misses of about
7%, so a 92.8 hit ratio. Now this is better than all of the
others, including this, so the linear combination, which is
simply the standard Satterthwaite interval
calculated on the combination of linear components, as well as the
Bootstrap quantiles, the bias-corrected intervals in the
bias-corrected and accelerated intervals, but as you'll see
these intervals improve, all of them,
as you increase your number of Operators from
4 to 8 and the number of Days from 12 to 24. So increasing the
levels of these random components
result in much better, much more
accurate estimates for the confidence limits, and so much so
that we now have
a method here
that is equivalent,
just, to an
alpha value of .05.
So. this improvement in performance of course, comes at
a cost, and one of those costs is the length of the intervals.
And so you can see here,
that with our
Bootstrap calibrated, well with all of our intervals in fact,
that when we have increasing number of Operators,
that the length of the interval is much more
bundled closer to 0 than it is when you've got smaller
number of Operators. You can see that this tails out much
further, so that's this blue area here. That's true for
all of them, but it's especially true for the
Bootstrap Calibrated interval. You can see this
long tail here.
On average, you're going to get longer lengths using this
method, but you have a more accurate method.
Exploring that a little bit deeper, you can see here
that this increase in length is true for four Operators, as
well as eight Operators, and it is significant.
Statistically significant.
The other thing that I looked at,
is the effect of adding repetitions, so the difference
between two repetitions and five repetitions, and what you'll see
here is that there really is no
difference. So looking across
the different sets of combinations from
four Operators and two reps to four Operators and five reps, about 6
measurements total versus 3 measurements, we really don't
gain anything. All of these are equivalent to each other.
So that's something to be aware of, that you see improvements in
accuracy when increasing the number of Operators, and you
don't see improvements when increasing the number of
One thing that I'd like to mention as a
possibility to improve upon these results is the
Fractional Random Weight Bootstrap, which we would have
liked to have been able to implement for this in time for
this conference. We weren't able to do that, to take this
and to apply it to random variance components, and so we
hope to be able to do that in future work and perhaps even
see an improvement upon the Bootstrap Calibrated interval.
And then the other thing that I'd like to highlight before I
go is the new
MSA designer that's been added to JMP 16, and so
from here what we can do is we can very quickly
create our own design in order to be able to
perform our own MSA or
Gauge R&R analysis. And so let's see, I'll do this with three
Operators and Five parts.
I'll label these
A, B and C.
And we'll do one repetition of each. So that's two
measurements total.
So here we've got a table with our
design. What I can do is I can press this button to very
quickly send that to the different operators, have
them fill out their parts, send that back to me.
And then I can add
those results together.
So I'll just sort this because I've got another table over here
where I've done this ahead of time. So I'll just add these
values over there. And now from the scripts within the table we
can quickly and easily do our own Measurement Systems Analysis
and Gauge R&R. So I hope you found this useful. I hope you
continue to enjoy the talks at this conference. Thank you very
much for listening.