Models, Bootstraps, and Monte Carlo...Oh My! Designing With Small Samples (2025-US-30MP-2503)

Hey everyone. My name is Dale Human. I'm an engineer at GE Healthcare here in Milwaukee, Wisconsin. Today I'm going to talk to you a little bit about making models, using bootstrapping to evaluate what you expect, the variation from those models to be, and then if you want to go a little deeper, you can use Monte Carlo simulation to further evaluate the variation you expect in your system based on variation on your inputs. There's a lot going on. Let's get started. I'm going to go through quick motivation.

This all comes down to conditions where you're starting with a small sample size in new product development, R&D type situations, you do a few tests, and from those few tests or those few measurements, you're trying to make a decision on how you want to proceed or how you think your system might behave from those few tests. That's really a good area to apply tools like Bootstrapping and Monte Carlo simulation. Let's look at how we do that in JMP.

We're going to start by thinking about a fairly simple system. In this case imagine you have a clamping system. We've got a screwdriver here. It's being used to tighten down in some assembly. We've inserted a load cell into the assembly to measure as we talk down this screw. How much force is being applied to the assembly? A pretty typical mechanical application. We want to understand if I measure the applied torque on the screw and I get the measured force from that torque, can we build a relationship and try to model how this system will behave in the future?

I've got a sample data table that I'll open up here for that. You can see here we've done a few tests, nine tests in this case where we applied some torque to the screw drive, and we've measured the force. We'd like to be able to understand the relationship between the force and the applied torque. Let's make a model in this case, because we only have one response variable and one x factor one input variable, the torque, we can use the fit Y by X platform.

I'll go to the analyze menu and then choose fit Y by X. We can put the measured force as our response and the applied torque as our input into our dialog. JMP says it's going to do a bivariate analysis in this case because we only have two variables. I'll click Okay. As JMP does, we've started with a chart, just a scatter plot of the data. We can visually look for relationship between applied torque and the measured force. As you can see here, there seems to be some relationship. As I increase the torque the measured force tends to go up. That makes sense.

It also looks like it maybe starts to plateau a little bit over here as we get to larger and larger torques. Maybe in this system you could envision. At some point, the assembly is clamped completely, and so continuing to apply more torque doesn't really help you from a force perspective, but we do see that relationship. Let's try to build a model. I'm going to go to the red triangle here. You could do fit line if for a linear relationship, but if I think there's this plateauing effect, I'm going to go to the fit special dialog instead, which allows me to choose some transforms on my X's and Y's.

In this case, I'm going to choose the natural log of X, which will help try to capture the idea that this system maybe has a plateauing effect here, so I'll choose natural log. I also turned off the centered polynomials option here. You could keep that on. It doesn't change anything. It's just a different way of looking at the equation. For now, I've turned that off. I'm just going to click OK. We can see here's our best fit line. This is done essentially a least squares regression using the natural log of the applied torque in the analysis, and our best fit curve. It seems to match our data pretty well.

That's good. We have the equation that has been developed from the analysis here, so we could see that. If you've done regression before you've probably looked at Rsquared values. We have our Rsquared values here which look very nice. They're very high above 90%. The adjusted Rsquared and the Rsquared are close to each other. In this case there's only one X. You would expect that to be the case.

If you look at the Root Mean Squared Error which is really like the standard deviation of the model if you will. It's a little large, maybe. It's a little hard to see that. I'm actually going to go to the fit that we've applied and turn on the confidence interval for the fit by going to the confidence shaded fit option here. We can see the uncertainty in the model there. You can look if I were sitting at say 100 Newton meters of torque her, my expected variation in my average measured force is somewhere around 1000 Newton meters. That the width of the confidence interval there.

That's a little large. Maybe you're thinking, well, you only have nine data points, so there's a lot of uncertainty in this model, and that's probably true. We also have the parameter estimates analysis down below here, which shows you, of course, the values of the parameters of the model, the slope and the intercept values which are here in the equation also. Then you also get a sense of the standard errors for these values, which is a quantification of the uncertainty in those values.

You can see there our intercept has a standard error of about 2000 newtons, and the slope term has an uncertainty of about 500 newtons meters I should say per Newton for the slope. We are trying to understand should we really expect this amount of variation going forward. If we were to gather more and more data, would we really expect the uncertainty to be this large?

Given the fact that we've made this model with only a small sample? Nine data points in this case. How do we evaluate what we really expect to happen, or are we overestimating our error in this model because we only have a small sample? One way you might do that is by using a technique called Bootstrap.

Let's talk about Bootstrap. Bootstrapping is really a "Simple" technique, where you are resampling from your original data many, many times to try to re-estimate the values of the parameters that you're calculating. In this case, we would re-estimate the slope and the intercept of this model many, many times to get a better sense of what's the real uncertainty that we might expect in this system going forward. That's the fundamental idea of what bootstrapping is doing. Visually, I'm showing in the figure here. We have a small sample of data in this example three data points.

What bootstrapping does is it grabs a random sample from your original data set, so here we have one random sample in this box here, it has three data points in it again, but I've chosen in this case randomly I happen to choose row number 3 twice and row number 1 once. Row number 2 didn't appear. You do this process many, many times, hundreds or thousands of times. Every time you grab a new sample, you then recalculate whatever parameter of interest you're looking at for your system. In our case, we'll ultimately want to look at the values for the slope and intercept of our model.

Whatever the parameter of interest is, you calculate that parameter for each of these different samples, and you do this many, many times to try to help understand what's the expected variation, even though I've got a small data set. That's the fundamental idea of bootstrapping. It's a common technique. It's nice because it's non-parametric. It doesn't have any assumptions about normality or things like that, so it applies in lots of different situations.

You can really use this anytime you have small samples. Again, the reason you might think about this is because you're trying to estimate the error in your model or in your calculation in your analysis. Oftentimes you're using things like variance to do those to look at the variation in your system. Remember variance you're dividing by n or n minus 1. If you have a small sample you're dividing by a very small number generally, and so you tend to inflate the variation that you might expect, but that might not be a realistic expectation if you were to continue to gather data. Bootstrapping is a way to try to get an estimate of that more realistic variation given that.

That's the basic concept of bootstrapping there. As a simple example, before we come back to the model that we've made, we can look at, for example, the distribution of values that we have here. Again we have nine values of measured force. Let's do a distribution analysis on that to start with. I'm going to go to the Analyze distribution menu and just look at force for now for this example. Let's look at the distribution of values.

I have that here. You can see you get a table of summary statistics I'm going to hide the quantiles for now. That's not really what I'm focusing on, but if you look at we have an average value of about 7100 Newtons in this case and a standard deviation of about 2000 newtons. A pretty large standard deviation because we've got a small sample of data here. This is being inflated by that small value of n.

Is there a way do we really expect this to be the variation in our population going forward or not. Given the fact that we've only measured nine units from the population? We can try bootstrapping to understand is this a realistic expectation for our variation or not? How do you do that? A JMP makes it very simple. Since we have this table with the parameter of interest, we're interested in the mean and the standard deviation. They're both in this table, so I'm just going to Right-click in that table. You'll see you have all the menu items that you might have seen before, but down at the bottom is bootstrap.

This allows you to resample your data many times. For each one of those individual samples it will recalculate all of these statistics. Then you can look at the distribution of those values to try to get a better understanding of how your system might behave with larger and larger samples. The default number of samples is 2500, so JMP will look at the values of my measured force, and it will grab nine values randomly from this original nine. Again, sometimes it's grabbing the same value more than once as we saw over in the figure.

You don't always get the exact same nine data points. That's that's the good thing about doing bootstrapping. It's going to do that 2500 times by default. I'll just click okay there and we'll let that run. It'll take just a couple seconds. This is a pretty simple calculation. Just be aware that if you have a very complicated model, doing a large number obviously could take some time. If you have a very computationally heavy model that you're trying to calculate. In this case we're just calculating means and standard deviations. That's pretty straightforward. Now we have a new data table that JMP has created.

You'll see that it has 2501 rows. Because it resampled our original data 2500 times. The first row is just the values from the original data itself. That's row zero. Then all of the other rows are resampled calculated values for each of the statistics that are in this table. If I bring this out of the way, because all of these values are in the same table, JMP is trying to bootstrap every one of these statistics. We don't really care about all of these other values at this point. We really want to focus on the mean, because I really want to understand what is the expected variation in my average performance of my system, in this case the average force.

I'm going to focus on the mean. JMP automatically builds a distribution analysis for each variable in your table. Over here in the distribution. We can run that script, and it will just open up the analysis of all these different variables. Now again we really want to focus on the mean. I'm going to hold down the control key and hide all of this distribution analyzes. Then just open up the mean just to make the screen a little bit easier for us all to see here, and I'm going to bring that next to my distribution analysis here

I'm going to hide the quantiles again. I'm not focusing on quantiles. That might be something you're interested in, but if we look at the values that have been calculated here. Here are 2500 different values for the mean that are coming from resampling this sample here. If I look at the mean it is about 7100 Newtons very close to what we started with. The standard deviation however, is much smaller. We originally had a standard deviation of about 2100 Newtons.

Now it's only about 670 Newtons, and that's because we are resampling from this large population many times. Our estimate of our standard deviation is probably getting closer to what we should expect, the real variation in this population to be. Here we're inflating our variation because we have a small sample here. We're resampling from the same population. All of the data points that were used for all of these calculations come from the same population as these nine data points, because it's drawing from the same data.

This is a probably more realistic representation of the variation we would expect to see in this population. That's the power of bootstrapping. Now you might be thinking, well if I just take an even larger bootstrap sample, does my standard deviation continue to shrink? Let's try it. I'm going to come back to my original data. Right-click in this table and choose bootstrap. This time let's do 10,000 bootstrap samples. I'm going to click OK.

Again it'll take a few seconds to run, so we'll let that run. Maybe more than a few seconds. Now we have a data table with 10,000 rows in it, again, each of them being a different sample of the same population. And we can look at the distribution from this table. I'm going to open that up. Again I'm just going to focus on the mean. I'll close all the distributions and open up the mean. Let's put that next to all of these other windows. Sorry, I'll just move things around a little bit, so we can see them all together.

Now we have a bootstrap sample of 10,000 values. We can see the mean again is still around 7100, but you notice the standard deviation is still around 670. That really shouldn't be surprising because all of these values are being calculated from the same population. The population has some mean value. It has some variation, some standard deviation. We really shouldn't expect that to change because we've got very large samples in both of these analyses.

It's different from this one because this was a very small sample right. But this is really allowing you to see that going forward. You might expect variation more of around the 670 Newtons range versus the 2000 Newton range. And of course, there are some confidence intervals that are being calculated down at the bottom here, with different levels of confidence to give you a sense of what's that expected variation for your population.

Again, you'll notice the values for the bootstrap samples are very similar to each other, and they're slightly tighter than the original confidence intervals for the mean from our small sample. Because we're using a probably more realistic estimate of the standard deviation. That's the power of bootstrapping. It's allowing you to get a sense of variation in your system. Even if you started off with just a small sample of data. Let's close those out. Now that we have the basic idea of what bootstrapping is doing for us, I'm going to close these tables to get them out of our way.

If we come back to our journal here, I've gone through how you do the bootstrapping here and shown that. Your standard deviation shouldn't really change. Your mean shouldn't really change because those represent the population from which your data has been extracted to begin with. That's the idea of bootstrapping. Let's come back to our model. I'll just open up that section there, and we can do the same thing with this model. In our model here, we've calculated estimates for the parameters of the slope and the intercept of this least squares regression model. The question is do I really believe that my error in these estimates should be this large.

We can bootstrap these estimates just like we did for the mean of those forces. Again, all you really have to do is Right-click in the estimate column here of the table and choose bootstrap and choose how many you'd like to do. I'm going to stick with 2500 this time just so it's a little faster. Give that a few seconds to calculate. Now we again have a table of 2500 values. I'm going to look at the distribution analysis that JMP has created. There are only two values in this analysis because we only had two estimates one for the intercept and one for the slope.

Again I'm going to hide the quantiles by holding the control key and just closing both of those simultaneously. Here we had standard error for the intercept was about 2000 Newtons in the standard error for our slope was about 450 Newtons. What did we get from the bootstrap? If we look at our standard deviation of the bootstrap sample it's still about 2000, in this case 2200. Similarly, the standard deviation for bootstrap of the slope is about 475, a little different, but near the original value of 452 from the least squares regression analysis. You might be thinking, well, wait a minute. Bootstrapping didn't help, and you're right. Bootstrapping didn't help much in this case.

That's actually to be expected because if you recall from least squares regression, least squares regression is designed to choose the fit that minimizes the variance of the model. These values for the estimate of the slope and intercept are chosen specifically because they minimize the error in your model. For least squares regression, bootstrapping doesn't do much for you. It turns out. I just wanted to use that as an example of how to apply bootstrapping to a model.

Now, if you're using other modeling techniques that don't give you an estimate of error, maybe you're using a neural network analysis or principal components' analysis, and you don't have a direct sense from the analysis telling you what the expected variation is going to be, what your standard errors are. Then you might use bootstrapping to help get a sense of that error in that model. Here I'm showing you that well bootstrapping doesn't do much. It generally agrees with what you get from least squares analysis, because least squares analysis is designed to minimize that variation.

That concept applies to any modeling technique. Again, because bootstrapping isn't a parametric approach. It's not making any assumptions on how your data is distributed? It can be used in all of those different kinds of modeling techniques that you might be using. That's one way you might use modeling or sorry, bootstrapping to help understand error for least squares regression. Not super useful. If you just wanted to understand variation in your means, as we did here with the force, then you might use bootstrapping to say, okay, well, let me get a more realistic sense of my variation going forward, even before doing any more detailed modeling. You could use it that way as well.

That's the idea of bootstrapping. I've shown it in the simplest form of let's look at expected variation in a mean value. Also, you can use it to understand the error in the estimates from a model. Not so much least squares, but other techniques. It can be useful for that.

Another thing that allows you to evaluate the effects of variation on your system is Monte Carlo simulation and JMP also allows you to do Monte Carlo simulation very simply. It's part of the modeling platform. Let's look at Monte Carlo simulation. Fundamentally, what does Monte Carlo simulation do? If you have a model, or generally you have an equation. Here we've developed a model equation from our analysis. It's the same model here. What Monte Carlo simulation allows you to do is to inject variation into the inputs for that model, and evaluate the effects of that variation on your output factors, your response variables.

What you do is you assume some type of variation in your inputs. In this case, for example, I'm assuming the applied torque will have a normal distribution centered on a value of interest with some standard deviation. Let me go and randomly choose a large number of values from that distribution that I've chosen, and then plug that large number of values into this equation and calculate my output so that I can understand if this is how my input varies. Here's what I should expect my output to look like. That's the basic idea of Monte Carlo simulation.

Again, you do this for a large number of samples in general to help you understand what variation you should expect in your response. JMP also allows you to inject other kinds of variation into your output. I won't talk about that here. Just be aware that that is an option for you. How do we do Monte Carlo and JMP? It's very simple. Once you have a model. In this case, we've built this least squares model. I'm going to go to the model fit and turn on the profiler. That's where you'll start. If you've seen the profiler before, it's just a graphical representation of the model equation that has been determined right from your previous analysis, and you can play around with your input value X and change that and understand what happens to your Y.

The other thing you can do from the profiler is Monte Carlo simulation. If you click on the red triangle for the profiler, you can turn on simulator. That's the Monte Carlo simulator. This allows you to specify how does the value of my X change. Then shows you what will happen to your response variables as you do that. Down here I can choose from this drop down what variation my X has, in this case my applied torque. I'm going to choose random. Then, once you choose random, you can specify what distribution you want to use for your variation of that input.

Let's assume, for example, we want to operate this system at 125 Newton meters, so that's our typical value of torque. You've done some quick measurements using this torque wrench that you're going to be using, and you've determined that if I program it to be at 125 Newton meters, I'm generally at 125, but I have a standard deviation of, let's say, 3 Newton meters. You can put in the mean and the standard deviation for what you expect.

Then all you have to do is click the simulate button over here and JMP goes and calculates a large number, whatever number you put in this box in down here. In this case it did 10,000 values. JMP has gone and randomly selected 10,000 different values from the normal distribution that has a mean value of 125 and a standard deviation of 3. It has plugged all 10,000 of those values into your model equation using the parameter estimates that were previously determined, and it calculates the response for all of those variables. This distribution is showing you what that's going to look like.

It's a little collapsed in this view only because the scale of this chart is a little narrow, so you could expand that if you wanted to. You can also just tell JMP to give you the values in a table down here there's the simulate to table. I'm going to open that and click the Make Table button. Here are the 10,000 different values of applied torque that were randomly selected from that normal distribution. They were plugged into the equation and then used to calculate the force.

Again JMP has done a distribution analysis for you saved over here. So I'm going to open that up. And this is showing you the distribution of the measured forces. It's this distribution over here just in a little bit more resolution so that you can see what's going on in your system. This distribution of my measured forces is showing me the effect of having variation in my input in my applied torque according to an assumed distribution, but it allows you to see what that variation is going to do to your response.

Now from here, you could do all kinds of different things that you might be interested in process capability analysis, a tolerance interval analysis and so on. All of that can be done at this point to help give you a sense of how your response will behave given this amount of variation in your inputs. What you choose here is up to you. Hopefully you can use either some historical data. You've looked at this torque wrench for that you use on this system, and you've seen that. Yes, the torque does vary according to this distribution, and so you can choose the appropriate distribution with the appropriate parameters to use.

You might have domain expertise where you can think about what's going on in your system and say, well, this should follow a uniform distribution between these values, or of course you can guess because this is a model. If you're not sure, start off assuming a normal distribution of some sort just so you can see what happens. Then you can do a lot of what if analyzes that you might be interested in. That's the Monte Carlo simulation. Very simple to do in JMP. Along with bootstrapping, both of these techniques allow you to evaluate variation in your system, based on small samples for bootstrapping or based on known variation in your inputs to evaluate what happens to your outputs.

We've talked about a couple of different things here. Bootstrapping again is a technique that can be used to understand what you might expect for variation in your system, even based on small samples. And then Monte Carlo analysis allows you to inject variation into the inputs of a model to understand what that might do to the outputs for that model as well. Very common techniques. Very popular, very powerful. Once you have a model, very useful tools for you to use. That's what I have to talk about today. Thank you very much.