The Power of Simulation In Design of Experiments (2026-US-30MP-2885)

Welcome, everyone. Today, Jacob and I would like to talk to you about how simulation is a powerful tool in designing experiments. First, however, we'd like to back up a bit, talk about simulation in general, and then move forward through various ways that JMP uses and supports simulation before we get up to design of experiments.

First of all, what is simulation? It's simply a way of mimicking a real-world response to a set of specified conditions. This is based on a model that predicts the outcome under given conditions. A deterministic model will always respond the same way to the same set of conditions. On the other hand, a stochastic model responds differently to the same set of conditions every time you run it. That's because it has included in it a source of random errors and noise.

Why would you simulate? There are a number of reasons, and the ones I show here are not an exhaustive list. First of all, simulation allows you to preview your analysis before you collect any real data and identify potential problems. Similarly, simulation could help you identify a problem with the design of the experiment or the model that it's based on. This simulation could help you confirm your choice of factor ranges. Are they wide enough? Are they too wide? Simulation can be used to provide empirical estimates of the power for your inferential test. This would be for the analysis of your data. Similarly, it can also provide empirical p-values and estimates of confidence intervals.

Now, in some cases, other types of models used in computer experiments are very compute-intensive. Even today, they take a long time. A simulation using a surrogate model can give you much better performance while achieving high degree of accuracy. Lastly, simulation is a powerful tool when teaching statistical concepts and principles.

All right, let's start walking through some of the ways that simulation is used in JMP. First of all, some of the inferential tests that we want use simulated p-values. The example here shows the Distribution platform, and we wanted to know about the goodness of fit of a normal distribution. We're using the Anderson-Darling Goodness-of-Fit test, and the p-value is based on simulation. Another place where p-values are simulated is in the two-level screening platform. Both the individual and the simultaneous p-values are derived from a large simulation.

Now, that's built in to JMP. What about your own simulations? It could be as simple as creating a new data column in your table and defining a column formula. Here we see the Formula Editor with a polynomial expression that's going to predict a value. But what makes it a useful formula for simulation is the last term. That's the stochastic term. In this case, every evaluation of this expression will return a new pseudo-random number from a normal distribution. We have a wide choice of functions that return these pseudo-random numbers. This screenshot shows the Scripting Index and the group of all the functions. There's quite a few. To use them, it's simply using the arguments that define the distribution parameters such as the mean or standard deviation.

All right, now there's two other built-in features where JMP will use simulation. The first one is called Simulate. This is used after you've analyzed your data, and you're interested in getting an idea of how often you would get a result like the one you obtained with your data. I'm curious about the p-value column. I right-click on that and select Simulate. Then I designate two things. First, the original data column, in this case Y, and the one to substitute in each run of the simulation. I'm using Y simulated for that. This is a great way to assess empirical power.

Next, in a similar fashion, I can use the Bootstrap feature as a way of resampling with replacement to generate both independent samples for nonparametric confidence interval estimation. Instead of using a formula, we're actually using the original data, but resampling with replacement. This is a good way to estimate confidence intervals. One of the tools that we use in our analysis once we've selected a model is the Prediction Profiler. For example, here we've optimized the response, and now we'd like to see or predict future behavior under these conditions. We can use the Simulator feature of Prediction Profiler for that purpose, in which we can add random noise to both the response and the factors. We have a lot of flexibility in defining how the response and the factors will vary or not in each future run.

Now, let's get closer to designing experiments. One thing you might use for that is one of the Sample Size Explorers. These provide nonparametric and parametric capabilities to generate a random example of such a test. Similarly, the Constant Stress ALT, the Measurement System Analysis, and other DOE platforms provide simulation as well.

Now we're going to get into DOE. Platforms such as Custom Design provide you with a simulation panel when you click Make Table. Here you enter information such as the coefficients you'd like to use for each term, and also you can select both the error distribution and the sigma value. If you close this window and later want to return to simulation, you can open it up again using the DOE Simulate Table script. Well, other DOE platforms provide simulation, but the panel is tailored to the kind of data and model you'll be using. For example, the Choice Design and MaxDiff Design platforms provide a specialized panel where you enter the marginal utility for each term in your model. Likewise, the Covering Array platform, which provides designs to help you identify conditions that would generate a fault or failure, you enter a failure count or a root cause as a factor interaction.

Now, let's talk about Easy DOE platform where we provide a specialized panel right in the workflow. Here we see the Data Entry tab. You would use this tab to enter your real data in the columns on the left. But here we're going to preview the results using simulation by opening up the simulation panel. This is version 18, where you could enter coefficients manually. You can set them all to zero and start over, or just click Random Coefficients, in which case, up to half of these coefficients would be set to non-zero value.

In 19, we enhanced it considerably. You have all the original capabilities, but now, instead of resetting to zero, you can initialize all of them to any value you like. Also, you can generate random coefficients with more control. It uses default values, but you can take control and fine tune things. I'd like to talk about that a little bit. Jacob is going to demonstrate these features in just a moment. Again, you can manually enter any coefficient value. You can initialize all coefficients to a specific value. You can automatically generate random coefficients based on default choices or opening up the advanced features. You can tailor these by setting a random seed or enforcing effect sparsity, effect hierarchy, or model heredity. I'm going to go into the last four in a little more detail.

Setting the random seed is helpful. When we use simulation, a lot of times we want to be able to reproduce those results. There are pseudo-random numbers, and the seed determines where we start in the stream. If you're publishing, or you want to go back and look at things maybe with your team later, you might want to have the same simulation results, and this allows you to do that. When you clear it, every time you click Random Coefficients, you'll get a new sample. That might be what you want when you're exploring. But when this option is set, you enter a seed value. Now, whenever you click that Random Coefficients button, you will produce the same random coefficients. It also will reset the random seed for the simulation of your response, and so you'll get the same response values. This is very helpful when reproducibility is key.

Next, let's talk about effect sparsity. There are three principles, this is the first one, that are really important in the design and analysis of screening designs. Now, you can use these features depending on the type of experiment you're running. With effect sparsity, we are trying to explore many factors, but we expect most of them have null effects. We're trying to separate, we say, the vital few from the trivial many. This option allows you to adjust the number of active effects. When you enforce it, it'll be a small number. You can also change the proportion of active effects in each order, the proportion of main or two-factor interactions or quadratic effects.

The next principle is effect hierarchy. In screening, we expect most of the effect to be produced by the main effects, and less so for the two-factor interactions and quadratic effects, and even less for higher-order terms. Here, we can adjust the size of the active effects to suit the screening assumptions. You can change the size of two-factor interactions and quadratic effects. That's relative to the main effects.

The final principle is model heredity. This addresses when lower-order terms are included in the model if two-factor interactions are included. This is based on observations that the vast majority of times that a two-factor interaction is important, one or both of the main effects are also important. Here we enable you to simulate with that in effect. This is important because now the model that you select on the next tab, the Analyze step in Easy DOE, enforces strong heredity. But model heredity isn't always the case. You can turn this off if you want to generate a response where it is not true.

All right, just a bit more. I want to talk about how these coefficients are generated, and it's very similar for the main effects, two-factor interactions, and quadratic terms. First we have to figure out how many main effects are active. That is depending on whether effect sparsity is enforced. If it is, then we multiply the number of main effect terms in the model by the sparsity proportion. If it's not enforced, we use a default value of three out of four. Then we randomly select which main effects are active, and finally, generate the coefficients for those using this formula at the bottom. It's very similar for two-factor interactions. We first have to determine how many will be active. We take the number of two-factor interaction terms in our model and multiply it by either the sparsity proportion, if sparsity is enforced, or the default value of 75% if it's not. Then we randomly select which of the two-factor interaction terms are active and generate the coefficient. This formula is the same as before, but inside the parentheses, you see the term h for effect hierarchy proportion. If this is enforced, first, we use the proportion you specify. Otherwise, it's simply one and has no effect.

Now, the last case, the quadratics. We determine the number of quadratics. We take the number of quadratic terms, and multiply it by the sparsity proportion when that's enforced. Otherwise, we use this default value of 0.75. We select which of those quadratic terms will be active and then generate the coefficients. Here, h is the hierarchy proportion when it's enforced, otherwise, it's set to one.

Now, just to give you an idea of how these proportions work. Here we're looking at the simulated coefficients for two-factor interactions and quadratics, where we vary the proportion size for the hierarchy. The first distribution is where that proportion is set to one. It's equal to the size of main effects. The second, it's reduced to one-half, and you see how that shrinks the range of these coefficients. Lastly, it's one-quarter, and it shrinks by a factor of two again. All right, now, we have three examples. We're going to look at different ways in which these random coefficients are generated and used, and I'm going to turn it over to Jacob for that.

Thank you, Mark. As Mark said, I'm going to be presenting three case studies showing how you can design an experiment in JMP and then simulate responses for that experiment. Let's start out in Easy DOE with a screening experiment. Suppose we're going to be developing a manufacturing process, and our goal is to maximize the yield of this process. We believe that there are nine factors that could impact the yield, but we don't expect all of them to be important, and we don't really know the relative importance of these effects. Our goal is to conduct this screening experiment to help identify what are the important factors.

Let's set up this design in Easy DOE and simulate a response for the experiment. I'm going to launch Easy DOE. Just to save some time, I went ahead and have all of my factors added, and I have my response added as well. If you've never used Easy DOE before, you can find it as the third item in the DOE menu. But once I have my responses and factors added, I need to move on to the Model tab and specify my assumed model. Since this is a screening design, I'm going to choose the first option, which includes all of the main effects in the model. I could use the Number of Runs box to adjust the number of runs. If I couldn't afford 16 runs, maybe I would lower it a little bit. But let me suppose that I'm okay including 16 runs in my design.

I'll click the Design tab to create the design, and then click to the Data Entry tab. This is the point in the experiment where I would actually run the experiment and record the value of yield for each run. But let's suppose I'm not ready to do that, so I'm going to simulate a response. I'm going to click to open the simulation panel. I'm going to drag this down to hide the factor plots just to give me a little bit more space to view the simulation panel. Since I selected the main effects model at the Model tab, we see that the model for my simulation panel contains the intercept and all of the main effects. Just as in previous versions of JMP, in JMP 19, you are able to click in each of the box for the coefficients and enter any manual coefficients you would like. You can also initialize all of the coefficients to a single value. If I wanted to set all of the coefficients quickly to one, I could do that by typing one and clicking Initialize Coefficients button, or I could click the Random Coefficients button to let JMP generate some random coefficients for me.

I'll set everything back to zero just so we start fresh, and let me open up the Advanced Options and think about how can we use these new methods in JMP 19 to simulate a response. You'll notice that since I only have main effects in my model, I'm not able to enforce effect hierarchy or enforce model heredity, and that's because those options only apply if you have interactions or quadratics. I really just need to think about, do I want to enforce effect sparsity or not, and would I like to set a random seed.

Since this is a screening experiment, I don't think all nine of my factors are actually active. I would like to enforce effect sparsity when simulating a response, so I'll go ahead and enable that. You also saw that when I hover over any of these options, you'll get a tooltip that pops up with an explanation of what that principle means if you need a reminder about that. I'll enforce effect sparsity, and I decide on the proportion of main effects I'm okay with being active, or the proportion of coefficients I'm okay being non-zero. The default is 0.5. Let's say I'm okay with that. I'm okay with up to half of the main effects being active. I will set a random seed just for reproducibility, and I'll click the Random Coefficients button. I always recommend looking at the coefficients and making sure you're okay with the random draw you got from the Random Coefficients button.

Here we would have a really large main effect for temperature, and really large relative to the error value of sigma. You would have a pretty large stir rate coefficient as well. Then you'll note that the blend time coefficient is pretty small relative to sigma. If I was expecting to see a larger effect, I can always increase this. Maybe I'll change it to 1.5 instead of 0.2. If you're happy with the coefficients that you're given, you can click the Simulate Response button and see that the yield column is now filled with my simulated responses. I can open the header graph and take a look at the distribution, and I see the distribution ranges from roughly -12 to +13. Maybe I want to change that to make it look a little bit more realistic with what I expect yield to look like. Something you could do is go back over to the intercept and change that. Maybe I'll change it to 75. I'll click Simulate Response again. Now my yield column has updated with my updated simulated response values. Maybe this looks a little bit more like what I would expect yield to look like.

If I'm okay with the simulated response column that I have, I could then move on to the Analyze tab, and I analyze the results of this experiment for my simulated response. In Easy DOE, I could click a term to remove it from the model, or I could just click the Best Model button and let Easy DOE run its stepwise procedure for me. That's how you could set up a screening experiment in Easy DOE and then use the new advanced options to simulate responses for a screening experiment.

Now, let's stay in Easy DOE, but let's create a different type of experiment. Let's now suppose that we've created a screening design just like we did in the previous demo to identify important factors that affect yield. We think that there's a significant relationship between blend time, pH, temperature, and yield. Now we want to identify what are the important factor settings that will maximize the yield. Let's also suppose that we believe the effect of at least one of these factors could be nonlinear. Now let's set that up in Easy DOE. I have my three factors and I have my response yield. I'll move to the Model tab and specify my assumed model. In this case, since I believe that the effect of at least one of the factors on yield will be nonlinear, I'm going to select this fourth option which will include all main effects, two-factor interactions, and quadratics. If I expanded the hint, I would see that this would be the appropriate model to use when I believe the effect of at least one factor would be nonlinear.

Let's say I'm okay with 16 runs. I'll click to the Design tab. We'll go ahead and stop it early and then click to the Data Entry tab. I'll once again open up the simulation panel and simulate a response. One thing to note is that since my model now includes two-factor interactions and quadratics, as we see here, I open up the advanced options and I now will have the ability to enforce effect hierarchy and enforce model heredity. Heredity will be enabled by default. If you choose to not enforce heredity, you would need to deselect that option.

Now, I need to think about, do I want to enforce effect sparsity, effect hierarchy, and heredity? I only have three factors. I believe from my previous screening design that all of them are going to be influencing the yield. I probably would not enforce effect sparsity in this case. If you did want to enforce effect sparsity, you could always set that proportion of active main effects to one, and then that would give you a little bit more control over how many two-factor interactions and quadratics were in the model. But I'm just going to turn that off for the purposes of this demo.

But I would like to enforce effect hierarchy, because although I am going to be having some second-order effects in my model, I believe that they aren't going to be as large as the main effects. I think that most of the variation in yield is going to be coming from the main effects. I would like to turn this option on. I am going to enforce heredity because any second-order effects that are in the model, I want the main effects to be included as well.

I'll set a random seed, and I'll click the Random Coefficients button, and let's take a look at the coefficients it gave me. I'm going to have the pH, temperature main effects. I didn't get blend time, but I can always add that in if I would like, and so I'll add that here. The reason I chose this random seed is I wanted to point out that even with Enforce Effect Hierarchy turned on, it's still possible to get a smaller main effect than a second-order effect because it is, after all, still a random draw from the formulas that Mark showed in his slides. I always recommend checking the coefficients and making adjustments as you see fit. I wanted the main effect to be larger than the two-factor interaction, so I'll just raise this temperature value. Maybe I'll set it to 1.75.

I could review all of the other coefficients. If I'm okay with everything here, maybe I change the intercept again, just like I did in the first demo. I'll click Simulate Response. I'll review the distribution of my simulated response, and if this is in line with what I expect to see, I can move on to the Analyze tab and analyze the results of the experiment to get a preview of the analysis for my simulated response column. That's how you could set up an optimization experiment in Easy DOE and then simulate a response for that optimization experiment.

Now, let's leave Easy DOE and let's go over to Definitive Screening Design. Let's say we want to create a Definitive Screening Design for six continuous factors. I have that set up here. I have my six factors, and my goal is, again, to maximize the yield of a process. I'm going to click the Make Table button. In my script, I had Simulate Responses enabled in the red triangle menu. Now when I click Make Table, we see I have this Simulate Responses window here. That was essentially what happened is it automatically ran the DOE Simulate Table script. If I look at my data table, we see that I have the response yield filled in with my default simulated response column, which essentially takes all of the coefficients and initializes them to one, clicks the Apply button. That's my default simulation model.

In definitive screening design, I have the ability to adjust the model that I'm using in Simulate Responses. To do that, I'll open up the Factors outline. My model already has all of the main effects. I can add any two-factor interactions that I like, I could add any power terms I like, or I could click the RSM button to use a response surface model to simulate the response. Now you'll see that my model includes all of the main effects, two-factor interactions, and quadratics. Just like in Easy DOE, I have the ability to initialize all of the coefficients to a single value, and I could use the Random Coefficients button to generate some random coefficients.

Let's open the Advanced Options, and we see that just like Easy DOE, I can choose to enforce effect sparsity, hierarchy, or model heredity, and heredity will be enforced by default. Now, I need to think about, am I going to enforce effect sparsity and hierarchy? This is a screening design. I probably don't think that all of my factors have a significant effect on yield. What I'm going to do is enforce effect sparsity. Let's say I'm okay with up to half of the main effects being active, up to 25% of the interactions being active, and 35% of the quadratics being active. I'm not going to enforce effect hierarchy here because let's suppose I think that I'm going to have at least one or two strong second-order effects. The reason I chose a definitive screening design is I know that definitive screening designs are good at picking out relatively large second-order effects. In order to get pretty strong second-order effects in my simulated response, I'm going to leave this option disabled.

But I will set a random seed. I'll use my random seed of 123, and I'll click the Random Coefficients button, and let's take a look at the coefficients that I have. I have a pretty large coefficient for propanol relative to the value of sigma. I have a really large butanol main effect as well. I have a pretty strong interaction between butanol and propanol, and an active quadratic effect for butanol, and then a not large effect for the propanol quadratic term as well. If I'm okay with these coefficients, I would click the Apply button, and we see that updated the yield simulated column. I would copy that over to the yield column and then run the Fit Definitive Screening table script to analyze the results of this experiment for that simulated response.

For this simulated response for yield, we could see that we found a strongly significant relationship between propanol and butanol and yield. We also found a less significant, but still a significant relationship between pH and yield. It's not surprising that we found that strongly significant relationship between propanol, butanol, and yield because we had really large main effects for the coefficient relative to the value of sigma. If we look at the pH coefficient, we see that the coefficient was zero, so that's not actually used in our model to simulate the response. But Fit Definitive Screening did find a significant relationship. We'll take another look at that in just a minute using an empirical power analysis. Take a look at how often would we expect that to happen.

Looking at the even order effect estimates, we see we were able to pick up on that really large propanol/butanol interaction, and we were also able to pick up on the butanol quadratic term as well. We were not able to pick up the propanol quadratic term that we know was in our model we used for yield simulated. But again, it's not surprising we didn't pick it up because the coefficient value was so small.

Now, what I'm going to do is I'm going to click the Run Model button to go over to Fit Least Squares. Then let's do an empirical power study to look at what power would we have to detect effects of this size. I'm going to go down to the Parameter Estimates table. I'm going to right-click the p-value column and select Simulate. Now I need to pick my column to switch in and my column to switch out. I only have one column to switch in because I only have one formula column. Essentially what this is going to do is it's going to replace yield in the analysis I did with a new column of yield simulated values. To make this run quick, I'm just going to do 100, and then I'll set the random seed for reproducibility. I'll click Okay.

This is what the Simulate feature produced for me. I have a data table of p-values for each run of the simulation. The p-values in the first row are my p-values from my actual analysis. The next 100 rows would be the p-values from the 100 simulations that I did. I could run the Power Analysis table script to take a look at the empirical power. But I'm going to close this and open up a data table from my journal where I used a larger sample size. Let me bring that up. Back to my journal.

Again, I did the exact same settings. I used the same random seed. The only difference is I used 5,000 instead of 100, just so we could have a large sample size to take a look at the power analysis script. I'm going to start off by looking at the propanol main effect, looking at the simulated power table. We can see that for every value of alpha, the rejection count is 5,000 and the rejection rate is one. What that means is in all of my 5,000 simulations, the p-value for the propanol effect was less than alpha. There was always a statistically significant relationship between propanol and yield. It's not surprising that we see this because the coefficient that we used to simulate the response was quite large, so we're able to pick up on that effect very well.

Same thing with butanol, the size of the coefficient was even larger, so it's not a surprise we see a perfect rejection count and a perfect rejection rate. What is more interesting is when we start to look at pH. If you recall, the pH coefficient we used was actually zero, but Fit Definitive Screening found a significant relationship between pH and yield. But if we look down at the simulated power table, we can see that for alpha as 0.05, we rejected that in 217 out of the 5,000 simulations. That's a rejection rate of about 0.04. The p-value was less than 0.05 in only about 4% of those simulations. Looking at the distribution of pH, we can see that this looks like a uniform distribution between zero and one. If you'll recall from hypothesis testing, assuming that the null hypothesis is true of there being no relationship between pH and yield, we would expect to see a uniform distribution of p-values between zero and one. Fit Definitive Screening did find an active effect of pH, but we really just got unlucky with our simulated response for that particular random seed. We would not expect to see that happen very often.

Then, looking at the second-order effects in our model, we get a perfect rejection count for the propanol and butanol interaction. It's also not surprising. That coefficient was quite large. If we look at the butanol quadratic term, looking at alpha as 0.05, we reject the null in 2,392 or about 48% of the simulations. A lower rejection rate, but to be expected because the coefficient was smaller. That's how you can create a design in Definitive Screening Design, use Fit Definitive Screening to analyze the results for a simulated response column, and do an empirical power study using the Simulate feature in JMP.

Now, I want to end just with a quick demo of these new options in Custom Design. Let's suppose that we're conducting a robustness experiment where we have three continuous factors, one categorical factor, and I'm going to set this up in Custom Design. We have our three continuous factors and our one three-level categorical factor. For the model, I have it set up to include all of the main effects and three interactions. Let's suppose I would like to be able to estimate all of the interactions if I had the run size to do so, but I can only afford 12 runs. What I've done is I've picked out the three interactions I estimate that I think are most likely to have a significant effect on the response.

I'm going to click the Make Design button, enable Simulate Responses, and click Make Table. We can see that the Simulate Responses window has the model that I used from Custom Design. I'll go ahead and open that up so we can see. This is not new in JMP 19. The model for Simulate Responses should be what model you used in Custom Design. This was the case in past versions of JMP as well. But I wanted to point out that that ability was maintained in the latest version of JMP.

What I really wanted to show was that the Advanced Options settings should always be in sync with the model you used. It doesn't matter that I didn't use a full model with all two-factor interactions. As long as you have at least one two-factor interaction, the Advanced Options will recognize that you have two-factor interactions in your model. For Enforce Effect Sparsity, I would be able to set the proportion of two-factor interactions that I wanted. I would be able to set the size of the two-factor interactions relative to the main effects and enforce effect hierarchy. I'm not able to do anything with the quadratics because it knows that I don't have any quadratics in my model. If I was to back up Custom Design and create a model with at least one quadratic effect, then these options would be enabled. The Advanced Options should always be in sync with the current state of the model that you have. The model that you have is directly chosen by what you did when you created the experiment.

I'm just going to close by simulating a response for this experiment. I don't really have many factors. I only have four, so I think they all have an impact on the response. I probably wouldn't enforce effect sparsity here. I am going to enforce effect hierarchy because I believe that the main effects should be stronger than the two-factor interactions that I've chosen. Let's say I'm okay with 0.5, and I would like to enforce heredity. I'll set a random seed, and I will click the Random Coefficients button to generate random coefficients. Maybe I want this to be a little larger, so I'll change that. I'll click the Apply button to actually simulate the response. That would populate the response column in the data table, which I'll bring back up here. Then I would just need to copy the Peel Strength Simulated column over to Peel Strength. These are my updated simulate response values. I could run the model script or the Generalized Regression script to analyze the results of my experiment for my simulated response.

The take-home is you can use the new advanced options in JMP 19 in order to tailor your simulated responses to the experiment that you created to get a preview of the analysis before you're actually ready to run the experiment. That's all we have to share. I want to thank you for your time, and I hope you try out these new Simulate Responses options in JMP 19. Thank you very much.