The Power of Simulation In Design of Experiments (2026-US-30MP-2885)

Hi, I'm Laura Higgins. I'm a systems engineer on the Global Technical Enablement team. Today, I want to talk about how design of experiments can power your innovations. You're here because you're interested in innovation. You're trying to make a better product or a better process, or you want to improve on something that already exists. Maybe you're trying to make something brand new. How can we take bigger steps in the process of innovation? How can we get to the next stage with less time, less money, and less effort? Ultimately, using analytics to create robust and high-quality products and processes is how we're going to improve innovation.

If innovation was easy, everyone would be doing it. But there's complexity that makes it hard. The complexity is in these underlying factors. The factors are the things that we're trying to change or adjust to meet our goal. We have to try new things, and we do this with experiments. Experimentation is essential for innovation. We're going to try new combinations of these underlying factor settings or try new factors that we've never tried before—new ingredients. Ultimately, experimentation is how we learn.

How do we reconcile these two things? On one hand, we have complexity making things hard, and we have to experiment for innovation on the other side. Analytics bring specificity. Let's be specific about what we mean for each of these things. For complexity, our factors—so again, the underlying components of what we're adjusting—these themselves can be complex. They can have an unexpected influence on one another, and we can just have a large number of things that we may or may not know is important, but we think we might need to look at. This unexpected influence is what's going to take us outside of the realm of our expert knowledge, and we have to go into this territory to do something new and better.

On the other side of things, experimentation is how we learn as a group, an organization. We have to do stuff a whole bunch of times. We sequentially experiment; we build on what we've learned, but we need a structured approach. Structure is imperative. What we really want to be doing are smart, strategic experiments.

Design of experiments is a framework that is big enough to capture both complexity and experimentation into one. There's a lot of room in this framework, and this is the message that I want to leave you with. There is no level of complexity that can't fit into this framework, and it's easy to get started. That's what I'm going to show you today. Doing a design experiment will bring progress, efficiency, and effectiveness to speed up your innovations. We do not start with a blank slate. We are going to leverage our expert knowledge, but we must get beyond doing these experiments that we only adjust a single factor at a time. These are traditionally called one-factor-at-a-time experiments. Again, we're trying to find the best combinations of things that explain causation. Let me show you what I mean by a one-factor-at-a-time experiment versus a design experiment.

Over here, we have our one-factor-at-a-time. On the x-axis, we have temperature, and on the y-axis, we have pH. In a one-factor-at-a-time experiment, here's what we do: we pick one pH, and we're going to try different settings of temperature. We find the temperature that's the best, and then we change pH at that temperature. In this illustration, color is representative of yield, and we find the best yield by this one-factor-at-a-time approach. In contrast, with a design experiment—so the same axes—what we're going to do is cover this entire design space, and we're looking more broadly at combinations than we do with just one-factor-at-a-time. In this example, we actually get a higher yield, and it's with combinations we would have never tried in our one-factor-at-a-time. It pushes the range, and we really start to understand the complexity. When we understand the complexity, we can understand causation.

This is what the two results look like: one is more of just a hill, and the other is a slope. You can imagine if you're standing out in nature, and you're on a hilltop. You think you've reached the peak, but it's a cloudy day. Then the clouds part, and you turn around, and you realize there's a much higher peak behind you—but it's far away, and you would have never found it just by walking around. You need a guide and a path. There are higher places that we can find, higher outcomes—so in this case, better yields.

What we're trying to do is understand the relationship between these factors and the responses. Here, I have a list of factors, and they can be things like machine or operator, temperature, pressure, humidity. Then we have typical outputs: things like cost, yield, impurity. We're going to have multiple responses that we want. Instead of this black box, what we're trying to do to this process is describe a statistical model that characterizes this process. This is how we're going to start to unravel these unexpected influences of factors on one another.

There's two different kinds of experiments that people talk about. One is screening, and this is deciding what's important. This lets us take a very large number of factors and look at the rank order of which ones are important. When we have a smaller number, we can actually start to understand the complexity and define this relationship within and between factors, and factors and responses. This sounds like a lot, and I told you it was easy to get started. Here's what we're going to do: screening is like the chocolate—it's like the good stuff—and characterization is like the peanut butter; it's got some complexity in it.

What we have in JMP is a platform called Easy DOE. It magically combines screening and characterization, and it helps us complete our objectives. We don't have to get bogged down with deciding, "When do I do one, and when do I do another?" It's going to help us do this automatically. I'm going to show you this.

Before we get there, we do need to briefly talk about steps in a DOE. As you stare at the slide, you're like, "Oh, my! That's a lot. What are we going to do here?" But this is exactly the set of steps that the Easy DOE is going to take us through—each one of these steps—it's going to give us a guide through each one. I'm not going to talk about the details here because I'm going to show you in the Easy DOE platform. This is a very classic picture. Many of you may have seen this, and this is exactly what we're going to do in Easy DOE.

I'm going to leave PowerPoint behind in a few minutes, but I also want to talk about augmentation. I think doing a designed experiment is great, but we cannot underestimate the importance of augmenting. When you do a one-factor-at-a-time experiment, it's hard to move on and do the next experiment; you're just taking a few values that you found and moving forward. When you have a designed experiment, you can utilize all of that knowledge that you've learned, and you do not start over from scratch. What you're doing is you're taking that underlying statistical model and adding more information to it. Augmentation is a very important part of doing designed experiments, and you get all of this extra knowledge without starting over. I think that should be one of the more compelling reasons to try using designed experiments.

Let's do some Easy DOE in JMP. This is JMP. This is a JMP journal. If you're not familiar with it, it's just a way to take notes, and I have data tables linked to it, and other nice things like that. Where to begin? One of the things that I do want to talk about is a little bit of understanding what relationships are. Just for a little background: if we have two factors on the x and the y, we can have a positive relationship or a negative relationship. We can also have something that's well described by a straight line or something that has to have some curvature in it. There's a little demo here that comes with JMP—it's built in—and it lets you add data points, and you click, so you can start to understand how this correlation works.

The other thing we need to understand is a regression. A regression is just… The correlation's the relationship between variables; the regression is actually this process of drawing a line. You can see here, again, I can change the relationship. This should help you start to understand what you're looking at when you interpret your own results.

There's one more thing I want to share with you before we get into our example. When we have a model—so a model is a statistical relationship; we all know y equals mx plus b—if we only have that, it's just called main effects. Here we have X1 and then X2 with our response variable on the y. What we see here is each X has an influence on the response, but the two, X1 and X2 do not influence each other. Maybe they're not straight lines, and this is what I was referring to before by the difference between a linear and a non-linear or curvilinear relationship. Quadratic is the mathematical way we describe curved lines. Again, X1 and X2 are not influencing each other, but they have this more complexity in the relationship to the response variable.

X1 and X2 might be changing each other, and this is the underlying complexity that starts to make your innovations hard and where designed experiments can help you understand this. You can see as we change one, the relationship… If we change X1, the relationship of the X2 and the response changes. You may have seen this without knowing it, and it just can be so frustrating if you don't have a way to actually define it. Then we can do both: we can have curved lines and our factors influence one another. These are all the levels of complexity that we need to think about when we go to make a designed experiment. But we're going to make that easy; we're going to do that with Easy DOE.

Where we start is under DOE, and we're going to do Easy DOE. You can see, as I showed you in the other slide, here are the different steps that you need to do. Actually, let me just open one that's already done. Here are the steps that I showed you in the slides: Define, Model, Design, et cetera. These are all right here for you. Also notice that there's this Guided and Flexible Mode. If you already know something about design experiments, you might want to choose the flexible mode. It's just going to give you additional options for design choices and types of analysis. But if you're new—which is the audience that I'm talking to—just put it on guided mode, and then I'm going to… Here's our blank one; let's look at our responses.

The response is what you want to measure. The first one we're going to do is just a screening design. We just want to understand the rank order of our factors. You might have a goal in mind, you might not have a goal in mind. There are hints here if you want to try to understand more, but this is really, it saved you from going… The documentation is really good; it takes you through examples, but it's all right here for you to reference as you make these decisions. I want to add a response and match a target. I'm going to delete that one. What we're doing is making coffee. I like coffee. We're going to look at the coffee strength. What we're looking for is somewhere between a value of 1.2 and 1.4.

You could also want to maximize or minimize a target. For example, if we had cost, we might want to minimize cost. Those are our responses. We also need to add factors, and these are the things that we're going to change in our experiment. Watching me type is a little boring, but let me just address what these different factors are. I'm going to just show you. You can have a table with factors. What I'm going to do here is I'm going to load these so we can look at them; we'll look at each one. Here I have the grind. Coffee can be coarsely ground, or, in this case, medium ground. This is a categorical response. If you're going to add a categorical response, it's something that comes in discrete units: red/blue, different vendors—in this case, the grind.

My temperature is a continuous variable. What you put in is you put in the lowest level and the highest level. Continuous variables are anything that might have a knob that moves continuously. Temperature, we can have any amount of variation between temperatures. Our next factor is also continuous, and this is time. Time is always continuous. We can have this go between 3 and 4. Charge has to do with the brew strength, so how long the machine is actually… The setting on the machine that changes the strength of the brewing.

Then we have two different coffee stations, because maybe there's a difference between stations that I am unaware about. This brings up the important idea of noise. One of the things that design experiments does is randomize so that you don't have to worry about everything. Because it could be that maybe the heating element on one of my stations isn't as good as the other one, but randomizing between them will help obscure that fact, and help take the noise that we don't care about, and we're not trying to define, and help us get a better handle on that.

The other one that's up here is a discrete numeric factor. This is for when you have something like an oven. Let's say you have just a couple of settings on the oven. While temperature could be a continuous variable, you're not actually going to run it that way. You could say, "Why wouldn't I just run that as categorical?" Because you'll get more information out of discrete numeric. If my two values for discrete numeric were 10 and 100, it will recognize that difference; or 10, 20, and 100, the difference between 10 and 20 is only 10, but 10 and 100 is 90, and it will take that into consideration. If I just cram those into a categorical factor, then it's just going to treat it as A, B, and C, and I'm going to lose that additional information that I could have gained.

Okay, let's move on. When I click my navigation controls, it is going to move me to the next tab here: Model. I think this is where people start to worry, and there's a nice little explanation here of what some of these things mean. It even says right now we're just screening: we just want to know which of these items are the most important. This is why I showed the illustration of interactions and quadratics, because we can put these in and test for them. One of the things you can see is that, to get more information about what's going on—whether our factors are linear or not, and if they interact with each other—it costs us more in the number of runs that we do. This is why if you have a large number of factors, you're better off doing a screening design and then subsequently doing a characterization design.

But all I have to decide in Easy DOE is which of those questions I'm trying to answer. In this case, I just want to know what's important; we're going to stick with that. It's made my design for me—here's my design. Then I can export the data; I'm going to make a JMP data table. Then I can go off. You need to run each run in order just like this set. The first time I do my experiment, I'm going to do it at station 2. I'm going to use a coarse grind, a temperature of 195, for a time of 3, and a charge of 2.4. The second time I do it, I'm going to also use station 2, I'm going to use a medium grind, then you can see the other variables here. The run order is important, and this is part of the randomization. If you run off, and you just sort this, then you've not respected what we're doing with the design of experiment.

It's a little beyond what we're going to talk about today, but if we go back to our model and we… No, I'm sorry, if we go back to our factors, and our flexible mode, we can add additional information about things that are challenging to change—big vats of acid if you're dipping something in it. This is going to change the underlying statistical model that we're building, but it is here, and you can do it, and you should use it. Industrial experiments frequently have things that are challenging to change, and we can't use all the randomization.

Let's continue on. This is where I've already done this experiment, so I am going to… Here's my data, and you can load the responses—so you just select a column from a data table, and that's the best way to get your data back in. On the previous one, you export the data, you export the data table, you fill it in, and then you can reload back into Easy DOE what your answers are. Then what's this little graph down here? This is showing us a preview of our results. Here is my coffee strength—my outcome variable on the y—then how each data point here is one of my experiments. We can get a preview of what the results look like for each factor. Let me make this a little bit taller. There we go. It looks like grind might be important. It's hard to know about some of these other ones. We're not just going to eyeball it, but it does show you the spread of the data for each of your data points.

As I click Next here on my navigation control, I get to the Analyze, and this is really cool. There are two different things here. We can look at everything in the model: the solid line means it's significant, and the dotted line means it's not significant. Just to be clear, in design experiments, "not significant" doesn't mean it's not important. It means you didn't detect something. Maybe if you did twice as many runs, you might have detected it. In this, we have no evidence that it's important. The full model is everything: if we want to see that, we can look at those results.

What we're looking for here is that our actual estimate is bounded away from zero. We do by looking at the confidence intervals, and we can see that in our grind, it's plotting this for us. In contrast for other ones, it's saying it's not significant, and you can see it crosses this line at zero. If I click at Best Model, it says, "The best model to explain our DOE includes grind," and maybe temperature. Temperature's on the border, so we might want to just keep that in for now. But time, charge, and station are not important, so we can move on and think about what else we might want to do next.

There's a prediction profiler, and prediction profilers are the way that we visualize equations in JMP. Again, here is our y-axis, and the blue is confidence intervals. Then each square is our outcome variable, and this is interactive. When we're here, we can not just click around in it, but wherever there's a red triangle in JMP, we have a lot of options. We can do optimizations here, and that can be a really important thing if you want to go back and actually try to verify what you did: go back and make one more pot of coffee, and see if it's the best. There is also a nice report here that you can print out, and it has everything on it. All right, so that is one example of a DOE; that's a screening design.

Let's go ahead and do a characterization design. In this one, our factors are going to be pH, temperature, and vendor. Our responses are going to be whether it's the target, and then we also want to keep track of impurity. You can see here, we're going to maximize or minimize. It's boring to watch me type, so I'll just load these into Easy DOE. I'm going to use this little red triangle right here: load my responses and load my factors. We can see that it's just like if I chose these using the plus button: target and impurity, I want to maximize my target and minimize my impurity.

My factors—I have three factors, and they are pH, which is continuous; temperature, which is continuous; and vendor, which is categorical. I'm trying to characterize the complexity of the relationship here. I don't know if I have straight lines, or if I have curved lines, and I don't know if these two things interact. That's what I want to know. See, this is the cool thing: when you have fewer number of factors, I could do any of these three kinds of designs to estimate these models for the same amount of runs. I'm going to pick the one that's going to give me the most information, looking at interactions and quadratics—it's also called response surface design.

Again, it's going to go off, and it's making the model for me. There are my 18 runs. I'm going to export my data. Here's my data table. Hopefully the, the magic of my simulation will work. I get in my time machine, collect my data. All right, and there's the data. The time machine is not included with Easy DOE. We're going to click Next, and now I'm going to load my responses. Again, if we look at these factor plots, we can start to get an idea of what our results look like. Here I have two outcomes: target and impurity. It looks like pH—there's a relationship maybe with pH. The other ones, it's a little hard to tell. Impurity might have something to do with my vendor. Let's actually run the model.

We have a lot of power here because our error bars are small. We can see a lot of solid green, and a couple of little dotted ones. If I click the best model, it's going to say, "I don't think these things are significant." One of the things here—the way this is written, this, it's pH*pH. This would be to see if that line is curved, and it says it's not. Likewise, temperature: probably not curved.

But we definitely have some interactions, and interactions are written like this: pH*temp. There's some more information down here if we want to keep looking. We can look at each response, because maybe we only care more about… This is for target, and then for impurity, we would find something that's maybe a little bit different, but we want to be able to look at these in a similar model. We can come over to Predict. Again, in our profiler, on top is target, and impurity is below, and each of these is pH. We can see maybe there's some curvature here with pH and temperature, and we can definitely see these flip around for vendor. Here, I'm going to click my Optimize button so I can find the settings that maximize my target and minimize my impurity. These are the settings I should go back and try to run: this pH, this temperature feature, and this vendor. You do need to verify that you've done a good DOE. Again, my report is here if you want it, and we can just export this report.

That's Easy DOE and JMP. There are other kinds of DOE. There's a lot of stuff in here, and they end up being slightly special kinds of DOE: there are designs for teaching the classics, designs for understanding what consumers want; if you're doing a computer experiment, there are space-filling designs; we have covering arrays if you're designing software; you can do formulations and mixtures—there are a lot of things you can do with mixtures; there are reliability and measurement system analysis designs; and there's advanced blocking techniques for additional complexity.

It's important when we're trying to innovate that we have confidence when we're trying to cover this large space of complexity with our design. We have complex interactions, and design of experiments will help us smartly leverage this space, and it will increase our ability to do innovation. With that, thank you very much.