Powering Innovation with Design of Experiments (2025-US-30MP-2592)

Hi, I'm Laura Higgins. I'm a Systems Engineer on the Global Technical Enablement Team. Today, I want to talk about how design of experiments can power your innovations. You're here because you're interested in innovation. You're trying to make a better product or a better process, or you want to improve on something that already exists. Maybe you're trying to make something brand new.

How can we take bigger steps in the process of innovation? How can we get to the next stage with less time, less money, and less effort? Ultimately, using analytics to create robust and high quality products and processes is how we're going to improve Innovation.

If innovation was easy, everyone would be doing it. But there's complexity that makes it hard. The complexity is in these underlying factors. The factors are the things that we're trying to change or adjust to meet our goal.

We have to try new things, and we do this with experiments. Experimentation is essential for innovation. We're going to try new combinations of these underlying factor settings, or try new things, new factors that we've never tried before, new ingredients, and ultimately experimentation, is how we learn.

How do we reconcile these two things? On one hand, we have complexity making things hard, and we have to experiment for innovation on the other side. Analytics brings specificity. Let's be specific about what we mean for each of these things.

For complexity are factors, so again, the underlying components of what we're adjusting, these themselves can be complex. They can have an unexpected influence on one another. We can just have a large number of things that we may or may not know is important, but we think we might need to look at. This unexpected influence is what's going to take us outside of the realm of our expert knowledge, and we have to go into this territory to do something new and better.

On the other side of things, experimentation is how we learn as a group, an organization. We have to do stuff a whole bunch of times. We sequentially experiment. We build on what we've learned, but we need a structured approach. Structure is imperative, and what we really want to be doing are smart strategic experiments.

Design of experiments is a framework that is big enough to capture both complexity and experimentation into one. There's a lot of room in this framework, and this is the message that I want to leave you with. There is really no level of complexity that can't fit into this framework. It's easy to get started, and that's what I'm going to show you today. Doing a design experiment will bring progress, efficiency, and effectiveness to speed up your innovations.

We do not start with a blank slate. We are going to leverage our expert knowledge, but we must get beyond doing these experiments that we only adjust to single factor at a time. These are traditionally called one-factor-at-a-time experiments. Again, we're trying to find the best combinations of things that explain causation. Let me show you what I mean by one-factor-at-a-time experiment versus a design experiment.

Over here, we have one factor at a time. On the x-axis, we have temperature, and on the y-axis, we have pH. And in a one-factor time experiment, here's what we do. We pick one pH, and we're going to try different settings of temperature. We find the temperature that's the best, and then we change pH at that temperature.

Now, in this illustration, color is representative of yield, and we find the best yield by this one-factor-at-a-time approach. In contrast with a designed experiment, so the same axes what we're going to do is cover this entire design space, and we're looking more broadly at combinations than we do with just one factor at a time. Now in this example we actually get a higher yield, and it's with combinations we would have never tried in our one factor at a time. It really pushes the range, and we really start to understand the complexity, and when we understand the complexity, we can understand causation.

This is what the two results look like. One is more of just a hill, and the other is a slope. You can imagine if you're standing out in nature, and you're on a hilltop, and you think you've reached the peak, but it's a cloudy day, and then the clouds part, and you turn around, and you realize there's a much higher peak behind you, but it's far away, and you would have never found it just by walking around. That you need a guide and a path. There are higher places that we can find higher outcomes. In this case, better yields.

What we're really trying to do is understand the relationship between these factors and the responses. Here, I have a list of factors. They can be things like machine or operator, temperature, pressure, humidity. Then we have typical outputs, things like cost, yield, impurity. We're going to have multiple responses that we want. Instead of this black box, what we're really trying to do to this process is describe a statistical model that characterizes this process. This is how we're going to start to unravel these unexpected influences of factors on one another.

There's really two different kinds of experiments that people talk about. One is screening, and this is really deciding what's important. This lets us take a very large number of factors and really just look at the rank order of which ones are important. When we have a smaller number, we can actually start to understand the complexity and define this relationship but within and between factors, and factors and responses.

This sounds like a lot, and I told you it was easy to get started. Here's what we're going to do. Screening is like the chocolate. It's like the good stuff. Characterization is like the peanut butter, it's got some complexity in it. What we have in JMP is a platform called Easy DOE. It magically combines screening and characterization. It helps us complete our objectives. We don't have to get bogged down with deciding when do I do one, and when do I do another? It's going to help us do this automatically. I'm going to show you this.

Before we get there, we do need to briefly talk about steps in a DOE. As you stare at this slide you're like, "Oh my! That's a lot. What are we going to do here?" But this is exactly the set of steps that the Easy DOE is going to take us through, each one of these steps, and it's going to give us a guide through each one. I'm not going to talk about the details here because I'm going to show you in the Easy DOE platform. This is a very classic picture. Many of you may have seen this, and this is exactly what we're going to do in Easy DOE.

I'm going to leave PowerPoint behind in a few minutes, but I also want to talk about augmentation. I think doing a designed experiment—is really great, but we cannot underestimate the importance of augmenting. When you do a one-factor-at-a-time experiment, it's hard to move on and do the next experiment. You're really just taking a few values that you found and moving forward.

When you have a designed experiment, you can utilize all of that knowledge that you've learned, and you do not start over from scratch. What you're doing is you're taking that underlying statistical model and adding more information to it. Augmentation is a very important part of doing designed experiments, and really, you get all of this extra knowledge without starting over. I think that should be one of the more compelling reasons to try using designed experiments.

Let's do some Easy DOE in JMP. This is a JMP journal. If you're not familiar with it, it's just a way to take notes, and I have data tables linked to it and other nice things like that. Where to begin? One of the things that I do want to talk about is a little bit of understanding what relationships are. Just for a little background. If we have two factors, and they're on the X and the Y, we can have a positive relationship or a negative relationship. We can also have something that's well described by a straight line or something that has to have some curvature in it. There's a little demo here that comes with JMP. It's built in, and it lets you add data points, and you click, so you can start to understand how to how this correlation works.

The other thing we need to understand is a regression. The correlation is the relationship between variables. The regression is actually this process of drawing a line. You can see here, again, I can change the relationship. This should help you start to understand what you're looking at when you interpret your own results.

There's one more thing I want to share with you before we get into our example. When we have a model, so a model is a statistical relationship, we all know y equals mx plus b. If we only have that it's just called Main Effects. Here we have X1 and then X2 with our response variable on the y.

What we see here is each X has an influence on the response, but the two X1 and X2 do not influence each other. Now, maybe they're not straight lines. This is what I was referring to before by the difference between a linear and a nonlinear or curvilinear relationship.

Quadratic is the mathematical way we describe curved lines. Again, X1 and X2 are not influencing each other, but they have this more complexity in the relationship to the response variable. X1 and X2 might be changing each other. This is the underlying complexity that really starts to make your innovations hard and where designed experiments can help you understand this. You can see if we change X1, the relationship of X2 and the response changes. You may have seen this without knowing it, and it just can be so frustrating if you don't have a way to actually define it. Then we can do both. We can have curved lines, and our factors influence one another.

These are all the kinds of levels of complexity that we need to think about when we go to make a designed experiment. We're going to make that easy, and we're going to do that with Easy DOE. Where we start is under DOE, and we're going to do Easy DOE. You can see, as I showed you in the other slide, here are the different steps that you need to do. Actually, let me just open one that's already done. Here are the steps that I showed you in the slides. Define, model, design et cetera. These are all right here for you.

Notice that there's this guided and flexible mode. If you already know something about design experiments you might want to choose the flexible mode. It's just going to give you additional options for design choices and types of analysis. But if you're new, which is really the audience that I'm talking to, just put it on guided mode.

I'm going to… Here's our blank one. Let's look at our responses. The response is what you want to measure. The first one we're going to do is a screening design. We just want to understand the rank order of our factors. You might have a goal in mind, you might not have a goal in mind. There's hints here if you want to try to understand more, but it saves you from going… The documentation is really good. It takes you through examples, but it's all right here for you to reference as you make these decisions.

I want to add a response and match a target. I'm going to delete that one. What we're doing is making coffee. I like coffee. We're going to look at coffee strength. It's somewhere what we're looking for is somewhere between a value of 1.2 and 1.4. We could just want to… You could also want to maximize or minimize a target. For example, if we had cost, we might want to minimize cost. Those are our responses.

We also need to add factors. These are the things that we're going to change in our experiment. Watching me type is a little boring. Let me just address what these different factors are. But I'm going to just show you what… You can have a table with factors. What I'm going to do here is I'm going to load these, so we can look at them. We'll look at each one. Here I have the grind. Coffee can be coarsely ground or in this case medium ground. This is a categorical response. If you're going to add a categorical response. It's something that comes in discrete units. Red/blue different vendors. In this case, the grind.

My temperature is a continuous variable. What you put in is you put in the lowest level and the highest level. Continuous variables are anything that might have a knob that moves continuously. Temperature we can have any amount of variation between temperatures. Our next factor is also continuous. This is time. Time is always continuous. We can have this go between three and four. Charge has to do with the brew strength. How long the machine is actually or the setting on the machine that changes the strength of the brewing.

We have two different coffee stations, because maybe there's a difference between stations that I am unaware about. This brings up the important idea of noise. One of the things that design experiments does is randomize, so that you don't have to worry about everything. Because it could be that maybe the heating element on one of my stations isn't really as good as the other one, but randomizing between them will help obscure that fact and take the noise that we don't care about, and we're not trying to define, and let us help get a better handle on that.

The other one that's up here is a discrete numeric factor. This is for when you have something like an oven. Let's say you only have just a couple of settings on the oven. While temperature could be a continuous variable, you're not actually going to run it that way. You could say, "Why wouldn't I just run that as categorical?" Because you'll get more information out of discrete numeric.

If my two values for discrete numeric were 10 and 100, it would recognize that difference or 10, 20 and 100. The difference between 10 and 20 is only 10, but 10 and 100 is 90. It will take that into consideration. If I just cram those into a categorical factor, then it's just going to treat it as A, B, and C, and I'm going to lose that additional information that I could have gained. Let's move on. When I click my navigation controls it is going to move me to the next tab here.

Model. I think this is where people start to worry. There's a nice little explanation here of what some of these things mean. It even says right now we're just screening. We're we just want to know which of these items are the most important. This is why I showed the illustration of interactions and quadratics, because we can put these in and test for them.

One of the things you can see is that to get more information about what's going on, whether our factors are linear or not, and if they interact with each other, it costs us more in the number of runs that we do. This is why if you have a large number of factors, you're better off doing a screening design and then subsequently doing a characterization design. All I really have to decide in Easy DOE is which of those questions I'm trying to answer.

In this case, I just want to know what's important. We're going to stick with that. It's made my design for me. Here's my design. I can export the data. I'm going to make a JMP data table, and then I can go off. You need to run each run in order, just like this says. The first time I do my experiment, I'm going to do it at station 2, I'm going to use a coarse grind at a temperature of 195 for a time of three, and a charge of 2.4. The second time I do it, I also use station 2. I'm going to use a medium grind, then you can see the other variables here. The run order is really important. This is part of the randomization. If you run off, and you just sort this then you've not respected what we're doing with the design of experiment.

It's a little beyond what we're going to talk about today. If we go back to our model and we… No, I'm sorry. If we go back to our factors and our flexible mode, we can add additional information about things that are really challenging to change. Big vats of acid if you're dipping something in it. This is going to change the kind of underlying statistical model that we're building. But it is here, and you can do it, and you should use it. Industrial experiments frequently have things that are challenging to change, and we can't use all the randomization. Let's continue on.

I've already done this experiment. Here's my data. You can load the responses. You just select a column from a data table. That's the best way to get your data back in. On the previous one you export the data table, you fill it in, and then you can reload back into Easy DOE, what your answers are. What's this little graph down here? This is showing us a preview of our results. Here is my coffee strength, so my outcome variable on the Y. Each data point here is one of my experiments. We can get a preview of what the results look like for each factor. Let me make this a little bit taller. There we go.

It looks like grind might be important. It's hard to know about some of these other ones. We're not just going to eyeball it, but it does show you the spread of the data for each of your data points. As I click next here on my navigation control, I get to the Analyze.

This is really cool. There is two different things here. We can look at everything in the model. The solid line means it's significant, and the dotted line means it's not significant. Just to be really clear, in designed experiments, not significant doesn't mean it's not important. It means you did not detect something. Maybe if you did twice as many runs, you might have detected it. In this, we have no evidence that it's important. The full model is everything. If we want to see that, we can look at those results.

What we're looking for here is that our actual estimate is bounded away from zero. We do that by looking at the confidence intervals. We can see that in our grind. It's plotting this for us. In contrast, for other ones, it's saying it's not significant. You can see it crosses this line at zero. If I click a Best Model, it says the best model to explain our DOE includes grind and maybe temperature. Temperature is kind of on the border. We might want to just keep that in for now. Time, charge and station are not important, so we can move on and think about what else we might want to do next.

There's a prediction profiler. Prediction profilers are the way that we visualize equations in JMP. Again, here is our y-axis, and the blue is confidence intervals. Then each square is our outcome variable. This is interactive. When we're here we can not just click around in it but wherever there's a red triangle in JMP, we have a lot of options, and we can do optimizations here. That can be a really important thing if you want to go back and actually try to verify what you did. Go back and make one more pot of coffee and see if it's the best. There is also a nice report here that you can print out, and it has everything on it. That is one example of a DOE. That's a screening design.

Now, let's go ahead and do a characterization design. In this one, our factors are going to be pH, temperature, and vendor. Our responses are going to be, whether it's the target, and then we also want to keep track of impurity. You can see here we're going to maximize or minimize. It's boring to watch me type, so I'll just load these into Easy DOE. I'm going to use this little red triangle right here. Load my responses and load my factors. We can see that it's just like if I chose these using the plus button target and impurity, I want to maximize my target and minimize my impurity. My factors, I have three factors. They are pH which is continuous, temperature which is continuous, and vendor which is categorical.

I'm trying to do characterize the complexity of the relationship here. I don't know if I have straight lines or if I have curved lines, and I don't know if these two things interact. That's what I want to know. See, this is the cool thing when you have fewer number of factors, I could do any of these three kinds of designs to estimate these models for the same amount of runs.

I'm going to pick the one that's going to give me the most information. Looking at interactions and quadratics. It's also called a response surface design. Again, it's going to go off. It's making the model for me. There's my 18 runs. I'm going to export my data. Here's my data table.

Hopefully the magic of my simulation will work. I get in my time machine, collect my data. There's the data. The time machine is not included with Easy DOE. We're going to click Next. Now I'm going to load my responses. Again, if we look at these factor plots, we can start to get an idea of what our results look like. Here I have two outcomes: target and impurity. It looks like there's a relationship maybe with pH. The other ones it's a little hard to tell. Impurity might have something to do with my vendor.

Let's actually run the model. We have a lot of power here because our error bars are small. We can see a lot of solid green and a couple little dotted ones. If I click the Best Model, it's going to say, "I don't think these things are significant." One of the things here, the way this is written, this pH*pH, this would be to see if that line is curved, and it says it's not. Likewise, temperature probably not curved, but we definitely have some interactions. Interactions are written like this, pH*temp. There's some more information down here if we want to keep looking. We can look at each response because maybe we only care more about… This is for target, and then for impurity, we would find something that's maybe a little bit different. We really want to be able to look at these in a similar model.

Now we can come over to Predict. Again, in our profiler on top is target and impurity is below. Each of these is pH. We can see maybe there's some curvature here with pH and temperature. We can definitely see these flip around for vendor. Here I'm going to click my Optimize button, so I can find the settings that maximize my target and minimize my impurity. These are the settings I should go back and try to run this pH, this temperature, and this vendor. You do need to verify that you've done a good DOE. Again, my report is here if you want it, and we can just export this report.

That's Easy DOE in JMP. Briefly, who else? There are other kinds of DOE. There's a lot of stuff in here. They end up being slightly special kinds of DOE. There are designs for teaching the classics, designs for understanding what consumers want. If you're doing a computer experiment, there's space-filling designs. We have covering arrays if you're designing software. You can do formulations and mixtures. There's a lot of things you can do with mixtures. There's reliability and measurement system analysis designs, and there's advanced blocking techniques for additional complexity.

It's really important when we're trying to innovate that we have confidence when we're trying to cover this large space of complexity with our design. We have complex interactions, and design of experiments will help us smartly leverage this space, and it will increase our ability to do innovation. With that, thank you very much.