Automatic Visualization of DOE Model

We at NNE, we help a lot of our customers with doing design of experiments, DoEs for a lot of different reasons. For example, robustness tests, process improvement, validation. One of the big frequent challenges we have is communicating the results to stakeholders. There might be many challenges. Some of them might not be as statistically inclined, mathematically inclined as us.

In some cases, it might even be a small math phobia. A lot of them will not be familiar with the JMP interface, so it can be a bit intimidating also, or just the complexity of the model we've made could be a challenge to analyze or understand.

Another big part is that some of our analysis has been quite time-consuming and labor-intensive. It can be a bit boring to make. When you get bored, you also make errors and that's not good.

What we wanted to do was make a method where we could quickly and easily make some graphs. We could show to stakeholders, communicate the results. They should be made automatically, so there's less errors, and it should be fast, so we don't spend too much time just waiting on our results. It should be able to handle a lot of models because we work with a lot of models.

We came up with the graphs previously. That still took a long time. Generally, it was two graphs we made, one to visualize the range of the outcome and another to visualize the sensitivity of the outcome to the variables we use in our DoE.

We found a way to automate analysis, where we do the calculations required by exporting our models to Python, solving some global optimization problems, and then visualizing the results back in JMP, and then normalize the result there also to allow comparisons.

You can see one of the graphs here. That's the one we make here. That's the range plot. Here we have a sensitivity plot. For the range plot here, we have made a small example. The data is from soy-bean cultivars, different kinds of soy-bean, where we modeled the crop yield of this and then make this graph.

I can show the process of making this graph. It is quite simple. We have the data here. We have, in some way, made a model. That comma comes up here. Then from our tool, we just press a small button, and then it runs. For some models, especially if there's a lot of levels to some of the nominal variables, it can be a bit intensive. It has to solve a global optimization problem for each combination, but even in the larger cases, we get the results quite fast.

Here you can see the result. It took 20 seconds or something. From here, we can just plot what we want. In general, it will make a graph. You can customize however you want. Then some of these variables are only for some analysis. Here, we just have almost instantly a graph where we can just see what we want, easily communicate to stakeholders.

For example, for this data set, we might want to find the best crop to use depending on the season. The point shown here is the nominal value. That is the value, the yield we will get if we use crop and have all other settings set to the middle value. The range, the bar here, it shows the range of the value can take.

For this crop, it can take a value between 4,700 or down to around 4,000. The dotted lines are the control limits, that is the nominal plus minus free standard deviation, along with, in this case, the target and spec limits. We can instantly see that for the season 1 here, we immediately have three or four good candidates that are well within our limits and even one that's almost exactly on target.

We can come and have that to season 2, where they generally lie a bit lower, but we can still find some that would work. This was one use case for it. Now we can close all of these. Because another thing, this just had one output variable.

Another case is that we can have more than one output, and those outputs can have different scales. We have here another example. This is simulated data, but based on a real example. We had a product that we wanted to analyze. It was a manufactured product. They measured many dimensions, and we wanted to see which dimensions were most critical for the outcome.

Here, we have two explanatory variables and five dimensions, and these all have different scales. I can just skip this. We can once again run our model. It takes 5 seconds or something like that to plot, and now we instantly have our results. We can also here quickly see the problem with different scales because if we just try to show it, we can only see the dynamics for dimension 4 because it's so much larger than the others. We could try to scale it, but we could have the same problem. Dimension 1 and 2 are larger than dimension 3, so we can only see one or two at a time.

What we do instead is we normalize our plots in some way. We could, for example, here, normalize it to the target. Here, we just put our variable down here. Now we have normalized it, so the target is always at zero, the upper spec limit is at one, and the lower spec limit is at minus one. Now we can instantly see, well, for dimension 3, if we have set up 2, we are always out of spec.

We can see that both dimension 1 and 2 doesn't have that much variability compared to the spec limit. In set up 2, dimension 2 will vary a lot, can vary a lot depending on our input variables. If we're in this setup, we would have to control our temperature quite well.

This one works if we have target spec limits. If we don't have that, you could do other stuff. We have another that's just normalized to intercept and spec limits. If we don't have a specific target, we just want to make sure we're inside spec limits, and that's normalized here. We can have a final one that's normalized to the intercept standard deviation, which always works, but we might still run into the problem that there is a different scale here.

For example, here we can see, dimension 4 is quite large compared to the residual variance, whereas dimension 1 doesn't have spec limits very large compared to the residual variance.

Finally, you might not be interested in just what your values can take, but also how would you want to control these outcome variables. Here we have another data set where we have this data is for calories burned as a function of age, height, weight, gender, stuff like that. We want to know how many calories could I burn as a factor.

We have the data set. We have our model that comes in right here. We press the button. It takes a few seconds, and then we have a graph. Here, we plot the sensitivities. What are the sensitivities? That is how much the outcome variable will change depending on your nominal settings and for each continuous variable you have.

For example, in this, we can see that, if it was in my case, If I do some exercise and I exercise for one more minute, I would expect to burn around 5.5 more calories. If I made it more intense, increased my heart rate by one, I would increase the calories burned by about 2.5 calories.

The other variables in the model might not be as impactful for practical purpose. My age might not matter that much… My weight might not matter that much. However, this is, of course, insensitive to the scale of the variable. If I can vary my heart rate more than the duration, it might still have a significant impact.

We have another way of normalizing that is just by taking the gradient, taking the sensitivity, and multiplying it by how big can the range be… How much can I vary my heart rate versus how much can I vary the duration.

Here we can see where before, the heart rate was not as impactful. We can now see that the heart rate might have just as big of an impact on the calories burned compared to duration because I would be able to vary my heart rate more than my duration.

We can also see that just by during the duration, I would be able to almost burn more calories. By doing more intense exercise, I would be able to burn around 60 more calories expected. We can actually see when I for how much I can vary both my weight. My weight could vary, or the age could vary. That might still have a practical effect, which we couldn't see before, where we're just looking at the individual, the estimated permanency effects.

Now we just close all this down. If we go back to start. What we were trying to do was make this automatic method of analyzing experiments. We have a method that now works. It takes relatively fast, and you can customize it to what you need depending on how you want your graph to look like. That is about it.