Experimental Design Training Meets Bayesian Optimization: A PPG Case Study (2026-US-30MP-2944)

My name is David Fenn. I'm an Associate Fellow working for PPG Industries at the Coatings Innovation Center just outside of Pittsburgh, Pennsylvania. Today, I'm going to be covering a practical evaluation of the Bayesian Optimization platform that's part of JMP Pro. Really going to be asking a simple question. Can Bayesian optimization help a formulator or a chemist working in the area of coatings to identify robust, high-quality performance solutions using fewer experiments?

To do that, we're going to be using a simulation that we've developed as part of our advanced experimental design training here at PPG. First of all, I'll go into some of the challenges that we have to deal with as coating chemists. Then I'll talk a little bit about the simulation that we're going to use to evaluate Bayesian optimization. I'll go into a little bit of the JMP workflow that we implemented.

Then, finally, talk about our conclusions and the lessons we learned from this exercise. It's going to be a practical look at Bayesian optimization, not really going to go into any of the details that lie behind it.

To help set the scene, it helps, I think, to understand the sort of challenges we face as coating chemists working for a company like PPG. If we look at the types of coatings that we need to develop, they have many different applications. Anything from infrastructure, automotive, aerospace, consumer electronics, food and beverage packaging, architectural, many different applications. Very often, those coatings serve a decorative purpose, so their presence is very visible, and they affect color or the appearance of the articles.

But sometimes, they're less obvious. May not even be aware that a coating is there, but almost always, they serve more than a decorative purpose. They have to achieve many different properties. For example, things like corrosion resistance, chemical resistance, barrier properties, adhesion, and many others.

As we're working as scientists and formulators in this area, we have to balance all of those different properties, and we have a wide range of variables that we can choose from to try and optimize this. The question we're always asking ourselves is with a limited experimental budget and working within such a large formulation area, how can we simultaneously optimize the properties we're interested in as efficiently as possible?

Traditional DOE has been a go-to tool for us for a long time to help us understand the systems we're dealing with. But there can be a number of problems in terms of implementing that.

One is, as I've said, we have many variables to deal with. How do you work out which variables to build into your experiments? Secondly, we're often starting with very little initial data to deal with. Then, finally, the experiments that we carry out, very often, they're expensive both in terms of time and resource.

The principle of Bayesian optimization, which is a much more sequential, goal-driven approach, seems potentially attractive. If we get into a little bit more detail, just comparing traditional DOE with Bayesian optimization, I think DOE is really strong if the primary question we're trying to answer is, how does this system we deal with work?

The DOE approach of structured, very often defined designs that might be factorial, fractional factorial, response surface designs, allow us to estimate the main effects that are involved and interactions that are involved, and then ultimately build a predictive equation that can predict future results. That tends to give us a very strong understanding of the entire formulation space that we're dealing with. But particularly when we have many variables, it can require a large upfront commitment in terms of how many experiments we have to run before we start to get any of this information.

If we contrast that with Bayesian optimization, a much more sequential approach, and really, that's good if the primary question we're trying to answer is based on everything we know and what we're trying to achieve in terms of our goal, what's the next best experiment to run? That's really the goal of Bayesian optimization.

Right from the start, even with very few experiments, it'll build a predictive model, but also an estimate of the uncertainty within this model. Then it's balancing two different approaches. Exploitation, which is based on the models we've got, what does the model think would be a good next experiment to run to get the best possible result?

Then exploration, which is more focused on where are the areas for high uncertainty, and where could we carry out experiments to improve our models the most and maybe even then identify better solutions. It's really balancing that exploitation and exploration type approach.

The output of Bayesian optimization is, what is the next best experiment to run? What's the most informative experiment that I can run next? A much more sequential approach. That, certainly when we're up against time pressure, that sounds like an attractive approach for some of the problems we deal with.

But one of the issues, I think, is if you imagine somebody as a coating scientist working on a high-profile project that's got a very defined timescale, it would be a brave choice, I think, to opt for Bayesian optimization just based on these promises with no prior experience of the area.

We really were trying to address that by applying Bayesian optimization in more of a simulation-type exercise, but a simulation that we're relatively familiar with. As I said, that simulation is part of our advanced experimental design training that we run at PPG, and I thought it might help just to give a little bit of background of that.

We call our training within PPG, we call it Sigma Logic. It's a structured pathway to deal with areas like problem-solving, experimental design, new product development. There are four critical areas that we deal with. First of all, critical thinking and project scoping. Then, understanding and managing variation. Then, finally, on to strategy of experiment, which is where we start to introduce some of the concepts that are involved in classical DOEs. Then, finally, holding all that together is the analytics. We use JMP as a platform for all of this training within PPG.

We first ran our first training course way back in 1998, and we've been running it continuously since then. Just to give you an idea of the amount of students we get through, since 2022, we've had 531 PPG associates come through our Sigma Logic training. It's a progressive pathway. All of our or virtually all of our scientists and engineering graduates joining the company take Initiator training, which really introduces the concepts and makes everybody familiar with the language that we use.

Then the option after that is to go on to Accelerator training, which is more in-detail practical application of the concepts that we covered in Initiator, and that involves about 36 hours of training. Then for those that want to get into much more detail and become more experts in this area, we offer a Master Catalyst training, which is about a further 100 hours of training. That's the part where this simulation is used as a competition.

Then, finally, the graduates from our Master Catalyst training actually become tutors or a lot of them become tutors that teach the modules within this. That's the background for the training that we do here at PPG.

Now, to go on to the particular exercise that we use. This is a simulation, and it's the development of a coating. We actually tell them this is a coating for garden sheds, although the application's not critical. But there are 12 factors that they're dealing with in this simulation. Four factors that are associated with the preparation synthesis of a resin, four factors that deal with the formulation that that resin becomes part of, and then four factors that deal with the application process for that coating. All of these factors have defined ranges that the students can explore.

Then there are five responses that we're dealing with here as well. You can see all of these responses have target ranges. Some of them, it's a specific target we're trying to hit. Some of them, we're aiming to minimize or maximize. Then, finally, we have a cost outcome where, obviously, we're trying to minimize the cost as well as optimize all these properties.

We score the students. They have a budget of 128 experiments that they can carry out over 4 weeks before they have to select their optimum levels for all 12 Xs. Then, once they've done that, we give them a thousand runs in this simulation, and we measure the capability based on these first four factors here.

Then we take whatever the minimum Cpk for whichever one of these four factors is, we take that minimum value, and we give them a capability score which ranges from 0 to 1. If their lowest Cpk is less than 1, they get a score of 0. If it's 2 or greater, they get a score of 1, and it's interpolated between that.

Then, finally, we give them a total score, which is their capability score divided by the cost of the product they develop multiplied by a thousand. Obviously, there's variation built into this simulation, a realistic amount of variation that allows us to do this capability analysis, and that's present all the way through the process.

This is quite a complex but realistic simulation. We know from previous teams' experience doing this, it is achievable, but it is challenging. But we liked it because it gives us a tool to rapidly test new experimental strategies as well, like Bayesian optimization. The other benefit here is that we have historical performance data from previous catalyst classes that we can measure Bayesian optimization against.

The question we're really asking here, can JMP Bayesian optimization find a robust, economical formulation with substantially fewer experiments than these students were able to find using standard DOE-type practices?

One final comment I'd make here is this is our Advanced Catalyst class. These are not new people to DOE. These are students who have been through a lot of training, who have been applying DOEs to projects for 4 years, 5 years, perhaps, and have just gone through some additional training. These are what I would consider to be subject matter experts for DOE.

That's the simulation. What we'll do next is we'll go into JMP, and we'll have a look at how we implement this. This was our starting point. We seeded the Bayesian optimization with a 16-run fractional factorial DOE. You can see we've got the 12 factors here and the 16 runs that we carried out. Then, here, we have our outputs that came from the simulation for each of these runs for our five responses, our five Ys.

If we go into any one of these columns, if I just select Column Info, you can see that there's desirability functions already built in to each of these columns. That's important because that is the basis for what JMP is going to take as our targets when we do the Bayesian optimization. That's our starting point.

Probably the easiest way to go forward next is what I'll show you is how we'd implement our Bayesian optimization based on this. To do that, we go into Analyze, Specialized Modeling, and select Bayesian Optimization. We'll put our 12 factors in as Xs. We'll select our five Ys, our responses, put them into Y. Then the only other decision we need to make is, how many experiments do we want to run at a time?

Now, you could do as little as one within Bayesian optimization, but since this is likely to have quite a long cycle... Now JMP will think for a little moment, and then it'll come back with this platform here. What we see here is, first of all, we'll look at the bottom section here. We have the models that we've built from these 16 runs that we've carried out. We can see there's quite a lot of uncertainty in a lot of these models, so not great models. That's exactly what we'd expect from our 16 experiments.

But the other important thing we're showing here is the overall desirability score that we've achieved from our 16 runs here. We can see straight away it's low for every one of these 16 runs. There was at least one of our responses that was well off the target region for these particular responses. That's our starting point.

Now to do our Bayesian optimization, we just select this batch selection tab, and this will give us our next six runs that we were requested. We'll just do that. There are a few options we could select here that I won't go into in any detail now, but once we've selected whatever options we want here, if we just select Make Table, then if I minimize this, we'll go back, and we can see that we've now got our 12 Xs, we've got suggestions for our next six experiments.

We just go ahead, we'd run these experiments, and then we'd carry out the experiment, measure our responses, and put them in here. That would be one cycle of our Bayesian optimization. That's how we get the process started.

Now I'll go into the situation after we carried out six iterations now. You can see this table is a lot more filled out now. We've actually carried out six different iterations of Bayesian optimization. JMP's recommended the experiments, we've carried them out, and we've measured them. I can show you basically how that progressed.

Here's the output after our six iterations. You can see right from the start, the models are a lot more defined now, much better models, less uncertainty involved in here. But I think the interesting thing to look at is this desirability. This is where we started with our 16 runs, and as we saw before, very low overall desirability. Our first iteration in our first set of six, nothing having a really good desirability function coming out of there.

Then, as we got into our second set, we're starting to see a bit of a jump in desirability, so we're clearly making progress here. Then by the time we got to our fourth iteration, we can see we've got really high desirability scores. Then the fifth and sixth really just confirmed that that was the optimized result. That was basically the progress that we saw by applying Bayesian optimization to this problem.

I'll now go back to the presentation and just put that in perspective a little bit. This first plot here, what I'm plotting here on the left-hand side is the iterations, so the collections of six experiments that we run as a time as part of Bayesian optimization. Then the observed joint desirability, so the overall desirability. The first attempt we made at this, round one, you can see we quickly progressed in terms of desirability, and then we reached a plateau by our third iteration and really didn't improve much after that.

But when we actually converted that to capability analysis, what we saw was the capability wasn't as good as we were hoping. I think that was because we set our desirability functions too close to the edge of the targets. When we took into account the variability that was in the system, we were actually achieving not 100% compliance with our product. Some of the later iterations, we made those desirability functions more difficult, more demanding, and the effect of that was it looked like, based on this joint desirability, it looks like it slowed the progress of Bayesian optimization down, and we actually plateau later, and we plateau at lower desirability.

But this is entirely because those desirability functions are more difficult to achieve. When we convert that into the overall score that's based on the capability of the system, we see the opposite. Our first attempt with the looser desirability functions plateaued at a moderate overall score.

But as we tightened up our desirability functions, we plateaued at higher scores. This final one where we had the most demanding desirability functions, we actually achieved a really good score, ultimately, by our fifth iteration. You can see the progress that we make. The main learning from that was that it's really critical to define these capability functions, these desirability functions, to get the best out of a Bayesian optimization.

Then, if we go on to compare performance of Bayesian compared to how the students performed using conventional DOEs, there's a lot going on in this plot, but just to talk through. The x-axis here is the number of cumulative experiments that were run. The y-axis is the lowest Cpk out of our first four responses. We're aiming to be in this green region here where the lowest Cpk is still greater than or equal to 2.

Then we have plotted here the performance of the teams in each of our catalyst classes. These are four classes, three classes that were run in 2022, 2024, and 2026. Each class would have had about four or five teams of four students who are working through this and really trying to progress.

Everything is shown here in gray, but the blue, the orange, and the green lines are the best-performing teams within each class. You can see those best-performing teams typically got to an endpoint of a Cpk of greater than 2 for the lowest performing response in about 100 runs, somewhere between 100 and 128 runs. The black line is what we were able to achieve with Bayesian optimization. We see we got there in about half as many runs, about 50 runs.

I think the conclusion from that is with carefully defined parameters, carefully defined variables that we're dealing with, carefully defined responses and desirability functions, with a combination of Bayesian optimization and subject matter expertise in this particular exercise, Bayesian optimization was significantly faster than traditional DOE.

Just to wrap up and cover what we've learned from this. In this 12-factor, 5-response simulation, Bayesian optimization reached a minimum Cpk of about 2 in about 50 runs. I think overall, it's important to think about what the question we're trying to answer is and then choose the best method. If we're trying to just understand the entire formulation space, then DOE really helps us to build that understanding.

But if we have a very goal-focused target, then Bayesian optimization can be a really efficient way of identifying high-performance candidates. Definitely have to be careful about defining what we mean by better, what our desirability functions are. Getting that right was proved to be absolutely critical. We're convinced that the best solution generally is combining both strengths.

Starting with a screening DOE seemed like a good idea, then moving on to Bayesian optimization, and then definitely, we'd want to confirm the robustness of the solution there and maybe even do an optimization DOE at the end there. Just in general and just to wrap up, we feel like the best fit for Bayesian optimization is situations like this where we have relatively costly experiments, quite a large number of variables to deal with, a meaningful time pressure on the project, and well-defined, optimization-driven goals.

Sounds great. I am so excited about Bayesian optimization. What a time to be alive, you know?

Yeah.

Who knew?

Yeah, it is. We were actually really quite surprised by how effective it was in this exercise.

Yeah. To cut your time in half or number of runs in half, that's amazing.

Yeah, it was interesting because, obviously, the first iteration, we didn't get much improvement at all. I think if we'd been in a real-life situation, we might have been questioning our choice to use Bayesian optimization. But then when we did that second iteration and saw things starting to jump up.

Especially with starting from 16 runs, you would think that it would be able to learn so much more with 16 than all the demos I've seen have been one run or three... Not one, but three runs or five runs. To start with all of that, you would think, "Oh, we'll go to the optimum immediately." But it took a while.

Right. It did, yeah.

That's really cool. Great. Thank you. I'll do all the process Not the processing, but all the files in the places that the people need to have them processed.

Yeah. The one thing I noticed when I was in JMP, there were a couple of pop-ups that came, and I'm sure I've done everything right to stop it, but for some reason, they still seem to crop up.

I was listening, but not watching, so I didn't see them. Do you want to show me an example of one?

I don't know which ones popped up. I can't remember what popped up, but there were two that came in the bottom right-hand of my screen. I don't know whether it's possible to blur those out or something.

Okay. Those kind of pop-ups. Yeah, I'll see if they can. We've got good editors, so I'll see if they can blur it out.

Right.

Do you think that there were only two?

I'm pretty sure there were only two, and it was... When I was using JMP. It wasn't when I was in the PowerPoint. I think pretty much every one was while I was using JMP.

Okay. I'll run through it then and see if I can note the times, and then they can go through and blur those out. Yeah, no problem at all.

Let me know if that's not possible. I could record that segment again if needed.

Okay.

Yeah.

Sounds good. Well, David, I will see you in October.

Yeah, looking forward to it.

Me too. Me too. Thanks very much.

No, my pleasure.

All right. Bye-bye.

Bye.