Problem Bayesian Optimisation

[#1]

In an earlier R&D study I sucessfully used a 4 factor mixture DOE (3 mixture factors and one 3-level categorical "Isomer" effect) to find optimal settings for a chemical formulation with three quality attributes Y1, Y2 and Y3.

I used the case in a DOE course and used simulation formulas to compute the responses Y123. Good models are obtained and optimization was straightforward and OK, see "ANALYSIS MIXTURE DOE CHEMICAL FORMULATION" in attachment.

Starting from a 12 run mixture Kowalski model DOE I am struggling with  Bayesian optimization (see annex "BAYESIAN OPTIMIZATION") This keeps failing because of a very unreliable Y2 model? Why does Bayesian recommends "Max Desirability" with this very unreliable Y2 model? Starting from other screening & space filling DOE's  I keep getting the same unreliable Y2 model.

What is the problem? How to solve?

[#2]

Hi Victor, I have tried your proposals with my JMP 18 without succes. Find in attachment a comparison between Scheffé & Gaussian modeling for the full 24 run DOE results: Gaussian modeling is not good showing lack of fit for Y1, very poor results for Y2; Y3 is acceptable; with these gaussian models it is impossible to find optimal mixture settings with profiler, Desirability = 0. Do you get better results with JMP 19? Hopefully a solution can be found, very much I would like to use Bayesian optimisation but my team is getting sceptical..

[#3]

The results from Gaussian prediction formula are very bad and different from Jacknife predictions?? 

No wonder why my Bayesian optimisation does not work, very inconsistant and unstable results..

[#4]

If results with JMP 19 are OK, what has changed?

[#5 · Accepted solution]

I have used the BO platform in JMP 19 with default settings, and the models for the different responses seem good:

I have saved the prediction formula and script from the Bayesian Optimisation platform (Gaussian Process model) from JMP19 in your table.

Concerning the "basic" Gaussian Process models, I have similar results for Y1, but you can improve the modeling by checking the option "Estimate nugget parameter" when launching the platform:

As the response Y1 seems to be a bit more noisy than the others, using this option improve the modeling by avoiding the Gaussian Process to perfectly interpolating the data. The model's results are much more interpretable and similar to the regression model, even if the model's predictions can be far from the measurements:

I have also saved the script to re-run the analysis and the prediction formula in your table.

Finally, Gaussian Process prediction formula and Jackknife prediction formula are different, you can read the JMP Help Actual by Predicted Plot. Jackknife prediction values are predicted Y values for the specific X row excluded, whereas the Gaussian Process model prediction values come from the averaging of each models trained with Leave-One-Out validation (Jackknife method).

Hope this answer will help you,

[#6]

OK Victor, thanks, that's clear I need to switch to JMP 19, I wonder why JMP 18 does not work and what has changed? By the way when I add another BO batch my 18 version gives an error and when I ignore this Y2 prediction is again competely blurred grey like before

Could you carry out this Bayesian optimisation from attached starting DOE based on a 12 run Kowalski model?

[#7]

Hello @frankderuyck ,

out of curiosity - where in JMP 18 do you have access to Bayesian Optimization? It was only introduced in JMP 19: https://www.jmp.com/en/software/new-release/new-in-jmp

Best

Florian 

[#8]

Sorry I was wrong, I am using version 19.0.1

[#9]

So why are my results different from Victor?

[#10]

On this screen you shared, click on "Check for Updates" (blue text on the left), a window will open and you'll get a link to download the updated version.

Thanks for the info @Florian_Vogt, the release notes about 19.1.x are not detailed enough to see if something has changed in the algorithm/calculations from the BO platform: https://community.jmp.com/t5/JMP-Knowledge-Base/JMP-New-Features-and-Release-Notes/ta-p/575202

[#11 · Accepted solution]

Good Morning @frankderuyck and @Victor_G ,

I can confirm that some of the improvements in the latest release solve the problem:

Before JMP 19.1.4:

With JMP 19.1.4 (current latest release):

Updating to the latest release includes improvements to the platform giving better results (as Victor showed).

Best

Florian