Problem Bayesian Optimisation

[#1]

In an earlier R&D study I sucessfully used a 4 factor mixture DOE (3 mixture factors and one 3-level categorical "Isomer" effect) to find optimal settings for a chemical formulation with three quality attributes Y1, Y2 and Y3.

I used the case in a DOE course and used simulation formulas to compute the responses Y123. Good models are obtained and optimization was straightforward and OK, see "ANALYSIS MIXTURE DOE CHEMICAL FORMULATION" in attachment.

Starting from a 12 run mixture Kowalski model DOE I am struggling with  Bayesian optimization (see annex "BAYESIAN OPTIMIZATION") This keeps failing because of a very unreliable Y2 model? Why does Bayesian recommends "Max Desirability" with this very unreliable Y2 model? Starting from other screening & space filling DOE's  I keep getting the same unreliable Y2 model.

What is the problem? How to solve?

[#2]

Hi Victor, I have tried your proposals with my JMP 18 without succes. Find in attachment a comparison between Scheffé & Gaussian modeling for the full 24 run DOE results: Gaussian modeling is not good showing lack of fit for Y1, very poor results for Y2; Y3 is acceptable; with these gaussian models it is impossible to find optimal mixture settings with profiler, Desirability = 0. Do you get better results with JMP 19? Hopefully a solution can be found, very much I would like to use Bayesian optimisation but my team is getting sceptical..

[#3]

The results from Gaussian prediction formula are very bad and different from Jacknife predictions?? 

No wonder why my Bayesian optimisation does not work, very inconsistant and unstable results..

[#4]

If results with JMP 19 are OK, what has changed?

[#5 · Accepted solution]

I have used the BO platform in JMP 19 with default settings, and the models for the different responses seem good:

I have saved the prediction formula and script from the Bayesian Optimisation platform (Gaussian Process model) from JMP19 in your table.

Concerning the "basic" Gaussian Process models, I have similar results for Y1, but you can improve the modeling by checking the option "Estimate nugget parameter" when launching the platform:

As the response Y1 seems to be a bit more noisy than the others, using this option improve the modeling by avoiding the Gaussian Process to perfectly interpolating the data. The model's results are much more interpretable and similar to the regression model, even if the model's predictions can be far from the measurements:

I have also saved the script to re-run the analysis and the prediction formula in your table.

Finally, Gaussian Process prediction formula and Jackknife prediction formula are different, you can read the JMP Help Actual by Predicted Plot. Jackknife prediction values are predicted Y values for the specific X row excluded, whereas the Gaussian Process model prediction values come from the averaging of each models trained with Leave-One-Out validation (Jackknife method).

Hope this answer will help you,

[#6]

OK Victor, thanks, that's clear I need to switch to JMP 19, I wonder why JMP 18 does not work and what has changed? By the way when I add another BO batch my 18 version gives an error and when I ignore this Y2 prediction is again competely blurred grey like before

Could you carry out this Bayesian optimisation from attached starting DOE based on a 12 run Kowalski model?

[#7]

Starting from a 10 run Fast Flexible Space Filling DOE cfr. attachment I get different results from the 12 run Kowalski start and also the 24 Sheffé model (cfr. 1st post); X1, X2 and X3 optimal settings are nearly the same but Space Filling gives Isomer 2 as result instead of Isomer 1. As with Isomer 2 the Scheffé & & 12 run Kowalski starts yield poor desirabilities I guess starting from a 10 run Space Filling is not OK, too few runs? Is there a criterion for an accetable starting DOE for Bayesian optimisation? 

[#8]

Think the problem is that the Space Filling DOE does not yield a balanced coverage across the 3 Isomers. Also, looking at a scatterplot per isomer  the coverage does not look very well. Guess this is an important point for attetion when selecting space filling for starting in case there is a categorical effect, correct?

[#9]

Hello @frankderuyck ,

out of curiosity - where in JMP 18 do you have access to Bayesian Optimization? It was only introduced in JMP 19: https://www.jmp.com/en/software/new-release/new-in-jmp

Best

Florian 

[#10]

Sorry I was wrong, I am using version 19.0.1

[#11]

So why are my results different from Victor?