Problem to find optimal settings for a mixture with Bayesian Optimisation

[#1]

With a 24 run mixture DOE I could find very good models and optimal settings for a ternary mixture and one categorical effect, this was presented in foregoing discussion; find in attachment again the analysis and below screenshots of the desirability results. 

With Isomer 2 desirability result is worse; below the settings when locking Isomer 2

Starting with a 9 run candidate space filling set (see attachment Succesful 9 run) set or a 12 run Kowalski set the optimal settings with Isomer 1 can be detected. 

However starting from a 6 run space filling candidate set I always get the poorer optimal settings with Isomer 2, even after several added space filling runs! What did I do wrong and how to get from this 6 run candidate the optimal Isomer 1 settings? 

[#2 · Accepted solution]

Hi @frankderuyck,

Did you upgrade your JMP version as discussed in the previous discussion ?

Looking at your last file, when launching the BayesOpt platform from scratch only using the 6 initial space-filling runs, the models seem already quite good :

So instead of using the automatic recommendation "Replicate Best Training Run" (I deleted this run in the batch), I manually force the use of the acquisition function "Max Multimodel Std Dev" and add 1 run in the current batch, as the differentiation between Isomer 1 and 2 is difficult because of the uncertainty of the model's predictions (you can look at the confidence intervals between isomer types on the Profiler). 

When relaunching the platform with this newly added run, the profiler seems to be more reliable, and the default run recommendation is done automatically with the acquisition function "Max Expected Improvement", with settings close to your optimum :

Once this automatically recommended run is added, the next option recommended by the platform is to replicate this best training run:

So given the relatively low complexity of your two responses, it is possible to start from a 6-runs space filling design and get an adequate optimum recommendation with 3 runs added, provided you think about which acquisition function is the most relevant given the learning of the models and their behaviors and you manually "enforce" this option.

Please find attached my runs situation with your 6-runs example. Done with JMP Pro 19.1.3

EDIT: I have missed the column Y in the optimization. However, I can obtain good results even when not considering it, as it seems to be negatively correlated to Column 6 2.
When considering Y and the two other Column 6 responses, you may need one extra Max Multimodel Std Dev run (so two in total, one after the other), before the Profiler shows good ordering and behavior of the different isomer types:

Once you have added 2 runs Max Multimodel Std Dev, you can start the optimization by enforcing the Max Expected Improvement criterion. You should get a solution close to the one obtained with your previous successful attemps (see file Successful 6 Run Fast Flexible Filling Design starter 2).

Hope this answer will help you,

[#3]

Hi Victor, yes I upgraded to las JMP version. Unfortunately I can't reproduce your results; BO always starts with space filling; when replication at Isomer 2 start and I am not happy with desirability I go over to Max Multimodel St dev but I have no succes.. In an earlier webcast on BO switching to Max Max pro criterion is recommended but also this does not work. In attachment my results (there are 3 outputs Y123) 

[#4]

How do you know that in first BO run Max Multimodel St dev is necessary instead o space filling? 

[#5]

I can't reproduce your results when starting with Max Multimodel St Dev?

[#6]

I'm not sure what you've done, but it seems you manage to get to the optimum, using only 1 run with Max Multimodel Std Dev criterion, and 1 run with Max Expected Improvement ? So mission successful ? 

From my side, I needed 2 runs with Max Multimodel Std Dev and 1 run with Max Expected Improvement (same inputs as the second Max Multimodel Std Dev) to get close to the optimum.

[#7]

I forgot to answer on this criterion part :

• Maximize MaxPro Criterion is a model-agnostic / model-free exploration of your design space. It does not consider the outcomes and learning from the Gaussian Process model, it's only an option based on maximizing the distances between points. The goal is to add points/experiments in area of the design space where it is "empty" (maximize distance between existing points and newly recommended points).
• Maximize Multimodel Std Dev is a model-based exploration of your design space. It considers the outcomes and learning from the Gaussian Process model, and recommend points in area where the prediction uncertainty is highest.

In your 6-runs scenario, the models are not bad at the beginning: R² for Y is the lowest, but is at 0,3566 so the model has been able to catch something). R² for Y2 and Y3 is very high, so model-free exploration is probably not the best next action, it is more interesting to leverage the learning of the models and try to reduce the uncertainty in their predictions.

Hope this answer will help you,

[#8]

Hi @frankderuyck,

I would recommend starting from scratch your scenario, leaving only the 6 space filling initial runs and deleting every other runs.
Please restart the BayesOpt platform from scratch with default settings, and force the platform to add 1 run with Max Multimodel Std Dev criterion two times. You should then have the same results for these two runs, and you can then force optimization with Max Expected Improvement, it will recommend an optimum close to the one you have with your DoE (and same inputs as the second Max Multimodel Std Dev run).

[#9]

Default R² = 0,25? At start not increase to 0,7?

[#10]

There is no hard rules on this. JMP BayesOpt development team has chosen a threshold of 0,25 to guide the model to exploration (R² < 0,25) or exploitation (R² > 0,25) phase, but this is a rule of thumb guided by their work and simulations and that shouldn't be your only motive. There is always specific scenarii where this threshold may not be adequate.

Instead, look at the model, profiler and plots, and find the best action that will improve the information gathered, either by:

• Enforcing model-free exploration to ensure the design space is sufficiently explored (Max Pro criterion),
• Enforcing model-based exploration to leverage the first initial models and reduce their prediction uncertainties (Multimodel Std Dev or Bayesian Desirability Std Dev criterion),
• Balancing exploration and exploitation (Upper Confidence Bound criterion), or 
• Exploiting your models predictions and start optimizing (Max Expected Improvement or Max Bayesian Desirability criterion).