Problem to find optimal settings for a mixture with Bayesian Optimisation

[#1]

With a 24 run mixture DOE I could find very good models and optimal settings for a ternary mixture and one categorical effect, this was presented in foregoing discussion; find in attachment again the analysis and below screenshots of the desirability results. 

With Isomer 2 desirability result is worse; below the settings when locking Isomer 2

Starting with a 9 run candidate space filling set (see attachment Succesful 9 run) set or a 12 run Kowalski set the optimal settings with Isomer 1 can be detected. 

However starting from a 6 run space filling candidate set I always get the poorer optimal settings with Isomer 2, even after several added space filling runs! What did I do wrong and how to get from this 6 run candidate the optimal Isomer 1 settings? 

[#2]

No succes after double Max Multimodel Std Dev, low desirability at Isomer 1 and BO goes back to Isomer 2

[#3]

Frank, please start ONLY with your 6 initial space filling runs, without any BO scripts in the table (remove the scripts and delete any other rows).
Then, launch the BayesOpt platform with default settings, remove any automatic batch recommendation and add 1 Max Multimodel Std Dev for 2 iterations, and you should be able to get results close to your optimum with isomer 1 (and matching my testings). As soon as the two consecutive Max Multimodel Std Dev runs have been added, you should see a correct ordering of the isomer types :

[#4]

Hi Victor, I followed your instructions cfr. attachment and I can't replicate your resuts? The models built with the 6 initial space filling rus are poor so I don't understand why to sart here with Max Multimodel st deviation: the outcomes are not learning a lot from poor Gaussian Process models?

[#5]

In your result you have only 2 outputs; there must be three

[#6]

And after only 6 initial space filling runs Gaussian Y1 is very poor so additional space filling runs are required to get acceptable model performance;  unfortunately, each time again, BO detects isomer 2 as optimal (?)

[#7]

I have the same problem starting from initial 10 run space filliing DOE cfr. attachment, what went wrong? We checked in the lab and the correct optimal solution is with Isomer 1. Isomer 2 gives worse results and Isomer 3 is totally unacceptable. Impprtant to note is that starting from 9 run initial space filling DOE's BO quickly converts to the right Isomer 1 setting! 

[#8]

Succesful 9 run space filling DOE. Strange that with lower #runs results are OK

[#9]

Some answers to your many replies:

• You're able to replicate the results, see the input and responses values for the second Max Multimodel Std Dev run in your file and mine. If after this second run you force the BayesOpt platform to add a run with Max Expected Improvement (instead of the automatic recommendation of MaxPro space filling run criterion), this second added run will be repeated, and is close to the optimum you're expecting with isomer 1.
• I had three outputs in the files I send: Y, Column 6 1 and Column 6 2.
• MaxPro Space filling criterion run are indirectly improving the model by forcing the exploration of the design space by maximizing the distances between existing design points and new ones. The effect on the model is not certain, depending on the complexity of the response. This is why in the JMP Help and in my answer I mention this option as a "model-free" exploration option. On the other side, Multimodel Std Dev and Bayesian Desirability Std Dev are model-based exploration of the design space : they use the model and its prediction uncertainties to recommend experiments in area where uncertainty is the highest.
  So to get acceptable model performance, you need both : 
  • Space filling runs to ensure a good coverage of the design space and make sure the model is able to generalize the learned behavior, and that this behavior is representative of the real phenomenon in the design space.
  • Bayesian/Multimodel Std Dev runs to ensure the reduction of prediction uncertainties to reach an acceptable level of precision.
    
    Please see my previous explanation here.

It's requiring a lot of time to look at all your different scenarii. What you have to remember is :

• Bayesian Optimization has the objective to reach your targets: as soon as an acceptable result is reached (based on your conditions), the loop will converge to it, no matter how global or local the optimum may be. So depending on how "tight" your target specifications are (and how complex the response surface is), you may have a premature convergence or not.
• By default in the BayesOpt platform, the exploration/exploitation part is automatically set depending on the R² threshold (default = 0,25): if the model R² is below the threshold, the recommendation will be MaxPro space filling runs. If it is above the threshold, it will start the exploitation with Max Expected Improvement criterion runs. The other options need to be enforced manually.
• I would highly recommend spending time looking and investigating your model; BayesOpt may look like an automatic experimentation loop, with good default experiments recommendation, but you can have better results by enforcing specific runs needed for the models.
• Your different scenarii with different number of space filling runs provide a different amount and precision of information. The less runs you have when starting, the harder the task, and so the more information and care your should bring to the models, by providing adequate runs with the right information, either by exploring the design space with MaxPro space filling runs or improving model uncertainties with Bayesian/Multimodel Std Dev runs. The strategy and choice of acquisition functions, number of parallel or sequential runs, etc... should be guided by the information you have, the precision and learning of the models obtained.  

Hope this answer will help you understand your different results. Please read carefully the files I have sent earlier, you were able to reproduce my results, you just need to enforce the same runs using the Profiler and dedicated acquisition functions (and not let the automatic selection works that give you a MaxPro space filling last run).

[#10]

Hi Victor in your succeful 6 run initial abov I only see 2 outputs Column 6.1 and Column 6.2 see below. There i a 3rd one Y, I don't see it 

Incuding Y I can't detect the right optimal settings also not with 10 run see attachment (I recoded the 3 outputs as Y1, Y2 and Y3)

What did I do wrong? It is important for us to understand.

With 9 intial runs it always works perfect, why?

[#11]

"Space filling runs to ensure a good coverage of the design space --> I have used fast flexible space filling, OK? Is assessment of the initial starting DOE is necessary? I saw a number of webinars where one  starts from just a vey limited #initial runs, sometimes even from an OFAT; guess this will only work when model is simple and when data are not too noisy, correct? This is not always the case and we don't know in advance.

Make sure the model is able to generalize the learned behavior, and that this behavior is representative of the real phenomenon in the design space" --> In my examples models have R² > 0,8, are there other criteria?