Bayesian Optimization: Maximize MaxPro Criterion

[#1]

Hi dear Community and JMP experts,

I'm confused by the denomination of one of the option in the batch customizer for Bayesian Optimization. One of the option is called Maximize MaxPro criterion in Bayesian Optimization platform. The Help in this section mentions: "This option is a model-free exploration of the factor space that avoids replication of any of the factor settings in both the training data and the current batch. Use this option for any batch size when one or more of the models is not fitting well."

Regarding the definition of this MaxPro criterion in the Fast Flexible Filling Design Details help section, "The MaxPro criterion maximizes the product of the distances between potential design points in a way that involves all factors. This supports the goal of providing good space-filling properties on projections of factors". It is also mentioned that the MaxPro criterion "strives to find points in the clusters that minimize the following criterion" (before looking at the formula):

As the product of the squared distances is at the denominator in the formula, minimizing this criterion does maximize the distances between potential design points. So maximizing the MaxPro criterion may actually lead to smaller distances between points, causing a higher risk of runs that may be clustered or "stacked" when looking at smaller subspaces of the experimental space. Looking at the Bayesian Optimization Profiler, it seems the formula used behind this acquisition function does not use (squared) distances at the denominator (like in the original MaxPro criterion), but at the numerator, since we can see concave desirability profile matching a -X² type of formula involving distances between original design points. MaxPro acquisition function is at the maximum where the distances between the new point and original design points are the highest:

So can someone clarify this confusion between the "minimize MaxPro criterion" of Space Filling designs and the "maximize MaxPro criterion" of Bayesian Optimization ? Does the MaxPro criterion in the Bayesian Optimization platform use the same formula as the one in the Space Filling platform ? What is the formula displayed in the MaxPro Space Filling Criterion acquisition function ?

Thanks in advance !

[#2]

Any chance to have an answer from JMP people or advanced users on this specific question/confusion ?

I really hope the documentation for JMP BayesOpt will be augmented, as it lacks some basic infos, for example about kernels/continuous correlation types formula and figures. There is already good documentation about the function types involved in the selection of the kernel:

https://fr.mathworks.com/help/stats/kernel-covariance-function-options.html
https://www.comsol.fr/support/learning-center/course/surrogate-modeling-theory-271/more-on-covarianc...
https://indico.cern.ch/event/1407896/contributions/6068668/attachments/2947366/5179857/Lecture_2_vka...

...

[#3]

Hi @Victor_G ,

  I completely agree with your assessment and questions about the new BeysOp platform. I've been wanting to try this out with colleagues at work ever since I first learned of it earlier this year, but I'm also balancing possible payouts to resource/time use and don't want to send colleagues down a rabbit hole without first trying to understand the fundamentals of how JMP implements BayesOp -- this part has been more challenging.

  There are some OK web tutorials on the early stages of development and the recent addition to JMP 19 Pro here:

https://community.jmp.com/t5/Phil-Kay-s-Blog/Active-Learning-in-JMP-Chris-Gotwalt-ENBIS-Workshop/ba-...

https://community.jmp.com/t5/Learn-JMP-Events/Developer-Tutorial-Bayesian-Optimization/ev-p/894910

  They're decent at giving a very high-level description of how things work (good for management, especially the second one up to about the 20min mark), but I still haven't been able to find more detailed explanations, or much in the way of how to tune/adjust the algorithms of the optimization and acquisition functions for the specific task at hand.

  In the tutorials, they acknowledge that there are risks of getting stuck in local maxima, but also claim their method can get out of this because it balances the desire to explore with the desire to optimize. The explanation is hand-wavy, and I understand they can't reveal everything under the hood, but some more insight into details would help. Especially for those of us in industry who see the value of this approach, but also would like to have more than a black box understanding so that IF things don't go as expected (which is most of the time), we can adapt so that there continues to be valuable information in the work that has been completed.

  Just some of my thoughts after trying to learn more about this latest platform in JMP.

  Thanks for the additional links and resources!

DS

[#4]

Hi @SDF1,

Thanks for your answer and complementary links.
While these links may be useful to get started practically with BayesOpt, they do not offer a deeper look at some technical choices like kernel, acquisition function forms (and equations), etc...

There are no guarantee to reach global optimum. I already have presented a use case in Discovery Seminar about BO where using only an "auto" mode (blindly following the platform recommendations for the addition of the next runs), I have a sub-optimal than what I can get when forcing exploration at the beginning, or even better, starting with initial runs created with a space filling approach: 

So I am a bit skeptical about the claims you mention: if your target has been reached during BO loops (no matter if it's a local or global optimum), then the platform will recommend replicate the point, and it will finish here. 

The topic of global vs. local optimum was a frequent question in this JMP Discovery seminar, as well as "how to use/trust the model'sp redictions", where I showed that calculating feature importance/sensitivity indices based on the Gaussian Process used during BO loop can be highly misleading : https://www.linkedin.com/posts/victorguiller_bayesianoptimization-designofexperiments-activity-74750...

This type of explainability method will only provide an explanation of the path followed by the algorithm, not necessarily the "real" importance/sensitivity of the factors.

Thanks a lot for the discussion and links !

Best,

[#5]

Hi @Victor_G ,

  Yes, I completely agree with you about the concerns regarding the algorithm and whether or not the iterations are accurately probing the design space in order to quickly find what is hopefully a global maximum (or even clustered maxima). To be clear, I didn't make any claims about the abilities, just was repeating some claims that were made in the online webinars. As I mentioned in my original reply, they're OK webinars to get started but don't dive deep into the methodology or algorithms and how to tune them -- definitely more steered toward more "casual" users of JMP and DOE methodology.

  To a certain extent, I agree and understand what's also being communicated/demonstrated in the presentations about the BO platform -- a lot of the time in industry, it's not so critical to have a deep fundamental understanding of how each factor interactions with all other factors and have this beautiful model. It's good enough to know that your 85%, 90% of the way there and have a stable process to generate your desired response. The extra 10%-15% can be optimized with small adjustments when you're near that optimum. But, being able to get close to that optimum quickly in the first place when not much is know about the operating space is very desirable.

  One thing none of the examples have done is shown what happens if you START somewhere near an optimum. What happens with the explore vs exploit algorithm if you've already gone through an BO optimization and found a "maximum". What if you then restart the process using a couple points near that maximum. Does it get stuck, or does it still search the design space? I can think of several scenarios where there could be at least two global maxima that can be reached via two different settings -- one of which might be more practical from a production standpoint. But, if the BO algorithm always gets stuck on one of the local maxima without exploring enough, then you could be missing important information.

  Thanks for sharing your comparisons running BO in "auto" vs supervised and how well the different approaches reach a desired goal. As with other experimental approaches, I guess I see it more as an iterative approach and another tool in the toolbox. The webinars "almost" make it sound like this might become the "new" replacement for DOE, even though the presenters state that's not the case, there's still this underlying tone that suggests it's the next big thing. As with anything -- USE WITH CAUTION.

  One big takeaway I have is that with all computer assisted modeling (call it ML or AI, or gen-AI, whatever), supervision of the process is key. Unsupervised computer assisted modeling or generation can EASILY hallucinate. I've given several different AI platforms prompts to generate silhouettes of people doing different kinds of tasks -- pretty "simple", no detail, just a silhouette. Well, the different AIs easily hallucinate extra limbs on the humans -- three legs or 4 arms. Even after responding that there was a mistake in the AI's generation, the next iteration will still be a hallucination. The process still requires some pretty intense human supervision and oversight. Currently, the BO platform also seems to need some careful supervision.

  Just some healthy skepticism.

DS

[#6]

Hi DS,

Sorry for the misunderstanding, I did understand that the claims were from JMP Team, not you (sorry for any confusion in my answer).

You're right, initial starting points (spread and location) are as important as the choices you'll make after launching the BO loop. Depending on how close from an optimum the initial points are located, we can think that the algorithm may be "attracted" to this area, particularly if it is ok with your target and objective. To be able to force the exploration, adding runs with a space filling (MaxPro) criterion may help, as it will generate points that have the maximum distances with points already present (or adding runs with an acquisition function with an emphasis on uncertainty, like Multimodel Std Dev or Upper Confidence Bound). User decisions when using BO can be very impactful on the outcomes of the project.

Exactly, BO is another "smart experimentation" methodology in the toolbox. I also think that in practice, it may be more effective to have a few runs at each iteration (not only 1 run by iteration), and keep varying the type of runs added with different acquisition functions, to ensure diversity in the batch of runs and avoid premature exploration stop. We could for example think about 5 runs per iteration, with one or two exploitation runs, two exploration runs (or more), and other uns in a "balanced" mode. I see BO with acquisition functions as an option to create "tiny"-DoE experimentation type, where acquisition function are similar to design optimality, and where the "design" augmentation is highly flexible in terms of runs size. But it comes at the detriment of some of the statistical "machinery" and safeguards like blocking or other noise reduction techniques, so the two methodologies are not exactly interchangeable. I agree, "use with caution", start with the experimental settings and constraints, and THEN choose the experimentation methodology.

Yes, computer modeling like Machine Learning are great at finding patterns and can broaden the modeling options, but the results should always be considered with caution, as some algorithms may be sensitive to overfitting (or hallucinations for GenAI). So always use domain expertise, critical thinking and a statistical mindset to assess and evaluate the model's results. 

Thanks a lot for the discussion, I am really looking forward to see how the BayesOpt platform (and documentation !) will evolve in the future releases.

Best,

[#7]

Thanks for the interesting discussion. To overly simplify, I think this comes down to two fundamentally different approaches to experimentation.

One approach is algorithmic. The objective is simply to find settings that produce a desirable outcome. Whether the underlying mechanisms are understood is largely irrelevant as long as the algorithm reliably converges to a satisfactory solution.

The second approach treats experimentation as a learning process. The objective is not only to identify good settings, but to understand why they work. That mechanistic understanding allows us to predict performance in new situations, recognize when conditions have changed, transfer knowledge to future designs, and develop more robust products and processes.

Neither philosophy is inherently right or wrong—they simply optimize different objectives. If the only goal is to optimize a single system as quickly as possible, Bayesian Optimization can be extremely attractive. If the goal is to build scientific or engineering knowledge that extends beyond the current problem, then experimentation must be designed to reveal mechanisms, not simply search for an optimum.

My concern is that we sometimes evaluate these methods solely by asking, "Did we find a better solution?" I think an equally important question is, "What did we learn that will still be useful on the next project?" Those are not always the same thing.

Data can identify where an optimum exists. Understanding mechanisms explains why it exists. Only the latter allows us to predict what will happen when the world inevitably changes.

[#8 · Accepted solution]

Hello, thanks for your interest in the Bayesian Optimization in JMP Pro. I understand your confusion around the phrasing "Maximize MaxPro", JMP is actually maximizing the reciprocal of the MaxPro criterion. This was to make it so that the all the run acquisition options were presented as maximizations rather than some being minimizations while others are maximizations. A better phrasing in hindsight would have been "Maximize Spacefilling Criterion" and place the details in the documentation that we were maximizing the reciprocal of the MaxPro criterion.

[#9]

Thanks @chris_gotwalt1 !

I knew something was different (and possibly inversed) when looking at the acquisition function form of MaxPro criterion from the Bayesian Optimization profiler and the equation from the Space Filling design section. Thanks for clarifying it.

I do get the simplified way of maximizing any acquisition function, but the same naming for this criterion as for Space Filling design got me confused.

I hope the documentation will be revised and enriched to avoid any confusion and bring more information about this BayesOpt platform possibilities to users.

Best,