Bayesian Optimization GP vs standalone GP

[#1]

Hi,

I'm trying to understand the behavior of gaussian process models in the "gaussian process" vs "bayesian optimization" platforms (JMP Pro 19.0.1 )

Fitting these simple simulated (deterministic) data: 

With Gaussian Process:

with Bayesian Optimization

Why is the confidence region for the BO model not shrinking to 0 at the data points, like I see in the GP model?

thanks

Matteo

ps.

This is the formula for Y:

4.3 + -2 * (:X - 1) ^ 2 + Exp( -3 * (:X + 3) ^ 2 ) + 3 * Sin( :X + 2 )

[#2]

Hi @matteo_patelmo,

For the moment, the technical documentation for Bayesian Optimization is quite limited, and some technical details are not available regarding the calculation of confidence intervals/error intervals. There might be several aspects leading to the differences you see between the two platforms :

User inputs:

• The adequate use of the Continuous Correlation Type for your dataset (more correlation types are available in Bayesian Optimization than in GP (only Gaussian and Cubic)),
• The use of "Estimate Nugget Parameter" in GP, which enables the prediction model to smooth over the noise instead of perfectly interpolating. This option is not by default in classical GP, but the nugget parameter seems to be estimated by default in OB, which can increase the differences (but not in your example with a perfectly known formula).

JMP platforms inner workings:

• The fitting (and validation procedure) of classical GP vs. GP from OB.
  • In classical GP, the fitting and validation procedure rely on Jackknife method, where you have a Leave-One-Out procedure where you calculate your metrics on the samples that are kept (training samples), and then average the results for each LOO samples. 
  • In GP from OB, the fitting and validation procedure seems a little different and seems to rely on a classical LOO procedure (not 100% sure), where the metrics are calculated on the excluded samples (test samples). This could lead to an increase of uncertainty shown in the Profiler and different predicted values.
    You can see the fitting is different between the two platforms as if you're launching your dataset with the same correlation types and estimating the nugget parameter in both platforms, the values for theta, nugget parameter and GP variance are different.
• The calculation of confidence intervals : 
  In classical GP, the display of confidence intervals in GP is done using the variance formula of GP and the quantiles How to save prediction/confidence intervals from Gaussian Process model? The variance formula uses the theta value, and a distance between each point in the range to the closest point measured. Hence the "zero uncertainty" at the location of already measured points.
  The GP from OB provides a "Measurement Error" : "The Measurement Error is calculated as the product of the nugget parameter, τ, and the Gaussian Process variance, σ2". It seems the measurement error is also used, in order to avoid a "zero uncertainty" situation in the area where you have already sampled, in order to reflect experimental error.

Of course these interpretations have to be confirmed by JMP technical staff, these are just my impressions looking at the working of the two platforms and the outcomes at the moment.

Hope to get the inputs from JMP Technical staff to better answer your interesting question !

[#3]

Maybe somebody from JMP can respond here?

The confidence intervals are used to predict the best location for the subsequent measurement.
Different confidence -> different prediction, right?

[#4]

It's not that simple, as the behaviour of the two GP-based platforms are different regarding confidence intervals calculation.
You always have some error in the OB confidence intervals calculations, whereas in classical GP platform, error is set at 0 for the locations where points are measured (hence my initial response regarding the difference in error/interval calculation and display).

Depending on the acquisition function used and the calculation of confidence intervals, you can have different "optimal" next sampling point recommendation :

Maximize Bayesian Desirability :
 
Maximize Expected Improvement Desirability (Optimize predicted response regarding target):

Maximize Bayesian StDev Desirability (focus on where the uncertainty is highest):

[#5]

Ah, right!
Important to start with non-0 CIs even for measurement positions.
Huge difference to fitting simulation results.

Tricky - without a model - and with just a single data point at this spot.

[#6]

Thanks @Victor_G for your insights.  Let's wait for some clarification from JMP staff

Matteo

[#7]

Gaussian Process + nugget parameter:
How to save prediction/confidence intervals from Gaussian Process model? 

[#8]

@hogi , please read the conversation, this is the exact same link provided in my first answer.

Let's wait for an official explanation from JMP Technical staff working on Bayesian Optimization ( @chris_gotwalt1 ?)

[#9]

Ah, sorry, I did not follow the link at first place - as it was listed under classical GP.
You are right, same link. it also covers the part with the Nugget parameter.