How to save prediction/confidence intervals from Gaussian Process model?

[#1]

Hi there, I can easily save the prediction formula and jackknife predicted values from the Gaussian Process model report window, however, there is no option to save confidence or prediction intervals as with other models. Is there an easy way to do this?

My ultimate goal is to produce a grid of contour plots across different levels of my variables at the lower 95% interval. Confidence intervals are displayed in the prediction profiler, but this does not give me a good overview of all the factor settings that achieve my desired response.

Many thanks!

Joff

[#2 · Accepted solution]

Hi @JoffJones ,

If you fit using the Gaussian correlation function, you can calculate the 95% confidence interval based on the following procedures.
1. Save the prediction formula and the variance formula from the red triangle.
2. Create a new column with the following formula (Please replace column name according to your data.).

• 95% upper confidence interval:

:Y Prediction Formula  + Normal Quantile( 0.975 ) *
Root( :Y Variance Formula  )

• 95% lower confidence interval:

:Y Prediction Formula  - Normal Quantile( 0.975 ) *
Root( :Y Variance Formula  )

Please note that sometimes I have encountered the situation that the confidence interval calculated using the formula does not match when I fit using the cubic correlation function.

Hope it helps.

[#3]

Hi @yuichi_katsumur 

Thank you for the suggestion. The problem I am having is that when I 'Save the Variance Formula', the values are either 0 or virtually zero. This doesn't reflect the variability that is shown in the confidence intervals on the Prediction Profiler - where the standard errors are between around 0.05 to 0.1.

Any ideas why?

[#4]

Hi @JoffJones ,

Are you using Cubic as Corretion type? If so, you may need to slightly change the formula for the variance calculation. If possible, could you please attach some dummy data that has zero variance?

[#5 · Accepted solution]

Hi @JoffJones @yuichi_katsumur . A comment that may be of some help here. If the nugget effect is zero, the prediction variance at your observed data points will be zero. The variance will be > 0 everywhere else. This reflects the idea that this is often called “spatial interpolation”; i.e., observed vs predicted will fit exactly…and to put it in OLS (ordinary least squares) language, R^2 = 1.

[#6]

Hi @MRB3855 ,

That explains it, thank you. I hadn't opted to 'estimate nugget parameter' but this does need to be added because there is noise in the system (i.e. it is not a computer simulation). When I select this option and save the prediction variance, I get non-zero values.

[#7]

Hi @yuichi_katsumur @MRB3855 ,

The solution now works to generate confidence intervals, thank you. I am wondering now if it is possible to generate prediction intervals for an individual run - do you know how to do this?

Many thanks,

Joff

[#8]

Hi @JoffJones ,

It's a little hard for me to explain what a Gaussian process confidence interval means.

The predicted value in a Gaussian process regression is the mean of the conditional distribution based on the mean and covariance matrix computed with the training data. Also, the 95% confidence interval is the region that contains the 95th percentile in the conditional distribution given the training data. It's important to note that Gaussian processes do not account for variation around the mean prediction like ordinary least squares regression does.

The interval in the Gaussian process can be close to a "predictive distribution" in the sense that the conditional distribution is calculated. On the other hand, Y will always have the same value for the same X (if there are no nugets effects), and there is no variation in individual observations for the same X. In this sense, it can be close to a "confidence interval around the mean".

If you have information on how to calculate the interval you want, please let us know.

[#9]

Hi @yuichi_katsumur 

Thank you for your reply. 

For some background, the data is from an empirical experiment with measurement error and so we also estimated the nugget parameter (see report attached).

What I would like to calculate is a lower 95% prediction interval, which tells me with 95% confidence that the next run of the experiment with the settings X will be higher than this value. Is that possible?

[#10]

Hi @JoffJones ,

Thanks for your explanation. I am not sure if the formula is correct, but you can try changing the formula as follows.
- Add nugget values to the variance formula you saved. The formula is complex, so be careful where you add the nuggets.

The original formula looks like this.

Maximum(
	sigma * (1 - Vec Quadratic( [n x n], Exp( aaa ) || Exp( bbb ) || Exp( ccc )) ),
	0
)

The modified formula looks like this.

Maximum(
	sigma * (1 - Vec Quadratic( [n x n], Exp( aaa ) || Exp( bbb ) || Exp( ccc )) + nugget) ,
	0
)

- Calculate the interval as a confidence interval
See the attached data for more details.
I checked with sample data and it seems to be a reasonable interval. I hope this helps.