topic Re: Mean / Individual Confidence Interval in Mixed Models for new data points in Discussions

Mean / Individual Confidence Interval in Mixed Models for new data points

Anja_W — Thu, 08 Jun 2023 21:15:01 GMT

Hi everyone,

I fit a mixed model with one random effect in JMP 17. I would like to get confidence intervals for new predictions, i.e. data points which where not used in the model fitting. However, if I go to the red triangle in the fit model platform and use "Save Columns -> Mean/Individual Confidence Interval", it just gives values for the data points used in the model fitting. Is there some way in JMP or JSL to get confidence intervals for new data points?

If I use models for predictions I am also interested in the uncertainty of this prediction.

I would appreciate any help!

Best,

Anja

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Wed, 10 May 2023 08:18:29 GMT

Hi Anja: If you add new rows to your data set, with the data point values you want to predict at, but leave the "Y" as missing ( . ). Then it will give you what you are looking for.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

Anja_W — Wed, 10 May 2023 08:44:09 GMT

Thanks for your quick reply.

Unfortunately, it doesn't. That is exactly what I tried. It just gives the Confidence Intervals for data points where "Y" is not missing and was used in the model fitting.

What version of JMP do you use? I use JMP Pro version 17.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Wed, 10 May 2023 10:21:42 GMT

Hmm. Are you sure you have all of your predictors in the added columns (with Y missing)? That means values for all factors (fixed, random, whatever) that are in your model (categorical and numerical), and any "by" variable you may have used. I use JMP V17, but the fact that you use Pro should not matter.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Wed, 10 May 2023 10:47:03 GMT

And @Anja_W ...you do have to be very careful here about what exactly you want. Do you want intervals for a certain combination of factor levels (random and fixed) that you used to build your model? Or, do you want intervals that are based on total variability of the data, and not for any particular level of the random effect(s)? If you want the former, then it should work as I described earlier. If you want the latter, then that is an entirely different animal and I'll point you here.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6916346/

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

Anja_W — Wed, 10 May 2023 11:25:28 GMT

Thank you very much. I forgot to add a value to the random factor since the prediction is supposed to be for a new group which I don't know yet. It works if I insert an already existing level of the random factor but actually does not depend on it.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

StefanC — Wed, 10 May 2023 13:46:33 GMT

I have been following this conversation with interest. As far as I can see prediction intervals for both the situations you describe can be calculated in JMP. If you want a prediction interval for a certain level of a random effect you choose

Save Columns --> Conditional Indiv CI

If you want a prediction interval where the random effect variability is included you choose

Save Columns --> Indiv Confidence Interval

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

Anja_W — Thu, 11 May 2023 07:19:20 GMT

Exactly.

But also for the latter "Indiv Confidence Interval" you need to provide a value for the random factor. This is what puzzled me. But the output will be the same no matter what value is provided.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Thu, 11 May 2023 07:54:08 GMT

@Anja_W and @StefanC . Points taken; it does appear this version of JMP has that option, which is an improvement (and perhaps I missed it in previous versions). That said, there is precious little information (that I can find) wrt how JMP actually calculates this "Indiv Confidence Interval". And the research in this area indicates that many of the "standard" methods out there for such intervals are incomplete and in need of improvement; for a summary of the issue relative to JMP and other software, see section 3.5 here ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6916346/ ). Therein, wrt JMP, it says:

"In JMP, the “Fit Model” platform allows to run a mixed model and the PIs can be obtained from the “individual CI” option in the “Save Columns” menu. There is no formula and no discussion in the help regarding the PI and its degrees of freedom. One can actually notice that the degrees of freedom for fixed effects (ie, Kenward-Roger) are used for the PI. This is not appropriate as explained in Section 3.2 with PIs too wide (see next section)."

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Thu, 11 May 2023 08:40:10 GMT

I've dug into this further. I calculated prediction intervals using the examples in sections 5.2 and 5.3 (in the link I provided, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6916346/) via JMP and via the method outlined therein. And, much to my surprise...they match! This is a wonderful improvement indeed. Thanks JMP for this update, and thanks @Anja_W and @StefanC for working together to tease this out.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

StefanC — Thu, 11 May 2023 09:17:08 GMT

@MRB3855 I know that PI's calculated in JMP used to be way too conservative for certain designs. But it appears this has been improved in recent versions. But like you say information regarding the calculation method is missing.

Great article by the way.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

StefanC — Thu, 11 May 2023 09:25:19 GMT

Just saw your post. That is indeed an improvement. Thanks for the great work!

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

eclaassen — Thu, 11 May 2023 16:47:35 GMT

The paper you referenced was correct when it was written; the DFs being used for Individual Confidence Intervals (ie Prediction Intervals) when a mixed model was fit were originally defaulting to the K-R DFs for a usual Confidence Interval. This was corrected (using the paper you referenced as a primary reference) as of JMP & JMP Pro 16. The Individual CI option was not available in the Mixed personality (the JMP Pro personality for fitting mixed models) prior to that version, though it was available if you fit a simple mixed model using REML under the Standard Least Squares personality. I can request that the Documentation team update the doc with more details, or at least point to that article for the math.

As for needing to specify a random effect level despite it not being used in the calculation of an Individual CI, that is a result of the typical behavior of omitting a row from all calculations if an X (whether fixed or random) has a missing value. With a random effect level specified with the Y value missing, the row isn't used to fit the model, but it is still in the data for all other calculations.

Re: Mean / Individual Confidence Interval in Mixed Models for new data points

MRB3855 — Fri, 12 May 2023 12:54:06 GMT

Thanks @eclaassen for the background and context...and the improvement! It’s sometimes hard to keep up with what’s new in the latest version, especially with what may be considered “minor” (but significant in my world!) updates like this. Indeed, it would be helpful to have some detail and/reference in the documentation.