cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
JMP is taking Discovery online, April 16 and 18. Register today and join us for interactive sessions featuring popular presentation topics, networking, and discussions with the experts.
Choose Language Hide Translation Bar
dale_lehman
Level VII

Profiler discussion continued

Recent discussion about how to interpret standard errors from the profiler have not converged to a solution, so I will start this again - with a different example and some more clarity (I hope) on my question.  I've attached one of the famous Anscombe quartet datasets - the one where a linear regression model is appropriate.  I fit a multiple regression model and saved the mean prediction standard error and the individual prediction standard error - as expected, the former is much narrower than the latter.  I also used the profiler and saved the bagged predictions - this gave me a standard error for the bagged mean (which I interpret to be analogous to the mean prediction standard error from the regression model) and a Bagged Std dev (which I interpret as analogous to the individual prediction standard error from the regression).  When I say I interpret these to be analogous, I realize that the standard errors from the model are derived from the regression model while the bagged standard errors are derived from a non-parametric bootstrap (in this case, with 100 bootstrap samples).

 

In this case (unlike my last example), the individual standard errors are close, although somewhat smaller for the bagged predictions than the model predictions.  But the standard error of the bagged mean is around 10% as large as the standard error of mean predictions from the model.  In fact, this can be verified by the documentation that says that the bagged standard error = standard deviation of the bagged predictions divided by the square root of (M-1) where M is the number of bagged samples (which is 100 here).

 

My question concerns how to use the standard errors of the bagged mean and the bagged standard deviation.  If I look at the confidence interval shown in the profiler, it appears to me to be consistent with the mean prediction standard error from the regression model - not with the standard error of the bagged mean (which is 10% as large).  If that is the case, then what are the standard errors from the bagged samples good for?

 

I think the importance (for me at least) is that JMP provides profilers for all the machine learning models.  This allows saving the bagged means and these two types of standard errors.  If these standard errors can be used to construct confidence intervals, then we get something akin to a sense of statistical significance from neural networks, random forests, etc.  I think that would be an important feature since most of those models don't readily provide any inferential information.  This would apply to classification models as well as regression models.

 

But I am not sure the bagged standard errors make much sense.  In the example I've provided, they look too small.  And, it isn't clear to me whether the confidence interval shown in the profiler is based on the standard error fo the bagged mean (it doesn't look that way in this example).  So, please help shed some light on my questions.

 

1 REPLY 1
dale_lehman
Level VII

Re: Profiler discussion continued

Nobody seems to have been able to shed any light on this subject.  Based on some more exploration, let me suggest a "solution" and perhaps someone can weigh on that.  It looks to me like the confidence intervals that are produced from the profiler after saving the bagged predictions, use the Standard Error of the Bagged Mean rather than the Bagged Standard Deviation.  I am starting to believe that that is incorrect - that the latter should be used rather than the former.  The difference is that the former is the latter divided by the square root of the number of bagged samples (-1).  It is true that the Standard Error of the Bagged Mean measures something - I think it is the stability of the bagging procedure (in other words, how sensitive the mean prediction is to the bootstrap samples).  But I think what we want from the confidence intervals is the reliability of the prediction and the confidence in how much each factor in the model affects that prediction.  I think it is the Bagged Standard Deviation that measures that.