JMP Wish List
We want to hear your ideas for improving JMP. Share them here.What inspired this wish list request?
This help page (https://www.jmp.com/support/help/en/19.1/#page/jmp/example-of-confidence-intervals-using-bagging.sht...) describes a method of using bagging to produce confidence intervals for a neural network predictive model. I think this can easily be extended to prediction intervals by using the RASE as an estimate of a normal error term. Also, the new profiler visualization of data points and prediction intervals can be used with this prediction interval implementation. This would allow one to visually check their model for coverage of data points. JMP currently only model checks on the point prediction of the model. This would help one make sure the error term is also correctly estimated.
What is the improvement you would like to see?
Here is a visual of the output of the confidence intervals from bagging:
I calculated prediction intervals using the RASE and normal error, then manipulated the column properties to use the new prediction interval visualization from JMP 18 to produce a plot like this. Basically, I'd like the default confidence interval profiler generation from the neural network bagging to generate a plot like the only below instead of the one above. I attached a jmp table of the tiretread example from the help page so you can see my calculations and implementation. The first script "Bagging Profiler", is identical to that from the help page. The second profiler, "Profiler of Pred Formula ABRASION", is my custom profiler with the addition of data points and prediction intervals.
Now that the prediction intervals have been calculated, I can check for data point coverage both visually in the profiler and numerically in the table.
Why is this idea important?
It is extremely strange to me that neural networks and similar predictive black box models are meant to be good at prediction, but they more or less ignore uncertainty. I always wanted prediction intervals to see if the model, including the error term, mimics the existing and new data well. The bagging to produce confidence intervals was a step in the right direction, but you still can't validate the model by comparing the data to the mean confidence intervals. You need to compare the data to prediction intervals. This approach essentially gives you a method to check the validity of the error term itself, and compare the actual data to the model's prediction intervals to make sure most points fall within. All of this will enable another method of model checking, and instill more confidence in the model if the fit appears to be good and the prediction interval coverage looks good. What is great about this is that is should be able to be used for most types of models that typically do not possess confidence or prediction intervals, like neural networks.