topic goodness of prediction in Discussions

goodness of prediction

Michal — Tue, 18 Sep 2018 14:44:18 GMT

Dear JMP experts,

would you advice me where can I find a Q square to compare goodness of prediction of two models constructed?

Eventually, What indicator do you use instead?

What does the "predicted RMSE" value under the graph in actual by predicted plot means?

thank you very much,

Michal

Re: goodness of prediction

Peter_Bartell — Wed, 19 Sep 2018 16:17:49 GMT

I try NOT to use a single statistic or visualization to answer the question "Which model predicts best?" That's falling victim to a dreade disease one of my statistics professors told me about called "mononumerosis".

But I look at many different statistics and visualizations...if I had to pick some must see visualizations then all roads lead to various types of residuals plots. JMP supports all the usual suspects so I suggest starting there. After all, these plots tell you alot about variation and performance that is explained and unexplained. And at the end of the day, I encourage you to consider the model that helps you solve your practical problem...so domain expertise is equally important if not more so, than the statistics and visualizations.

Re: goodness of prediction

Dan_Obermiller — Wed, 19 Sep 2018 16:29:19 GMT

I tend to agree with Peter's assessment. Understanding what makes a good model is like putting together a jigsaw puzzle. No single piece gives you the entire picture.

I would tell you where to find "Q Square", but honestly, I have never heard of that statistic. JMP offers several other statistics to help you assess the fit of a model like RSquare, RSquare Adjusted, AICc, BIC, etc. But again, no single number is really sufficient.

Finally, underneath the actual by predicted plot, JMP displays RMSE. It is not a "predicted RMSE". The predicted part is completing the labeling of the X-axis variable. Enclosed is a simple example (from a bad model) where they were trying to predict Claim Amount. Notice the labeling of the X-axis is "Claim Amount Predicted". The RMSE is just that: the Root Mean Square Error which you can see in the Summary of Fit table.

Re: goodness of prediction

Michal — Thu, 20 Sep 2018 07:50:04 GMT

Thank you very much for your answers! BR, Michal

Re: goodness of prediction

Michal — Thu, 20 Sep 2018 07:52:00 GMT

Thank you very much for the answer, M