topic Re: Predicted Rsquare in Discussions

Predicted Rsquare

CYLiaw — Fri, 22 Nov 2019 19:42:02 GMT

I wanted to use predicted Rsquare to test if my model is overfitting or not. I am not really familiar with this, so I have a few questions regarding predicted Rsquare.

1. How much smaller for the predicted Rsquare is a sign of overfitting? If the Adjusted R-square is 0.94, and the predicted R-square is 0.84, is it okay?

2. I don't fully understand how the predicted Rsquare was calcuated. I know that it takes one data point out each time, get a regression model, and put that data point back and get a R-square. It repeated for all data points and average the obtained R-squares. But how to get those regression models ? Does JMP use machine learning approach to obtain the model? Are those regression models different from the model I choose?

3. I found that it is not always true to say that the predicted R-square will drop more if there are more factors in the model. I found that, for example, the model with 2 factors can have a lower predicted R-square than a model with 3 factors (although the third factor has a p value (much) bigger than 0.05). In this case, should I include 2 or 3 factors in the model?

Re: Predicted Rsquare

Mark_Bailey — Sat, 23 Nov 2019 16:54:11 GMT

You should not use the R square for model selection. It always increases when you make a model more complex (e.g., add a term to the linear predictor in regression). It always decreases when you make a model less complex.

Using cross-validation, though, can make the R square more useful for model selection. You might expect that the validation R square would not increase as you over-fit the data. Specifically:

There is no way to establish how much of a difference between the training R square and the validation R square indicates over-fitting. It is a subjective decision. One can say that the model for which the two R square estimates over-fitting the least of all the candidate models.
There is an efficient computation of the 'leave one out' statistics using the 'hat' or 'projection' matrix.
I would not use the change in R square to select the model. I would consider other information but mostly pay attention to a criterion such as AICc or BIC.

Re: Predicted Rsquare

statman — Sat, 23 Nov 2019 17:13:01 GMT

Just to add to Mark's comments, one of the methods to determine model over specification is to use the delta between the R-square and R-square adjusted. R-quares increase as the number of degrees of freedom in the model increase (regardless of whether those DF's are important). R-square adjusted takes into account the "importance" of the DF's in the model, so adding unimportant degrees of freedom to the model, the delta will increase (the R-square adjusted will not increase at the rate of the R-square).