All possible models multiple regression - assessing R2 change between models

[#1]

Hi All,

Relatively new to JMP. I have performed an all subsets multiple regression as I am attempting to find the best model of IVs that predicts my DV. Based on AICc, BIC and Mallows there is a model that appears best. However, there is another model that has one additional variable that has similar AICc etc (though worse), but a better adjusted R2. Is there a way in JMP of statistically comparing the change in R2 between the two models ("R2 change"), as can be done with SPSS? I want to assess if this increase in R2 is statistically significant. I suspect it is not and I should stick with the less complicated model. 

Many thanks,

Phil

[#2]

Hi @philc86 , and welcome!  If the difference between models is just one of the potential variables (as you describe), then the p-value for that additional variable is what you are after (same as testing for a significant increase in R^2). And, in general, see the following:

https://www.stat.uchicago.edu/~yibi/teaching/stat224/L04.pdf 

[#3]

Hi MRB3855,

Thank you for this. Yes, I want to test if there is a significant increase in R2 statistically speaking on adding new variables. In SPSS I can do this by using hierarchical regression (adding all but the additional variable to block 1, then adding the additional variable in block 2). I can't seem to see how to do this in JMP however. Do you know how to do this? I would rather keep to using just the one statistics program. 

Phil 

[#4]

Hi @philc86 : For more than one additional variable, I don't see a way to do it directly in JMP (unless I'm missing something). The article I provided a link for describes how to do it, but you will need to run the two models (full and reduced) to get the pieces of information you need (see pages 31 to 33) to make the calculations yourself..  If you'd like to see that capability in a future version of JMP, you can add it to the JMP "Wish List". https://community.jmp.com/t5/JMP-Wish-List/idb-p/jmp-wish-list

And here is someone who has made a video on how to do it..

 https://www.youtube.com/watch?v=OwXHkoLOse8

[#5]

Welcome in the Community @philc86 ! 

Hi @MRB3855,

Not directly related to your suggestions or @philc86 initial demand, but if conditions to use the platform The Fit Two Level Screening Platform are met (Overview of the Fit Two Level Screening Platform (jmp.com)), it could help identify important active terms and help model(s) selection. Note that effects are entered into the analysis following a hierarchical ordering.

This platform can be found in DoE, Classical, Factor Screening, Fit Two Level Screening.

Hope this may help the discussion,

[#6]

Hi @philc86, 

One way you can do this in JMP Pro is by using the SEM platform; it allows you to specify the full model, the restricted model, and then gives you a statistical test of the difference. This test is a chi-square but it'll lead to the same inference as the R^2 in hierarchical regression. 

Indeed, a multiple regression model can be fit in the SEM with the same results as the Fit Model platform. One particular advantage of doing this analysis in SEM is that it handles missing data with a cutting-edge algorithm by default, so no data are thrown away leading to increased power.

There's a bit of a learning curve if you've never used SEM before, however, I'm including a couple of JSL scripts here in case they help:

1) Running the script in "Compare_FitModel_to_SEM.jsl" will show you the results of a multiple regression using the Fit Model platform and the SEM platform. You can look at the output side by side to see the same results:

2) The script "Hierarchical_Regression_in_SEM.jsl" fits two models in the same SEM report; the full model, which has all the variables predicting the outcome, and the restricted model, which has all but one (or more) variables without an effect on the outcome. In the SEM path diagram, a regression effect is depicted as an arrow, so you can see the full model has all the effects:

and the restricted model has one effect fixed to zero:

In this example, the chi-square difference test shows a statistically significant amount of misfit induced by fixing the effect of one variable to zero:

Thus, we conclude that the full model is the better model.

One possible disadvantage of doing this in SEM is that the variances and covariances of the predictors are all estimated as part of the model, and thus, you might need a reasonably large sample size for all to work properly. Also, if you have categorical predictors, they need to be entered as numeric binary variables (like dummy-coded variables in multiple regression).

HTH,

~Laura