cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
JMP is taking Discovery online, April 16 and 18. Register today and join us for interactive sessions featuring popular presentation topics, networking, and discussions with the experts.
Choose Language Hide Translation Bar
Nancy
Level II

Lack of fit test issue

Hi, I built a simple logistic model to predict loan chargeoffs. The variables included all appear to be significant and the model passed the whole model test. However, the model didn't pass the Lack Of Fit test. So my questions are: 1. Does it imply that some important variables or interactions are missing thus it is not a good mordel? 2. Can I still use the odds ratio/estimate to calculate the impact of each variable? I attached the results here. Any comments or suggestions are welcome and really appreciated. If you have any questions, please also let me know. Thanks!

 

1 ACCEPTED SOLUTION

Accepted Solutions
cwillden
Super User (Alumni)

Re: Lack of fit test issue

Failing Lack of Fit test means the model is not accurately capturing all of the trend apparent in the data.  In other words, you still have signal that's unaccounted for.  Adding more interactions or higher order terms often help.  Sometimes, you may have a hockey-stick effect with a plateau or something that you can't really get at with a regular polynomial model.  First step, I would add all possible 2-way interactions a squared term for interest rate and then check for lack of fit.  If it's still an issue, check back in here and we can examine what you got.  If the lack of fit issue is solved, you can trim back some of the unneeded model terms using your preferred model selection method.

-- Cameron Willden

View solution in original post

4 REPLIES 4
cwillden
Super User (Alumni)

Re: Lack of fit test issue

Failing Lack of Fit test means the model is not accurately capturing all of the trend apparent in the data.  In other words, you still have signal that's unaccounted for.  Adding more interactions or higher order terms often help.  Sometimes, you may have a hockey-stick effect with a plateau or something that you can't really get at with a regular polynomial model.  First step, I would add all possible 2-way interactions a squared term for interest rate and then check for lack of fit.  If it's still an issue, check back in here and we can examine what you got.  If the lack of fit issue is solved, you can trim back some of the unneeded model terms using your preferred model selection method.

-- Cameron Willden
Nancy
Level II

Re: Lack of fit test issue

Thank you so much! You suggestions helped a lot. I checked more variables and found that I missed some important ones, especially the "payment type". So I adjusted the model quite a lot. Now it passed the lack of fit test. However, I am not sure if I should keep "Risk Tiers". It is marked as unstable. If I remove it, the lack of fit test p value will drop from 0.971 to 0.817. Do you have any suggestion in case like this? Thank you!

cwillden
Super User (Alumni)

Re: Lack of fit test issue

Based on just the 2 lack of fit p-values, I would not hesitate to drop that effect. I tend to get unstable estimates with logistic regressions when I get too greedy with my model relative to the size and quality of the data set.
-- Cameron Willden
Nancy
Level II

Re: Lack of fit test issue

Thank you for sharing of your insights and the quick reply. I really appreciate it.