topic Re: Lack of fit test issue in Discussions

Lack of fit test issue

Nancy — Tue, 30 Apr 2019 20:56:18 GMT

Hi, I built a simple logistic model to predict loan chargeoffs. The variables included all appear to be significant and the model passed the whole model test. However, the model didn't pass the Lack Of Fit test. So my questions are: 1. Does it imply that some important variables or interactions are missing thus it is not a good mordel? 2. Can I still use the odds ratio/estimate to calculate the impact of each variable? I attached the results here. Any comments or suggestions are welcome and really appreciated. If you have any questions, please also let me know. Thanks!

Re: Lack of fit test issue

cwillden — Tue, 30 Apr 2019 21:54:09 GMT

Failing Lack of Fit test means the model is not accurately capturing all of the trend apparent in the data. In other words, you still have signal that's unaccounted for. Adding more interactions or higher order terms often help. Sometimes, you may have a hockey-stick effect with a plateau or something that you can't really get at with a regular polynomial model. First step, I would add all possible 2-way interactions a squared term for interest rate and then check for lack of fit. If it's still an issue, check back in here and we can examine what you got. If the lack of fit issue is solved, you can trim back some of the unneeded model terms using your preferred model selection method.

Re: Lack of fit test issue

Nancy — Tue, 30 Apr 2019 22:42:28 GMT

Thank you so much! You suggestions helped a lot. I checked more variables and found that I missed some important ones, especially the "payment type". So I adjusted the model quite a lot. Now it passed the lack of fit test. However, I am not sure if I should keep "Risk Tiers". It is marked as unstable. If I remove it, the lack of fit test p value will drop from 0.971 to 0.817. Do you have any suggestion in case like this? Thank you!

Re: Lack of fit test issue

cwillden — Tue, 30 Apr 2019 23:00:03 GMT

Based on just the 2 lack of fit p-values, I would not hesitate to drop that effect. I tend to get unstable estimates with logistic regressions when I get too greedy with my model relative to the size and quality of the data set.

Re: Lack of fit test issue

Nancy — Tue, 30 Apr 2019 23:07:54 GMT

Thank you for sharing of your insights and the quick reply. I really appreciate it.