"Stability" of RMSE across validation/training/test sets
Jan 23, 2019 1:09 PM(230 views)
Is there a good common practice of saying that RMSE (or whatever model parameter) are "stable" across the three sets. Is there any rule of thumb stuff? Within 10% of each other, etc?
Currently I'm just taking another RMSE vs the mean for the points, but don't know if that would be a valid metric or if there's already somethign out there that accomplishes this. This assumes the same dataset.
Sorry, Vince. I'm not aware of any common practices on this. I agree it seems like a really good idea. However, I think it will depend on a lot of variables. For example, data size: as data size increases I would expect the "best" models to be more stable across validation sets. It sounds like this could be an interesting MSc project for someone to do some simulations and see if they find something useful.