There is something strange about your models. Look at the misclassification rates. Where "5" = 0 there is almost no misclassification. Where "5" = 1 there is almost complete misclassification. You would have to share more about what you have done in order to work out what the problem is.