Hello @Shelsi,
I've been struggling myself with this issue. Something that worked out for me was caption boxes.
Instead of adding labels, you can go over your graph, do right-click, add, caption box. Then it will give you probably the general mean value on the right top corner (by default). You can then do again right-click, go caption box (there's an option after you've inserted it), and select the option per factor. That will show for every bar the mean value. In the same place, you can do all kinds of adjustments, eg place the caption box at the top, center, bottom, or even get other statistics like median. Just make sure there's enough space above your bars, by increasing the magnitude of your y-axis.
I hope this is helpful.