With this much data you should expect some points to lie outside the confidence interval (regardless of the %). If it is a mean confidence interval, indeed most points would like outside the interval. If they are individual predictions, then fewer should be outside the interval. Since the model is not perfect (it is only a model), of course some points will randomly vary outside the interval.
However, the graph you provide suggests something else is going on. There is an entire section of points outside the interval - this strongly suggests a lurking variable (likely a nominal variable) that is giving two groups of responses. So, conditional on x, y appears to fall into two distinct ranges. Failure to identify this lurking grouping variable means that the resulting confidence interval is too wide - and also, that at least one group will systematically (not randomly) lie outside the interval.