The design is not orthogonal when there is even a single non-zero correlation. The question is not actually about the design matrix. It is about the model matrix that is used to estimate the parameters.
The question, "is the correlation too high," is still valid. The answer is that it depends on the variance of the response. The correlation of the estimates will inflate the variance of the estimates. This inflation reduces the power of the tests for significant estimates. It will increase the uncertainty in the predictions of the model. If your response exhibits slight variation, you can tolerate more considerable inflation due to correlation. If your response shows a substantial variation, you might be unable to take any inflation.