First I'd like to add my welcome to the community.
Mark and Pete have provided great feedback and I'm not sure I can add much, but here are my thoughts:
1. You mention instability a couple of times in your posts. What you are trying to do, I think, is optimization. This should only be done on stable processes as the optimization will be different under different "levels of stability". Perhaps you should experiment (or sample) to understand why there is instability first? Also the level setting suggests you are still in the "understanding causal structure" space, not in the optimization space (you should already know the first order model). This is akin to mapping the base of the mountain. First get to the top of the mountain, then map it.
2. It seems like you are in the world of mixtures where there is a constraint on the % of each component in the "batch"? As you increase the amount of 1 component in the mixture, you will have to decrease the amount of another...they have to add up to 100%
3. You must be aware of what "creates" the MSE estimates when interpreting p-values. In your experiment there are no degrees of freedom to estimate random errors. You have no replication, so the MSE is a function of higher order effects in your design (e.g., quadratic-by-linear interactions). If these are active, then your MSE will be large and your F-values will be small and your p-values will be large. If you run several of the "identical" trials from your first experiment again, what do you get?
4. The variability in your data explained by your model is quite low (and your residuals look "funky"). This is a clue there is noise (variables unaccounted for in your model) that may be accounting for the variation. This could be measurement error, ambient conditions, set-up, raw material variation, etc.
"All models are wrong, some are useful" G.E.P. Box