I need more details. What are the Factor A and Factor B levels? What are all the terms in your model?
A Discrete Numeric factor is modeled as contributing continuous effects. This factor choice means that you decide the levels and JMP decides the minimal set of terms in the model. So without details I was able to create a illustrative example. I added two continuous factors X1 and X2 with ranges -1 to +1. I added quadratic terms for both factors. I crossed X2 with X1*X1, like your case. I added a Disallowed Combinations script similar to what you want. Here is the design.
Here is the design space:
The constraint that I defined disallowed combinations in the middle of the space.
You still have not shared the contents of the Log when you get the error while making your design.