You can only assign one data column to the X, Predictor Formula role. The formula, though, can use more than one predictor variable.
Yes, I made a new data column with a formula to define the custom model for Nonlinear. Please select Help > Books > Using JMP and see the Calculate Values with Formulas section in the Work with Your Data chapter devoted to building formulas. Then see the later Formula Editor chapter for more details. Lots of good information and examples there!
There are also more examples specifically of using column formulas for custom models in Nonlinear in the chapter devoted to this platform in Help > Books > Predictive and Specialized Modeling book.