Try the Materials Informatics Toolkit, which is designed to easily handle SMILES data. This and other helpful add-ins are available in the JMP® Marketplace
JMP has powerful, world-class design of experiments (DOE) capabilities. See how to use the DOE platforms to make the most of your experiments. This topic may cover Custom Design, Constraints, Screening Design, and Split Plot Designs.
I would say that for more than 2 factors it is going to be too complicated to try something like the Fit Y by X approach. I would use the candidate set approach instead.
My feeling is that you are unlikely to be able to imagine constraints involving more than 3 factors (it makes my head hurt to think about it). Even 3 factors is unlikely IMO.