I'll start my thoughts with the answer to your last question. Yes there are limits with regard to the computational power of the PC being used. For JMP system requirements I suggest reading here: JMP System Requirements Some of this is also predicated on the version of JMP you are using. In general, these requirements are either binary (must have for JMP to function at all) or 'the more the merrier', with some recommended minimum, such as RAM. There is a highly recommended minimum...but if you've got more, JMP will perform faster, with fewer headaches for the user.
Now some general thoughts on '...the best way to design such an experiment in JMP?' question:
1. What is the overall goal of the problem at hand? Screening 'categories' which I'll call factors from here on out? Optimizing the response? Maybe if it's both start with a screening style experiment...then move sequentially to optimization type experimentation?
2. Are there restrictions on randomization over and above those you've articulated?
3. Are the factors continuous? And does placing any factor at 'absent' make the whole system behave in a way that is atypical wrt to the response? In my experience with chemical based systems in experimentation sometimes complete absence induces a very nonlinear effect (think step change as an example) in the response for the factor, making modeling for optimization purposes more problematic.
4. Is this truly a DOE mixture experiment where the sum of the factor levels must equal 100%?
5. This is clearly a problem for JMP's Custom Design platform...so in general I think you'll start there.