It sounds to me like you need more memory in your computer, but some more information might be helpful.
What type of join are you trying to perform? How big a table (rows and columns) are you trying to join to how big a table? How much memory do you currently have? What version of JMP? What OS?
I would also recommend submitting your question to JMP Support.
If in fact insufficient RAM is the culprit, one technique you might want to try, if you are running JMP 13, is JMP 13's Virtual Join capability. One of the attractive design features of this capability is to reduce the memory footprint of two or more JMP data tables that are required for some analysis purpose. Here's a link to the JMP online documentation for the Virtual Join capability: