I've opened some pretty large files before in JMP, but 5GB seems a little outrageous. The 64-bit version of JMP might be able to handle it if you had enough RAM in your system, but 2GB is probably not enough even with the 64-bit JMP.
I recommend you break your tables up into more manageable sizes and open them when needed and close them when you're not using them. And don't forget to clear your table from memory between cycles because JMP doesn't do it automatically. Use the Clear Globals() function.