Visible and invisible tables are tracked by JMP. You can keep an eye on them in the home window, for example. Private tables are tracked only by the code that creates them. If you fail to close a private table before overwriting the JSL variable holding its handle, the table is still in memory but unreachable. It will be closed when JMP terminates. (Window->CloseAllDataTables can't find the private tables and won't close them.)
I'm also pretty sure JMP does not expose a way to turn a visible or invisible table private. You can turn a private table to an invisible one, if you have the handle, like this:
dtA = Open( "$SAMPLE_DATA/Big Class.jmp", private );
// getting the data box makes it invisible
b = dtA << New Data Box();
// then show window makes it visible
dtA << Show Window;
Unfortunately, the invisible keyword is a keyword and can't be passed as a variable, at least to open(). Not sure about subset, etc. Probably the best you can do is add a call to a user function that does something like above after each point where you create or open a table.
Craige