how to optimize the cpu/memory usage in jmp on Windows

[#1]

hi, I often have to handle data tables with few million rows and 20s to 30s columns. The frustration is whenever I need to edit a plot basing on this big table, the system start the waiting sign(turning ring) which take 1 to few minutes. It normally consume ~60% CPU and few GB memory if I look at the "task manager". Luckily I am using 64bit jmp on 64 bit Win7 workstation which has enough resources but I still have to wait and waste time on waiting.

I hope there is an option/method to optimize the cpu/memory usage or just hold the refreshing till user let it to do so.

Thanks!

[#2]

The simplest thing is to work with a random subset of your data while you're tuning a plot appearance, then save the script and reapply it to your full data table. Otherwise, strategies probably depend on the specific platform and task you're doing.

[#3]

thanks, I will try to use small subset of my data to work out the script then.

[#4 · Accepted solution]

Compress Selected Columns

JMP lets you compress columns in a data table to minimize the size of the file and reduce the

amount of memory required to analyze data. This feature is helpful when numeric columns

contain many small integers or when any column contains fewer than 255 unique values. For

example, compressing columns in a data table with 389 columns and 85,000 rows might

decrease the file size from 250MB to 33MB, depending on the type of data.

dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
dt << Compress Selected Columns(
{:age, :sex, :height, :weight}
);

[#5]

Can you import a csv file and apply the compression initially before already importing it into memory?

[#6]

CSV import does have the column compression as an option (off by default):

It will choose the best of several alternatives after scanning the data.

JMP has two kinds of compression for data tables. This one is the column compression and works by noticing a limited set of distinct values and using one of JMP's alternate column formats. This will make the in-memory footprint smaller.

The other kind of compression uses a zip-like mechanism to compress columns. This can be slow to open and even slower to save, and offers no in-memory benefit...only on disk. It is on the red triangle, something like compress when saved. Particularly helpful for huge tables of repeating values.

Both can be used together.