topic Re: How to minimize the size of an integer table in numeric format? in Discussions

How to minimize the size of an integer table in numeric format?

lwx228 — Fri, 09 Jun 2023 18:10:15 GMT

The tables need to save are all integer numbers.

Thanks!

dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
dt << delete columns( 1 :: 3 );
New Column( "A" );
Column( "A" ) << Formula( 1000 + Row() );
dt << run formulas;
Column( "A" ) << deleteFormula;

Re: How to minimize the size of an integer table in numeric format?

Craige_Hales — Tue, 08 Feb 2022 12:59:08 GMT

Minimize in memory, or on disk?

In memory: use a short numeric format, numeric(1), numeric(2), or numeric(4). These specify a signed integer of the same number of bytes with some reserved values for missing. They are all smaller than a double (8 bytes) which can represent integers up to ~50 bits...a much larger range than the 8/16/32 bit range. Also saves space on disk. (I don't think it is required, but a recent comment in the community said the pref for short numerics has to be on to use them. After turn on pref, look at the available numeric types when creating columns and look at the JSL that gets saved. The ranges are roughly +/-126, +/-32000, +/-2e9.)

On disk: use compression. Two approaches, either use the red triangle to do a zip-like compression of all columns, or use the compress-columns utility that looks to see if the column has a limited number of values that could be represented as a lookup. The zip approach is slow when saving, faster when decompressing, but not as fast as loading the uncompressed data.

You can use a numeric(1 or 2 or 4) and zip the file to good effect if there is still some redundancy in the data that zip can squeeze out. But it is unzipped in memory so doesn't help there.

Re: How to minimize the size of an integer table in numeric format?

lwx228 — Tue, 08 Feb 2022 13:40:04 GMT

Thank Craige!

I need to reduce the size of the file stored on the hard disk.But I don't know how to do that with JSL.

Re: How to minimize the size of an integer table in numeric format?

Craige_Hales — Tue, 08 Feb 2022 13:59:16 GMT

I made a file with the GUI, set the column to numeric(1) and used the red triangle to request compression on disk. Then I grabbed the script.

New Table( "Untitled",
    Add Rows( 0 ),
    Compress File When Saved( 1 ),
    New Column( "Column 1", Numeric( 1 ), "Continuous", Set Selected )
)

Re: How to minimize the size of an integer table in numeric format?

lwx228 — Tue, 08 Feb 2022 14:33:37 GMT

Thanks Experts!

Re: How to minimize the size of an integer table in numeric format?

guillaumebugnon — Tue, 08 Feb 2022 16:17:42 GMT

Hello,

You can set this preference and after by default all tables will be compressed.

or by scripting as mentioned by @Craige_Hales

Compress File When Saved( 1 );

You also could compress Column by using

Compress Selected Columns( );

Re: How to minimize the size of an integer table in numeric format?

lwx228 — Wed, 09 Feb 2022 07:58:29 GMT

It's amazing how much the file size is reduced.