How to use JSL to make TXT files of decimal data into binary format files?

[#1]

Decimal files use recent tabs to divide data into three columns.
They are all integers:
The data in the first column ranges from 90,000 to 199,999
Column 2: positive integers from 100 to 900,000
Third column: positive and negative integers or 0

They have a lot of rows.
There is no limit to how to save them as binary files or file suffixes using JSL.
And   need to figure out how to re-read restored decimal data from this new binary into JMP.
Thank you very much!

[#2]

I want to reduce the size of the file in this way.

[#3 · Accepted solution]

Open the preferences and look for the option to allow short integer columns. Create a data table with the data and use four bytes for each of your columns (available in the column dialog after you turn on the option.)

Using a 4-byte integer

I forget the exact rules, but approximately:

• 1-byte columns represent +/- 1e2 and missing,
• 2-byte +/- 3e4 and missing and
• 4-byte represent +/- 2e9 and missing.

Optional: turn on the whole table compression before saving. That will use a zip-like compression that might compress even more, but only on disk, at the expense of slower saving and loading due to compression.

Using 4-byte integers will be half the size of 8-byte doubles. I think that will be true even after loading in memory. Turing on compression for the 8-byte doubles is likely to make the file almost as small as the compressed integer file because it should find the same redundancies to squeeze out.

[#4]

Thank Craige!

I have formatted each column like this, but still can't figure out how to save the binary file to the computer's hard drive?

[#5]

The JMP data table is a binary file. Save it in any of the usual ways (file->save, <<save) as a .JMP extension.

Yes, there is some meta data in the file, but if you have 1,000,000 rows of three 4-byte numbers, 12,000,000 bytes, the overhead of the meta data will be about ( 752 bytes / 12,000,000 bytes ), or < .01% overhead. In exchange, you 

• Can load the file for free
• Don't have to worry about remembering the column names
• Can share it with other JMP users
• Can add more columns if you need to, even mix and match 8-byte doubles
• Don't need to invent another file extension (.dat, .bin, .myformat  ?)

If you need to share the data with non-JMP users, a compressed CSV will be pretty small and very portable.

All that said, if you must make a binary file of some other format, see JSL BLOB in an ESP32 Clock  for an example using matrixToBlob. That file is holding highly structured data with self-contained offsets within itself to represent run-length encoded data for a bitmap, connected to a lookup table for each pixel's value. It was hard to get correct and hard to test and hard to use, but gave me the results I needed.

[#6]

I SEE.

Thank Craige!

[#7]

How to format these columns to 4-byte INTEGER using JSL.I tried to write several did not work, also did not search.

dt=Current Data Table();
Column(dt,1)<<Data Type(Numeric)<<Set Modeling Type(4-byte integer);

Column(dt,1)<<Data Type(integer)<<Set Modeling Type(4-byte);

Thanks Experts!

[#8]

This conversion to binary is also not recorded in the JMP 16 log.

[#9]

I'm pretty sure if you save the table script (of a table with 4-byte columns) and then edit that script you will see the proper Syntax for creating columns in the 4-byte binary format.

[#10]

I think it is numeric(4)