topic Re: How to use JSL to make TXT files of decimal data into binary format files? in Discussions

How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Fri, 09 Jun 2023 19:57:38 GMT

Decimal files use recent tabs to divide data into three columns.
They are all integers:
The data in the first column ranges from 90,000 to 199,999
Column 2: positive integers from 100 to 900,000
Third column: positive and negative integers or 0

They have a lot of rows.
There is no limit to how to save them as binary files or file suffixes using JSL.
And need to figure out how to re-read restored decimal data from this new binary into JMP.
Thank you very much!

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Mon, 27 Sep 2021 11:53:10 GMT

I want to reduce the size of the file in this way.

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Wed, 29 Sep 2021 02:25:24 GMT

Open the preferences and look for the option to allow short integer columns. Create a data table with the data and use four bytes for each of your columns (available in the column dialog after you turn on the option.)

Using a 4-byte integer

I forget the exact rules, but approximately:

1-byte columns represent +/- 1e2 and missing,
2-byte +/- 3e4 and missing and
4-byte represent +/- 2e9 and missing.

Optional: turn on the whole table compression before saving. That will use a zip-like compression that might compress even more, but only on disk, at the expense of slower saving and loading due to compression.

Using 4-byte integers will be half the size of 8-byte doubles. I think that will be true even after loading in memory. Turing on compression for the 8-byte doubles is likely to make the file almost as small as the compressed integer file because it should find the same redundancies to squeeze out.

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Wed, 29 Sep 2021 09:42:39 GMT

Thank Craige!

I have formatted each column like this, but still can't figure out how to save the binary file to the computer's hard drive?

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Wed, 29 Sep 2021 11:50:39 GMT

The JMP data table is a binary file. Save it in any of the usual ways (file->save, <<save) as a .JMP extension.

Yes, there is some meta data in the file, but if you have 1,000,000 rows of three 4-byte numbers, 12,000,000 bytes, the overhead of the meta data will be about ( 752 bytes / 12,000,000 bytes ), or < .01% overhead. In exchange, you

Can load the file for free
Don't have to worry about remembering the column names
Can share it with other JMP users
Can add more columns if you need to, even mix and match 8-byte doubles
Don't need to invent another file extension (.dat, .bin, .myformat ?)

If you need to share the data with non-JMP users, a compressed CSV will be pretty small and very portable.

All that said, if you must make a binary file of some other format, see for an example using matrixToBlob. That file is holding highly structured data with self-contained offsets within itself to represent run-length encoded data for a bitmap, connected to a lookup table for each pixel's value. It was hard to get correct and hard to test and hard to use, but gave me the results I needed.

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Wed, 29 Sep 2021 12:18:58 GMT

I SEE.

Thank Craige!

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Wed, 29 Sep 2021 14:13:12 GMT

How to format these columns to 4-byte INTEGER using JSL.I tried to write several did not work, also did not search.

dt=Current Data Table();
Column(dt,1)<<Data Type(Numeric)<<Set Modeling Type(4-byte integer);

Column(dt,1)<<Data Type(integer)<<Set Modeling Type(4-byte);

Thanks Experts!

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Wed, 29 Sep 2021 14:11:24 GMT

This conversion to binary is also not recorded in the JMP 16 log.

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Wed, 29 Sep 2021 14:44:07 GMT

I'm pretty sure if you save the table script (of a table with 4-byte columns) and then edit that script you will see the proper Syntax for creating columns in the 4-byte binary format.

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Wed, 29 Sep 2021 14:45:39 GMT

I think it is numeric(4)

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Wed, 29 Sep 2021 23:31:21 GMT

I still find this code.

I tried to write several did not work

Column(dt,1)<<Data Type(numeric(4))<<Set Modeling Type(Continuous);


Column(dt,1)<<Data Type(4)<<Set Modeling Type(Continuous);

Thanks!

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Thu, 30 Sep 2021 02:01:17 GMT

(view in My Videos)

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Thu, 30 Sep 2021 02:25:06 GMT

I see. I usually use JMP14 or 15.Because it's still Windows 7.

I tried another computer last night using JMP 16.I just saw it. "16.0.0"

Thanks for Craige's unreserved patient help.

Re: How to use JSL to make TXT files of decimal data into binary format files?

Craige_Hales — Thu, 30 Sep 2021 09:25:13 GMT

The short numeric columns have been in JMP for a long time. Probably back to JMP 5, maybe longer. The preference menu has changed over time, but I think the script will work fine in old versions of JMP.

Re: How to use JSL to make TXT files of decimal data into binary format files?

lwx228 — Thu, 30 Sep 2021 14:42:25 GMT

dt=Open("/C:/1/ss.txt",columns(
New Column("c000001",Numeric(4),"Continuous",Format("Best",12)),
New Column("c000002",Numeric(4),"Continuous",Format("Best",12)),
New Column("c000003",Numeric(4),"Continuous",Format("Best",12))),
Import Settings(End Of Line(CRLF,CR,LF),End Of Field(Tab,CSV(0)),Strip Quotes(1),
Use Apostrophe as Quotation Mark(0),Use Regional Settings(0),Scan Whole File(1),
Treat empty columns as numeric(0),CompressNumericColumns(0),CompressCharacterColumns(0),
CompressAllowListCheck(0),Labels(0),Column Names Start(1),Data Starts(1),Lines To Read("All"),
Year Rule("20xx")));

Thanks!