Fast text file format for importing to JMP

[#1]

I produce large delimited data files (typically csv, or tsv) which will ultimately be imported into JMP by users. Typically 1-3GB in size. I can specify the format of the files and would like to get some opinion on which is the best non JMP format to use and are there any hacks for speeding up the importation of large text based data files. 

Cheers, Troy

[#2]

Depending on the settings in the preferences, it's possible to import txt import - with point and comma .
"not possible" means: data can be imported but numbers will be interpreted as characters.

JMP can handle numbers much more efficiently than text. The bottleneck to import them as strings - an then convert them to numbers could be too narrow to get your 1-3GB into the computer memory.

With JMP 17(?) compact columns were introduced. This allows to store character columns (with repeating entries) much more efficient in the computer memory. But there could be a bottleneck to get the data from a csv file into the compact column.

If you want to load numeric data in a "grid", please have a look at hdf5.
JMP can import such files, but the functionality is restricted. Better load the data via Python.

[#3]

If there is time/date data in the input file, there are some restrictions: 
CSV import force MDY or DMY date format 

[#4]

Have you already tried something as you have the files available?

• Which options have you tried to open the files?
  • JMP "ways" of opening file: Open / Multiple File Import
  • Python Integration in JMP18 (DuckDB / pandas / polars and so on)
• Did you run into some performance issues? 
• Which file formats have you tried and which you can create?
  • Usually .csv is pretty good file type as it is easy to create and generally easy to open
  • There are other filetypes, but they can be more difficult to create and have their pros/cons such as Parquet, Pickle or Feather (you will have to use JMP's Python integration to load these into JMP).
  • Could database be an option? sqlite would be most likely the simplest option and JMP can open it easily
• How many files should user be open at the same time?
• How many users are there?
• Will all users open the same files? Could you open the with some automated process and create a jmp table which users would then use or store the results to database instead of text files?