Open up JMP table format

[vince_faller]

I would love if the JMP table format were open.  As in I could build a jmp table from pwsh/python (or whatever) from scratch. 

I have two use cases

1. make my pipeline more efficient so that I don't have to wait for a JMP machine to open up to do some backend pre-processing while still having all of the wonderful features of JMP when I'm ready for it.  
   1.   I am currently doing this by having JMP run some stuff on a machine that just preprocesses, but JSL isn't built to handle this. 
2. interactive html isn't up to what I need it to do for everything so being able to read a jmp table to make some custom web frontend stuff would be useful

Obviously I could just use a csv for data, or switch everything over to another language.  But being able to read/write the jmp data format directly would make a world of difference. 

[Sarah-Sylvestre]

Hi @vince_faller, thank you for your suggestion! We have captured your request and will take it under consideration.

[SamGardner]

@vince_faller, we have decided to not pursue this feature request.  We appreciate your input and contributions to the user community.  

[guillaumebugnon]

Hello @SamGardner,

It's a shame that this is not a priority.
It would allow more use of jmp in some organizations instead of sometimes doing the rest of the processing in python or R.

[vince_faller]

Yeah unfortunate.  Maybe they'll put out a jmp python library or something just to read a jmp table and nothing more. 

[ThomasDickel]

Hi, I find this feature extremely useful. This would open up JMP tables to the world. Imagine you could convert JMP data for easy visualizing with Power BI.

Or it would also be very interesting if I could create a JMP table using PowerShell reading and combining existing data sources.

@martindemel 

[txnelson]

I have found a fairly usable workaround, by generating a JMP script that creates the data table.  It allows for the inclusion of all of the Column Properties, format columns, Table scripts, etc.  

[Paul_Nelson]

As Bryan commented JMP knows how to handle SQLite database files as well as others have mentioned HDF5.  I also mention that JMP's Python integration recognizes pandas.DataFrame objects from the Python Get( df ); JSL call which in turn creates a JMP Data Table.  This unfortunately this is achieved by having pandas write a temporary CSV file to the filesystem, and JMP reading in that CSV file.  Works well for small data, but not so much for anything big.  

There are a lot of improvements to the Python Integration in JMP 18 that will be unveiled in Manchester.   Stay tuned....

[Paul_Nelson]

JMP 18 Python integration adds a jmp.DataTable object which gives direct access to a JMP data from Python.  The ability to create a JMP data table, modify and access cells, add columns and rows, ...   This supported through an import jmp package.  A Python aware script editor that does syntax highlighting, and ability to run the pure Python scripts right from the editor.  JMP 18 installs an isolated and embedded Python.org based Python 3.11.x distribution.  Think of this as a JMP specific virtual Python environment.  You can also run JSL from Python using the jmp.run_jsl() command.

You can now easily create parquet, hdf5 or other based file formats using external packages such as pyarrow and h5py, or JMP's internal support for HDF5.  Simply iterate across the columns of the jmp.DataTable object and add them to the py file object type of your choice, parquet, pandas, ... The reverse also easy. Such as iterating across the parquet columns using the schema to set the JMP column type and add it column at a time to the JMP data table.  You can still use your external Python workflow, but the Python supported within JMP is only the JMP installed Python.

One thing to note, the import jmp package does not exist independent of the JMP environment.  It is not Python script code nor is it a shared library extension to Python, It is C++ code that is internal to the JMP executable.  While there is a definite use case for an external Python package supporting JMP's data file format, such support is not presently in the works.  

As the developer who wrote the custom code to read and write JMP data tables in the short lived iOS Graph Builder app, working the JMP Data Table binary format outside of the JMP code framework is a non-trivial task.  The iOS code I wrote only supported a subset of the data table's functionality.  This was well before things such as linked tables and images in the data table was even implemented in JMP.  If you have ever had to work directly with an OLE based file format, or Python pickled object as a binary format, the JMP binary format is similar in concept. It is the result of streaming out a series of complicated nested objects to disk.  Additionally the file format supports gzip compressed column data.  The actual structure varies file to file because if an object is not present it the table it doesn't get streamed out, minimizing file size.

[vince_faller]

So just to reiterate/clarify, there's no pypi package, but somewhere in the docs there's some setup for python (same as 17?) that should work as long JMP 18 is installed it should just work?  Will it work across virtual environments?  I remember that being a pain last time I tried.  (RTFM is also an acceptable response).  

So it should be easy to do something like this? Roughly

df = pd.read_csv(whatever)
jmp.newdatatable(df) # if we have to write a wrapper or whatever it's fine.  
jmp.runJSL('df << new table script("thing", print("thing");')

Looks like I need JMP 18.