JMP® 18: Going beyond simply satisficing Pythonistas

[#1]

One year ago, JMP users who also used Python made it clear they thought our Python interface was less than optimal. Ouch! We thought the JMP and Python integration we introduced more than five years ago was hitting the mark. Realizing it was only satisficing (in other words, that it was good enough but not the best), we decided to address the challenge head on.

JMP Principal Systems Developer Paul Nelson @Paul_Nelson  spearheaded the effort to create a consistent, full-featured, integrated development environment for writing and executing Python scripts within JMP in JMP 18.

I recently spoke with Paul about the upgraded JMP and Python integration. Paul emphasized that the goal was to provide a significantly more productive environment for Python developers. Some key improvements include:

A consistent environment that will work immediately when installed, without configuration. Python supports unlimited customer Python environments. Adding new JMP and Python capabilities that could not be tested in unlimited customer Python environments was untenable. The solution? JMP 18 includes an embedded version of Python that is the basis for a fully tested environment that will work immediately upon installation.

Direct memory access to Python data tables in a live environment. In previous versions, Python users accessed a copy of a JMP data table. Now, JMP 18 provides direct access to JMP data tables from the Python environment, so  Python users can access, create, and modify JMP tables from Python scripts within JMP.

A new Python Script Editor that uses Python commenting and code coloring, as well as an embedded log. The previous integration used JMP Scripting Language (JSL). JSL was familiar to JSL developers but was often uninviting to JMP users with Python experience but no JSL experience. In JMP 18, users can open and run Python scripts directly in the JMP script editor and run JSL from their Python scripts. The JMP Scripting Index documents and gives samples of the new functionality. Having a consistent environment for Python scripts also provides a foundation for writing add-ins using Python as well as JSL.

Hear more from Paul:

Paul Nelson's hobbies include carving and molding practical and interesting houshold objects.

If you missed Paul's March 21 live webinar on this topic, view the webinar video Developer Tutorial: New Python Integration and Scripting Capabilities in JMP 18 to see him demonstrate the interface.  And, you can see a recoding of Paul presenting as part of the March,2024  JMP  Peaks, Pipes, Presets, Profiles, and Python in JMP® 18 (start at ~Time 40:31).

By the way, not being a Python programmer, I hadn’t heard the rumor that Python originator, Guido van Rossum, was reading scripts from “Monty Python's Flying Circus” when he started working on the new language. He wanted a short, unique, intriguing name, so he decided to call it Python. Sounds plausible!

[BHarris]

Wow, this sounds awesome!

[tsl]

Comment / Question on bringing data back to JMP from Python.

Currently we can do this :

dt = Python Get( df )

where 'df' is a Python dataframe.

I have noticed that what the above command actually does is write the dataframe out to a CSV in the $TEMP folder, then invoke a JSL to read it in, then delete that temporary CSV. For large files there's obviously an overhead to both writing the file and scanning it / reading it in.

It would be very nice if this could be an in-memory transfer direct from Python to JMP

[Paul_Nelson]

In JMP 14-17 both the Python Send ( dt ) and Python Get( df ) operated by saving CSV through a temporary file.  This is one of the major changes in JMP 18.  With JMP 18, the Python Send ( dt ) now creates a jmp.DataTable object in Python.  This is not a copy it is a live in memory reference to the data table.  It is able to be read, and modified directly from Python.  From JSL if you do a Python Get ( dt ), the data table you will get the in-memory data table reference.  Left for compatibility concerns was the original Python Get( df ); method of getting a pandas dataframe into a JMP data table as a copy.  

You now have the power to take a data frame in Python and simply iterate on the data frame columns and create in-memory, the equivalent JMP data table.  See jmp.DataTable. under the new Python category in JMP 18's scripting Index.  There is also a Python subdirectory to Sample Scripts.  In there you will find a JMP2pandas.py script that shows two ways to create in-memory a pandas.DataFrame from a jmp.DataTable object, and one example of creating the jmp.DataTable object from the pandas.DataFrame ( in-memory )

# create JMP DataTable from pandas DataFrame
dt2 = jmp.DataTable('BC2',df.shape[0])
for j in range( df.shape[1] ):
	dt2.new_column(dt[j].name, dt[j].dtype)
	dt2[j] = list(df.iloc[:,j])

Where df is a pandas dataframe, and dt2 is the new JMP DataTable.   ( note dt is an also a JMP jmp.DataTable object,  the type parameter is a jmp.DataType enumeration value. ). The example file opened 'Big Class.jmp' as dt, creates a pandas.DataFrame, the creates dt2 from the data frame.

[BHarris]

Out of curiosity, will JMP v18 support formula columns with python code?

[Paul_Nelson]

Formula columns do not support Python code in JMP 18.  It is something that is being investigated, and would be a great feature.  There was an enormous amount of infrastructure that needed to be added to make JMP 18 robust, work on startup, support a Python aware script editor, enhanced error reporting, add scripting index, and opening up a bridge from Python to JMP.  Many things that had been lacking for some time, have now been enhanced to make what be believe to be an inviting and productive Python environment within JMP. 

Now that the heavy lifting has been done, we are not calling it a day.  Much more to come.

[BHarris]

@Paul_Nelson We're very excited to try it out, and are super grateful for your work so far!

[hogi]

Besides 

dt2 = jmp.DataTable('BC2',df.shape[0])

to create a data table in Jmp,

is it also possible to refer from Python to an already-existing data table and continue working with this table from Python?

[Paul_Nelson]

dt = jmp.current()

dt = jmp.table('BC2')

You can get the current data table, and you can get a table reference by name.

[hogi]

one can even index the column

dt['colname'][row-1]

is the same as 

dt[row]["colname"]

in JSL.

Very useful if a function in JSL is missing: just make a little detour to Python an continue with JSL.

[Paul_Nelson]

Follow up note on Expression columns.   While the jmp.DataTable.Column object does not support reading or writing the column from Python, you can create the column from Python and work around the issue using jmp.run_jsl()

...
# with a data table dt, having columns 'list' and 'np.array'
# add a new expression column with a formula using run_jsl()

# create an expression column from Python
dt.new_column('expression', jmp.DataType.Expression)
jmp.run_jsl('''
dt = Current Data Table();
:expression	<< formula(:list + :np.array);
''')

...

Getting an Image from a column is a bit more involved, but is possible using the run_jsl(), but that is a good topic for a blog post all of its own.