Executing a python exe from Run program()

[#1]

I have a python executable (so I or others don't need python installed on the pc) which can analyze some data...

I would like to call and run this from JMP, but I am not sure the best way as I would like to be able to feed variables to this python exe from JMP as well as get the return.

I have done this successfully using the integrated python commands, but never with a stand along python exe file. One of the variables I'd need to feed in is a data table. Currently I'd need to have JMP save the table as a .csv in a temp location, then this python exe can run and edit that .csv ... then JMP can open the updated CSV (I don't have that stream coded in JSL as I feel there must be a better way)

Any thoughts? I have minimal experience with Run Command() so any starting pointers will be appreciated

Thanks!

[#2 · Accepted solution]

Using an executable built from a Python program via RunProgram, or calling the Python interpreter with the Python script from RunProgram is similar. Neither one has the integration that using the integrated Python has. You'll need to use something like a CSV file to send data to the Python program and to get data back from the Python program. You can also pass command line parameters to the .exe. (Or to the python interpreter.)

In your case I'd suggest writing the CSV to $temp directory and printing the results in python. Pass the csv path via the command line. Use RunProgram(...readfunction("text")...) to retrieve the string of printed output. delete the temp dir file.

[#3]

@Craige_Hales Ok, that makes sense... that's where I was leaning, but outputting the information into the string for retrieval would be easier than a 2nd csv to deal with.

[#4]

I've not done this and can't easily test it on my system at the moment, but it looks like you can read from an executable's stdout and write to an executable's stdin streams using JSL: https://community.jmp.com/kvoqx44227/attachments/kvoqx44227/discovery-2016-content/67/1/Run%20Progra... 

Python can read from stdin (and output to stdout) as a file-like operation: Python - stdin, stdout, and stderr - AskPython

So I'm reasonably sure you could ` << Write` your data table line-by-line to the stdin of the executable from JMP, use a file-like method to read the lines from JMP in Python, construct those lines into a Pandas dataframe, run your Python operations, then output the Pandas dataframe line-by-line from Python to the stdout, feeding that back to JMP.

[#5]

@Michael_MXAK Hmm, that's also an interesting idea. For smaller files it may work fine. Some of these files may get rather large so that line by line method may not be overly efficient... It's great food for thought, though.

[#6]

The advantage of CSV or Pandas is you don't have to parse the python output. If your output looks like csv, you can receive it as readprogram(blob) and open the blob with the csv option to get a table.

[#7]

Yeah, I'd say it's worth testing. I've run a similar stream writing/reading function on Unix for going from a very large Pandas dataframe to a Postgres table and it's blazing fast. I don't know what JMP's implementation is like, but maybe you can write the entire table to the stdin in one go? As a gigantic list of lists maybe...

For reference, I tested the Python method on a Pandas dataframe that was something like 3 GB in size and it took less than 10 seconds to complete. The method is this:

def psql_insert_copy(table, conn, keys, data_iter):
    """
    Parameters
    ----------
    table : pandas.io.sql.SQLTable
    conn : sqlalchemy.engine.Engine or sqlalchemy.engine.Connection
    keys : list of str
        Column names
    data_iter : Iterable that iterates the values to be inserted
    """
    # gets a DBAPI connection that can provide a cursor
    dbapi_conn = conn.connection
    with dbapi_conn.cursor() as cur:
        s_buf = StringIO()
        writer = csv.writer(s_buf)
        writer.writerows(data_iter)
        s_buf.seek(0)

        columns = ', '.join('"{}"'.format(k) for k in keys)
        if table.schema:
            table_name = '{}.{}'.format(table.schema, table.name)
        else:
            table_name = table.name

        sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
            table_name, columns)
        cur.copy_expert(sql=sql, file=s_buf)
        
df.to_sql('example', schema='demo', con=DL.engine, method=psql_insert_copy)

Which I found on some site somewhere ages ago (I'm not that clever ). It uses a data iterator and writes it row by row to the stdin of the postgres cursor, but it's still extremely quick. 

[#8]

There is a working example in Make a Video in JMP with FFmpeg . It is using the FFmpeg executable, not a python executable, but it does show both reading from stdout and writing to stdin. The read function just echos to the JMP log, but you can buffer the text and do something else if you like. The write function is streaming images to FFmpeg for encoding, one at a time.

With FFmpeg, it is useful to have the real time echo to the JMP log as images are passed in to FFmpeg; it lets me see if something is wrong in the middle of a long process. If you have a shorter process, with a single (possibly large) output at the end, you'll find it more convenient to use readfunction("text" or "blob") which lets JMP buffer the stdout into a single string that is returned. It also doesn't require waiting (see end of example) for the RunProgram to terminate to get the results. On the other hand, using readfunction(function(...)) does allow JMP to start processing output while it is being generated.

If the python program outputs a CSV to stdout, readfunction should use a blob for two reasons (even though the data should be 7-bit ASCII): a blob can be bigger than 2GB but a JSL string can't, and you can load the blob as dt = open( blobVariable, "csv"). Put a header on the CSV and it is useful even if you only need to return a few named values as a table with a single row.

There is some example python code buried in Beowulf, Newton, and Mr Hanson that reads command line parameters -- search for getopt --, but you can probably find easier to use examples elsewhere. In that example, JMP is using plink (from putty) to run python on a remote machine, and using an NFS disk on the network to share data.