Python Code unable to generate output in JSL (JMP 17 Pro)

[#1]

Hello All,

I have a JSL script that creates a Datatable, and I have a python code block within the JSL script which I want to pass the DataTable to make predictions on the DataTable using a trained machine learning model. Before the python code block, the DataTable is accessible and even opens in JMP. However, after the python code block when I try to access the predictions made by the neural network model based on the DataTable using "Show(preds_df)", I get error "Name Unresolved: preds_df in access or evaluation of 'preds_df'". I am using JMP Pro 17. Below is my code block for review. 

<

Clear Log();
Clear Symbols();
Names Default To Here( 1 );
 
// Define the list of options for the dropdown
SITE = {"SITE_1", "SITE_2"};
MODELS = {"NN_MODEL", "RNN_MODEL"};
 
PythonConnection = Python Connect();
conn = PythonConnection << Is Connected;
PythonConnection << Send( dt);
close(dt);
PythonConnection << Send(MODELS);
 
PythonConnection << Execute("
import pandas as pd
from sklearn.preprocessing import StandardScaler
import torch
import pickle
import numpy as np
 
// Model Selection
if 'NN_MODEL' in MODELS:
      model = 'NN_MODEL'
else:
      model = 'RNN_MODEL'
 
# Extract categorical and numerical features from data
categorical_features = ['NAME', 'COUNTRY', 'SCHOOL']
numerical_features = ['AGE', 'SIZE', 'HEIGHT', 'WEIGHT']
 
# One-Hot encoding for categorical features in place
process_df = pd.get_dummies(dt, columns=categorical_features, drop_first=True)
 
 
# Normalize numerical features
scaler = StandardScaler()
new_df[numerical_features] = scaler.fit_transform(process_df[numerical_features])
 
# Exclude bin columns from the model input
B_cols = ['B1', 'B8', 'B9', 'B10']
B_columns = [col for col in B_cols if col in process_df.columns]
input_df = process_df.drop(B_columns, axis=1)
 
labels = process_df[B_cols]
ids = process_df[B_cols].index
 
# Load the trained model based on user selection
if model == 'NN_MODEL':
    model_path = '.\\nn_model.pth'
    model = torch.load(model_path)
    model.eval()
 
    # Convert DataFrame to tensor
    in_tensor = torch.tensor(input_df.values, dtype=torch.float32)
 
    # Predict using the model
    with torch.no_grad():
      nn_preds = model(in_tensor)
      nn_preds = torch.round(nn_preds).to(torch.int).numpy()
     nn_preds = np.clip(nn_preds, a_min=0, a_max=None)
    preds_df = pd.DataFrame(nn_preds.numpy(), index=ids, columns=labels)
else:
    model_path = '.\\rnn_model.pkl'
    with open(model_path, 'rb') as file:
        model = pickle.load(file)
    rnn_preds = model.predict(input_df)
    rnn_preds = np.clip(rnn_preds, a_min=0, a_max=None)
    preds_df = pd.DataFrame(rnn_preds, index=ids, columns=labels)
 "
);
 
Show(preds_df)

>

[#2]

Very quickly looking, preds_df is just a pandas dataframe, not JMP table, so you cannot access it from JMP.

[#3]

Is there a way to convert the dataframe to JMP datatable, to visualize it outside the python block.

[#4]

You can. I don't remember if JMP17 is able to do it directly but you can for example go through a .csv file: create .csv from the pandas dataframe and then pass that path to JMP and open it.

[#5]

Try this at the end of your JSL script:

dt = Python Get( preds_df );
dt << New Data View;

Also JMP 18 can run Python scripts natively so no need to embed them in JSL scripts.

[#6]

I got "Send Expects Scriptable Object in access or evaluation of Send, "dt" << /*###*/New Data View/*###*/ "

[#7]

First the syntax for  << Execute ( ) is wrong.  See the example in the scripting index.  The first parameter for Execute is a JSL list of inputs, followed by a JSL list of output variables, followed by the script, and then optional echo(1 or 0) parameter.

The script should be delineate by JSL's 'raw' string markers "\[  script here  ]\" to ensure white space and line endings are preserved.  I am surprised you are not getting a syntax error, but I see the same lack of error on 19... I'm looking into that.  The show preds_df for me simply printed 

dt = .;

which is missing. This then of course means:

dt << New Data View

Throws an error because dt  ( missing ) is not scriptable.

The below code replicates your issue.

conn = Python Connect();
conn << Get Version();

conn << Execute( "

import pandas as pd
import numpy as np

pandas_df = pd.DataFrame(
    {
        "A": 1.0,
        "B": pd.Timestamp("20130102"),
        "C": pd.Series(1, index=list(range(4)), dtype="float32"),
        "D": np.array([3] * 4, dtype="int32"),
        "E": pd.Categorical(["test", "train", "test", "train"]),
        "F": "foo",
    }
)
print(pandas_df)
]\", echo(1) );

dt = Python Get(pandas_df);
show(dt);
dt << New Data View;

This code runs as expected

conn = Python Connect();
conn << Get Version();
conn << Execute( {},{},"\[

import pandas as pd
import numpy as np

pandas_df = pd.DataFrame(
    {
        "A": 1.0,
        "B": pd.Timestamp("20130102"),
        "C": pd.Series(1, index=list(range(4)), dtype="float32"),
        "D": np.array([3] * 4, dtype="int32"),
        "E": pd.Categorical(["test", "train", "test", "train"]),
        "F": "foo",
    }
)
print(pandas_df)

]\", echo(1) );

dt = Python Get(pandas_df);
show(dt);
dt << New Data View;

A couple things of note JMP 17 is the last version that utilized Python from the user's environment, rather than an embedded Python installed with JMP 18 and newer.  In JMP 14-17 the Python Send (dt); created a pandas dataframe in the Python environment by saving the Data Table to a temporary CSV file, then causing pandas to read that CSV file into a dataframe. Python Get( df ); behaved analogously buy having pandas write the dataframe to a CSV file and have JMP import the CSV file.  This meant that you had 2 copies of the data the JMP data table, and the pandas dataframe.  

This changed in JMP 18 with the introduction of the embedded Python 3.11.x and a builtin jmp import package that provides a jmp.DataTable class. This introduces an in-memory live reference to the JMP data table accessible from Python.  Python Send( dt ); in JMP 18+ now creates a jmp.DataTable object instead of a pandas datagram.  dt2 = Python Get( dt ) just gets the data table reference assigning to the variable.  dfdt = Python Get( data_frame ); still uses the old mechanism to convert and load the dataframe as a data table via CSV file.  In JMP 18 there is a Python sample script in the scripts directory shown how to do the original dt->csv->df  and df->csv->dt conversions.  JMP 19 supports the Portable Dataframe Protocol so it's a one-liner to go dt->df or df->dt in memory.  Samples named dt2pandas.jsl, dt2pands.py, dt2pandas2dt.py

For JMP 17:

Names Default to here(1);

dt = Open("$SAMPLE_DATA/Big Class.jmp");

// I have multiple Python's on system so I choose 3.10 where I have 
// matplotlib, pandas, numpy, ... installed
Python Init(Use Python Version("3.10"));
Python Send(dt);
close(dt);

dtdf = Python Get("dt");
Show(dtdf);
dtdf << New Data View;

This works fine on my machine.  If you can't get this to run please examine your Log to see if you are showing errors. Modifying Python Init() either for your environment if necessary.

JMP 19

conn = Python Connect();
conn << Get Version();

conn << Execute( {},{},"\[
import jmp

import pandas as pd
import numpy as np

pandas_df = pd.DataFrame(
    {
        "A": 1.0,
        "B": pd.Timestamp("20130102"),
        "C": pd.Series(1, index=list(range(4)), dtype="float32"),
        "D": np.array([3] * 4, dtype="int32"),
        "E": pd.Categorical(["test", "train", "test", "train"]),
        "F": "foo",
    }
)
print( pandas_df )

# JMP 19 create a data table direcly from a dataframe
dt = jmp.from_dataframe(pandas_df)
print( dt )
]\", echo(1) );

show(dt);
// dt << New Data View;   // already on screen

Much more can be done with just running the Python script from the editor no JSL wrapper needed in JMP 18+