Issue Sending Pandas DataFrame from Python to JMP 18 Data Table

[#1]

Context:
I am using JMP 18 with Python integration to merge multiple datasets. The merge and processing in Python works correctly using pandas, but when I try to send the final pandas DataFrame back to JMP using Python Get(), the resulting JMP Data Table is empty. I want to create a JMP table from my pandas DataFrame in Python and access it in JSL.

Example of What I’m Trying:

// Send JMP tables to Python
Python Send(jmp_table1);
Python Send(jmp_table2);

// Python processing
Python Submit("
import pandas as pd

def jmp_to_pandas(jmp_table):
    cols = list(jmp_table)
    data_dict = {}
    for col in cols:
        data_dict[col.name] = list(col)
    return pd.DataFrame(data_dict)

# Convert JMP tables to pandas DataFrames
df1 = convert_jmp_to_pandas(jmp_table1)
df2 = convert_jmp_to_pandas(jmp_table2)

# Merge tables
merged_df = pd.merge_asof(df2, df1, left_on='Time', right_on='Time', direction='nearest')
");

// Attempt to bring merged DataFrame back to JMP
dt_merged = Python Get(merged_df);
dt_merged << Set Name("Merged_Data");
dt_merged << Show Window;

Problem Observed:

• dt_merged is created but contains no data. [but the data frame is generated as I can save it to csv and use it]

• Using global in Python does not seem to fix the issue.

• Python Get() does not appear to recognize the pandas DataFrame correctly.

What I Want:

• Convert a pandas DataFrame into a JMP table from Python.

• Access it in JSL with proper name, show the window, and manipulate columns.

• Avoid creating empty tables and maintain all rows/columns.

Potential Solutions I’m Considering:

• Exporting to CSV from Python and then importing to JMP - tried and works but it's something I don't want. 

• Ensuring the DataFrame is declared global before Python Get(). - didn't work

Environment:

• JMP 18

• Python 3.x (embedded)

• Using inside a JSL script

Any guidance or working examples of sending pandas DataFrames back to JMP from Python would be greatly appreciated!

[#2]

Check out the "Building a Pandas DataFrame by Looping Through Data Table Columns" example from  Scripting Guide > Python > Creating a Pandas DataFrame from a JMP Data Table . Also I think in JMP19 this is easier  Python Dataframe to JMP Datatable without intermediate csv (JMP Wish List) but I haven't tried yet due too multiple issues with JMP19.

[#3]

Your dt_merged does contain data, it does not contain a view.

use 

dt_merged << New Data View;

instead of

dt_merged << Show Window;

The table exists but doesn't yet have an associated window.  

Also note that in JMP 18 Python Get( df ) does use dataframe -> CSV    CSV -> data_table internally just like JMP 14-17 did.  Only the JMP data table has live access to the data across JMP / Python boundaries. 

In JMP 19 the situation is much better.  JMP supports the Portable Dataframe Protocol and it's a 1 line of code to go from a JMP dt => a pandas DataFrame and another 1 line to go DataFrame to DataTable.  This works for any dataframe like package that supports the DataFrame Protocol and the conversion occurs in memory.

Your other choice is creating the merged data table in Python using the jmp.DataTable APIs and walking the dataframe columns creating JMP columns one by one in the new data table.  Looks like you have time data as part of the tables.  This takes care to get correct.  In JMP 14-18 you have to marshal between JMP's data time representation and Python's data time representation. Epoch date for JMP is Jan 1, 1904 ( same as Excel on the Mac) vs Python's Epoch Date is Jan 1, 1970.  

JMP 19 properly marshals datetime values across the JSL / Python boundaries.

There are examples in the 'Python' section of the scripting index, for the jmp import package.

I would highly recommend updating to JMP 19.  JMP 18 was a foundational change to Python Integration in JMP .  While JMP 18's support was not as complete as we would have liked, it provided so much capability we did not want to hold off until JMP 19.  

[#4]

I also want to point out that that the jmp.DataTable object is iteratable so you can directly use jmp_table in your for loop. It already behaves like a list of columns

def jmp_to_pandas(jmp_table):
    data_dict = {}
    for col in jmp_table:
        data_dict[col.name] = list(col)
    return pd.DataFrame(data_dict)

[#5]

I did observe the dates in my dataframe are quite bizzare please refer the screenshot. this is date format and only few are affected, date-time doesn't seem to be affected. 

[#6]

This is date in seconds. you can display any number as date by changing the column settings:

[#7]

Hey thanks, I figured this out but I am working with jsl and python code in between. Is there a better way to take care of this in the code itself. 

1 - process in python using epoch time offset

2- reassign the column format  to each columns in the jsl code. [tedious] 

[#8]

1. Unix Timestamp to Real Date & time 
2. 
:col << Format( "yyyy-mm-dd" );

[#9]

The screenshot was an example, I have 100s of columns :\ , with inconsistent formatting in the source data . Thanks btw

[#10]

JMP19?