JMP19 - Dataframe Interchange Protocol deprecation warning

[#1]

Seeing this in my JSL Logs recently, has jmp19 adapted to something else ? This will eventually be phased out, i have used a custom fuction to convert jmp table to pandas dataframe in jmp18 but that didn't work for jmp19. So, what are the ways to achieve the same now? 

I run my python code inside the JSL Script. 

Python Send(o28k_raw);
Python Submit(
raw_data = pd.api.interchange.from_dataframe(o28k_raw)
);

This warning: 

<string>:18: Pandas4Warning: The Dataframe Interchange Protocol is deprecated.
For dataframe-agnostic code, you may want to look into:
- Arrow PyCapsule Interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
- Narwhals: https://github.com/narwhals-dev/narwhals

[#2]

Have you seen Going further with Python in JMP 19 - JMP User Community?

Working with Pandas Dataframes became easier in JMP 19, using the new function jmp.from_dataframe(). 

[#3]

I found below mentioned code from the same link but the query is diff as I have mentioned.

pd.api.interchange.from_dataframe()

[#4]

Interesting, when I run the following Python script I do not see any warnings: 

import jmp
import pandas as pd

dt = jmp.open(jmp.SAMPLE_DATA + "Big Class.jmp")
pandas_df = (pd.api.interchange.from_dataframe(dt))
print(pandas_df) 

Could you check to see if the error occurs when running the Python from within JMP's Python editor, or if it is specific to Python executed using Python Submit? 

I also tried the following at it did not give me an error: 

dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
Python Send( dt );
Python Submit( "pandas_df = (pd.api.interchange.from_dataframe(dt))" );
Python Submit( "dt2 = jmp.from_dataframe(pandas_df)" );
Python Get(dt2);

Could you test these scripts to see if they also give the same warning on your end? 

[#5]

I see this warning when I run python executed using Python Submit as well as when I run the python script from JMP's python editor

pandas_df = (pd.api.interchange.from_dataframe(dt))
/*:
<string>:1: Pandas4Warning: The Dataframe Interchange Protocol is deprecated.
For dataframe-agnostic code, you may want to look into:
- Arrow PyCapsule Interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
- Narwhals: https://github.com/narwhals-dev/narwhals

//:*/
import jmp
import pandas as pd

dt = jmp.open(jmp.SAMPLE_DATA + "Big Class.jmp")
pandas_df = (pd.api.interchange.from_dataframe(dt))
print(pandas_df)
/*:
<string>:5: Pandas4Warning: The Dataframe Interchange Protocol is deprecated.
For dataframe-agnostic code, you may want to look into:
- Arrow PyCapsule Interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
- Narwhals: https://github.com/narwhals-dev/narwhals

[#6]

Thanks for the reply. The Python snippet above is from the scripting index, so it usually runs without any issues. 

In case you are not running the latest version of JMP (19.1.2), it might be a good idea to update and then try again. If you are already on the latest version, then I would recommend reaching out to JMP Technical Support support@jmp.com. 

[#7]

The warning comes about with pandas 3.0 and greater

[#8]

The warning is coming from pandas itself.  

We have made the updates internally to utilize the PyCapsule interface.  Exactly when that ships will be a matter of testing to ensure stability.  This will also add the jmp.from_arrow() function and extension of the jmp.DataTable object to support the Arrow interface requirements.  The jmp.from_dataframe() is reworked internally to utilize the new functionality.

import pandas as pd

dt = jmp.open(jmp.SAMPLE_DATA + "Big Class.jmp")
pandas_df = pd.DataFrame.from_arrow(dt)
print(pandas_df) 

[#9]

Thank you for the clarification.

My understanding is that the current deprecation warning originates from Pandas rather than JMP, and that future JMP releases will support the Arrow/PyCapsule interface directly, allowing workflows such as:

dt = jmp.open(...)
pandas_df = pd.DataFrame.from_arrow(dt)

without relying on the DataFrame Interchange Protocol.

In the meantime, I am trying to determine the best approach for maintaining production scripts that are deployed across JMP versions.

Historically, in JMP 18, I relied on a custom conversion function:

def jmp_to_pandas(jmp_table):
  data = {} 
  for col in jmp_table: 
     data[col.name] = list(col) 
  return pd.DataFrame(data)

After moving to JMP 19, I switched to:

pd.api.interchange.from_dataframe(...)

which works but now generates the Pandas deprecation warning.

Would your recommendation be to continue using the interchange API until the Arrow support ships and I have to re-visit the production deployed script, or would you suggest reverting to a custom DataTable to DataFrame conversion for production code that needs to remain stable across Pandas and JMP version updates?

Also, for the upcoming Arrow implementation, will pyarrow be a required dependency, or will the Arrow interface be exposed through JMP without requiring users to install additional packages?

Thank you for the additional context.

[#10]

Generally, our recommendation would be to continue using the more modern methods of DataFrame conversions like the interchange protocol due to their performance benefit. If scripts specifically require access to console output, then suppressing the warning may be of help but is generally not recommended:

import warnings
warnings.simplefilter(action='ignore', category=DeprecationWarning)

In terms of stability, this deprecation is marked Pandas4Warning, which means that these warnings will be enforced in pandas 4.0 (see Updated deprecation policy). Historically, pandas has had a major release roughly every ~3 years though they have not (to my knowledge) indicated the estimated release date. We are still confident that the Arrow PyCapsule interface will be available in JMP by pandas 4.0.

Without knowing the specific JMP/pandas versions the workflow relies on, a custom DataTable conversion still may be more suited to the use case. It also should be noted that jmp.from_dataframe_using_csv is also available and has seen no significant difference with the major release of pandas (tested with 2.3.3 vs 3.0.0). That function is available in JMP 19 and will continue to be available with JMP 20.

As of the current implementation, jmp.from_dataframe() will call the Arrow PyCapsule interface (jmp.from_arrow) internally when called with a pandas.DataFrame to help backwards-compatibility so no adjustment to the script itself should be necessary. However, to your next question, pandas does have pyarrow as a required dependency when using these methods, so you may have to revisit the execution environment to install pyarrow. To our knowledge, this seems to be more specific to pandas since polars does not require the additional pyarrow dependency. 

We'd be happy to clarify or answer any more questions you might have about Python or DataFrame conversions so feel free to ask whenever.