JMP 18 Python kernel interfering with miniconda Python installation (Run Program)

[#1]

JMP 18 brings native Python. This open-source Python installation uses conda instead of pip, but it seems broken now.

https://github.com/industrial-data/pyJMP/

The way pyJMP works is by executing the following bat file using RunProgram(). 

set JMP_PYTHON_PTH=C:\ProgramData\miniforge_JMP
CALL %JMP_PYTHON_PTH%\Scripts\activate.bat %JMP_PYTHON_PTH%
SET current_path=%~dp0
echo %current_path:~0,-1%
SET notebook_file=jmp_addin_minimum_features.ipynb
SET notebook_file_path=%current_path%%notebook_file%
echo %notebook_file_path%
python --version
jupyter kernelspec list
start /wait /B  ""  jupyter nbconvert --execute --to notebook --inplace %notebook_file_path%

Notice that Python is activated first and then we call a function inside Jupyter.

The following error only occurs in JMP 18, which seems to call its own Python installation (v 3.11 instead of 3.10 installed with conda). If one executes that .bat file from Windows Explorer or a different cmd, the code works as expected.

C:\Program Files\JMP\JMPPRO\18>set JMP_PYTHON_PTH=C:\ProgramData\miniforge_JMP 

C:\Program Files\JMP\JMPPRO\18>CALL C:\ProgramData\miniforge_JMP\Scripts\activate.bat C:\ProgramData\miniforge_JMP 
C:\Users\MrX\AppData\Roaming\JMP\JMP\Addins\Predictor_Explainer
C:\Users\MrX\AppData\Roaming\JMP\JMP\Addins\Predictor_Explainer\jmp_addin_minimum_features.ipynb
Python 3.10.10
Available kernels:
  python3    C:\ProgramData\miniforge_JMP\share\jupyter\kernels\python3
[NbConvertApp] Converting notebook C:\Users\MrX\AppData\Roaming\JMP\JMP\Addins\Predictor_Explainer\jmp_addin_minimum_features.ipynb to notebook
[NbConvertApp] ERROR | Module use of python311.dll conflicts with this version of Python.
Traceback (most recent call last):
  File \!"C:\ProgramData\miniforge_JMP\lib\site-packages\traitlets\traitlets.py\!", line 656, in get
    value = obj._trait_values[self.name]
KeyError: 'kernel_dirs'

It is unclear how to avoid this. Any ideas?

[#2]

Give this a read https://community.jmp.com/t5/JMPer-Cable/JMP-is-Pythonic-Enhanced-Python-Integration-in-JMP-18/ba-p/....

It is likely that the  PyJMP package listed above will not work with v18 with some modification.  

[#3]

Also, the new python integration in JMP probably eliminates the need for PyJMP.   Other packages  and tools that rely on PyJMP can probably be rewritten to use the new Python integration.  

[#4]

True, we will have to rewrite our Predictor Explainer add-in at some point.

The main difference is that JMP 18 uses pip while pyJMP uses conda packages.

For some cases where .whl (wheels) are not included, conda is still advantageous as it includes specific OS binaries without having to compile.

[#5]

To my surprise, despite what is written in the article, it does interfere.

I am not sure how, but Jupyter finds JMP 18 kernel when the .bat file is called from JSL only via Run Program().

[#6]

JMP 18 installs Python 3.11.5 with JMP 18.0, and 3.11.9 with JMP 18.0.1 security update.  This allows us to provide the deep integration necessary to provide the built-in jmp package.  This was also done due to the numerous problems many had in getting JMP and Python configured to work together, as well as solving the issue of differing Python installations from desktop to desktop.

JMP 18 no longer supports running Python other than distribution supplied with JMP.

[#7]

Thanks, Paul. 

I understand the challenge and appreciate the efforts made in JMP 18. This is why we ended up developing PyJMP and writing temporary text files to pass the data back and forth.

Still, I do not understand how a .bat file executed with RunProgram() interferes with JMP 18 Python env.

Do you have any guesses?

[#8]

The question is, does your add-in depend on JMP running Python code?

I am assuming yes.  JMP 18 only uses the Python that JMP installs.   JMP itself loads the python311 dll.  So if any of your Python code is trying to get JMP to run code that needs to load Python310.dll you will get the error since Python311.dll is already loaded and active within JMP.  We renamed python.exe to jmp_python.exe to prevent Run Program() from accidentally finding JMP's Python when trying to run python.exe or python3.exe, so your batch script shouldn't be trying to run the Python 3.11 executable ( unless you are trying to run this on one of the early JMP 18 EA releases ).

One thing you might have to look at,  is making sure that in your batch file you ensure the PATH to the conda environment's python310.dll precedes the path to the JMP executable directory since the python311.dll resides in the JMP executable directory.   Yet, I would think that if your Python3.10 is executing it would be loading the python310.dll.

[#9 · Accepted solution]

I found the root cause of the problem, following the ideas shared by Paul (thanks!).

RunProgram() starts and includes the JMP installation folder in the PATH.

C:\Program Files\JMP\JMP\18
C:\Program Files\JMP\JMPPRO\18

This folder contains jmp_python.exe, but also python311.dll and others.

Thus, to avoid problems I changed the current directory in my .bat script.