topic Re: Problems with Reproducibility in Discussions

Problems with Reproducibility

onurseref — Sun, 23 Mar 2025 16:28:04 GMT

I teach using JMP and I became aware of some reproducibility problems in the following cases:

1) My students who use different operating systems like windows and MacOS get different results from models that require randomization like NN even if they use the same random seed like 123. This problem is frustrating because assignments that use a random seed for reproducibility reasons are supposed to have a unique true solution for comparison, and different results confuse students.

2) Some methods like SVM has a conventional tuning design to find a good combination of parameters; however, no random seed option is offered in the tuning design interface, which leads to different results that, again, confuse students.

I'm not sure if these are known issues. Nevertheless, I would appreciate any feedback that can address them.

Re: Problems with Reproducibility

Ben_BarrIngh — Wed, 23 Apr 2025 12:41:02 GMT

Hi @onurseref,

I think this is a good case to bring to our support team at support@jmp.com - if you email them they will be happy to help.

Thanks,

Ben

Re: Problems with Reproducibility

Ross_Metusalem — Fri, 25 Apr 2025 19:14:29 GMT

Hi @onurseref, for tuning SVM or other scenarios where no random seed is available inside the modeling platform, you and your students can use the Random Seed Reset Add-In. Set a random seed before launching the platform, and you'll get the same SVM tuning design each time.

The add-in page mentions a known issue with random seeds across machines and operating systems. As Ben has suggested, you can get more information on that from our Technical Support team.

Re: Problems with Reproducibility

onurseref — Fri, 25 Apr 2025 20:57:25 GMT

Thanks @Ross_Metusalem for the Random Seed Reset Add-in solution. I'll give it a shot to see if it alleviates some of the reproducibility problems. I understand that some issues might remain due to architectural differences between operating systems. I'll keep my fingers crossed.

Re: Problems with Reproducibility

Mark_Zwald — Sun, 27 Apr 2025 07:34:15 GMT

Different CPU architectures will produce different results from a random number generator, even with the same seed value. In this case a M series CPU from Apple versus x86 CPU from Intel vs x86 CPU from AMD. One option is you could generate M series MacOS, Windows Intel and Windows AMD results for the same seed for your assignments.

Re: Problems with Reproducibility

PatrickGiuliano — Mon, 28 Apr 2025 01:45:19 GMT

Thank you @Ben_BarrIngh we picked it up for additional discussion with this customer.

Cheers,

-Patrick Giuliano (JMP Technical Support)