Problems with Reproducibility

[#1]

I teach using JMP and I became aware of some reproducibility problems in the following cases:

1) My students who use different operating systems like windows and MacOS get different results from models that require randomization like NN even if they use the same random seed like 123. This problem is frustrating because assignments that use a random seed for reproducibility reasons are supposed to have a unique true solution for comparison, and different results confuse students.

2) Some methods like SVM has a conventional tuning design to find a good combination of parameters; however, no random seed option is offered in the tuning design interface, which leads to different results that, again, confuse students.

I'm not sure if these are known issues. Nevertheless, I would appreciate any feedback that can address them.

[#2]

Hi @onurseref,

I think this is a good case to bring to our support team at support@jmp.com - if you email them they will be happy to help.

Thanks,

Ben

[#3]

Thank you @Ben_BarrIngh we picked it up for additional discussion with this customer.   

Cheers,

-Patrick Giuliano (JMP Technical Support)

[#4 · Accepted solution]

Hi @onurseref, for tuning SVM or other scenarios where no random seed is available inside the modeling platform, you and your students can use the Random Seed Reset Add-In. Set a random seed before launching the platform, and you'll get the same SVM tuning design each time. 

The add-in page mentions a known issue with random seeds across machines and operating systems. As Ben has suggested, you can get more information on that from our Technical Support team.

[#5]

Thanks @Ross_Metusalem for the Random Seed Reset Add-in solution. I'll give it a shot to see if it alleviates some of the reproducibility problems. I understand that some issues might remain due to architectural differences between operating systems. I'll keep my fingers crossed.

[#6]

Different CPU architectures will produce different results from a random number generator, even with the same seed value.  In this case a M series CPU from Apple versus x86 CPU from Intel vs x86 CPU from AMD.  One option is you could generate M series MacOS, Windows Intel and Windows AMD results for the same seed for your assignments.