Generally speaking you can recreate much of the contents in piecemeal fashion within JMP wrt to the report elements generated by the JMP Pro Model Comparison platform. The biggest challenge is you'll have LOTS of JMP report windows open simultaneously and comparing specific report elements across the windows may get visually overwhelming.
For example, to recreate say, the JMP Pro Model Comparison report element for Actual vs. Predicted value scatter plot (which I find to be one of the most telling visualizations when comparing model performance in the Model Comparison platform) you'd have to dump each model's actual vs. predicted values into separate columns for the predicted values in one central data table and then maybe some kind of Overlay plot in Graph Builder? For the Prediction Profiler, you'd have to one by one manually manipulate each independent model's Profiler slider bars. Yikes...by the time you cajole all this...over multiple separate projects or practical problems...I'm thinking your time is the most costly commodity wrt to your company...and you've paid for a JMP Pro license in just your time.
I do not know of a JMP Add In that recreates this environment.