I emailed JMP Technical Support these questions and they replied as follows:
- Yes, the RMPRESS statistic is calculated from the results from different folds. And then the model with the smallest values of RMPRESS is chosen as the best model.
- RMPRESS is the square root of the average squared residual from this prediction.
Hopefully this will help anyone else with similar questions.