JMP and now JMP Pro have always computed RMSE the same way: as defined by the analysis of variance. It is the square root of the mean square error. The mean square error is the quotient of the sum of squares error divided by the degrees of freedom of the error. The degrees of freedom of the error is the degrees of freedom for the corrected total minus the degrees of freedom of the model. The degrees of freedom for the corrected total is N - 1. (The correction for the mean response costs one degree of freedom.)
Here is a simple linear regression of weight versus height using the Big Class data table in the Sample Data folder:
The degrees of freedom for the error is DF(CT) - DF(M) = 39 - 1 = 38. A single parameter is required for height.
Here is a multiple regression of weight versus age, sex, and height:
The degrees of freedom for the error is DF(CT) - DF(M) = 39 - 7 = 32. Seven parameters are required for age, sex, and height.
It's actually divided by the total sample size minus the number of model terms (n - df_model). For example, if I do an ANOVA with 15 data points and 1 categorical factor with 3 levels, the model will have 1 intercept term and 2 model terms for the categorical factor (3 total). So, the RMSE will be the sqrt(SSE/(15-3)).