Try the Materials Informatics Toolkit, which is designed to easily handle SMILES data. This and other helpful add-ins are available in the JMP® Marketplace
The RMSE is available from the Summary of Fit table after using ANOVA command.
It is a pooled estimate of the within group variance, not the overall standard deviation. The latter estimate would include between group variation, too.