How to generate a Learning Curve Plot for a machine learning method?

[#1]

Dear JMP users,

Is it possible to generate a plot of learning curve such as shown below? Both scripting answer or interactive answer would be greatly appreciated.

For example, for a machine learning method (e.g. linear regression), I would like to plot error rate in the Y-axis and training set size in the X-axis.

(figure source: http://blog.revolutionanalytics.com/2015/09/why-big-data-learning-curves.html) 

Also, is it possible to plot such learning curve for multiple models (e.g., different regression models) in the same plot?

[#2]

Yes, it is possible to generate this 'learning curve' with JMP. Scripting this simulation would probably be the best since the sample size is the simulation parameter. You might be able to use a formula with the simulation feature in JMP Pro.

Yes, you should be able to repeat this simulation with any modeling approach. You just capture a measure of the error for the training set and the validation set.

The blog post provides the R script, so you could use it, as is, if you have R installed on the same computer as JMP. A short JMP script is necessary to run R with this script.

[#3]

Thanks for your reply, Mark. I see running R script inside JMP would be an option.

I was wondering if you could point me to the general approach to or a brief outline of steps I need to take to use the similuation feature, which appears to be something new.

[#4]

I have some examples in the following blog entries, together with some sample scripts:

http://www.pega-analytics.co.uk/blog/logistic-regression-pt-2/ 

http://www.pega-analytics.co.uk/blog/stepping-carefully/

[#5]

This is super helpful, Dave. Thank you.

[#6]

I would only add to Dave's wonderful offer that the approach taken in the blog post that you cited could be used with JMP. The functions are different but there is nothing special or unique about the approach. I would start with Dave's links, though.