Subscribe Bookmark RSS Feed

How to generate a Learning Curve Plot for a machine learning method?

UserLightNew

New Contributor

Joined:

Aug 28, 2017

Dear JMP users,

 

Is it possible to generate a plot of learning curve such as shown below? Both scripting answer or interactive answer would be greatly appreciated.

 

For example, for a machine learning method (e.g. linear regression), I would like to plot error rate in the Y-axis and training set size in the X-axis.

LearningCurve.png

(figure source: http://blog.revolutionanalytics.com/2015/09/why-big-data-learning-curves.html) 

Also, is it possible to plot such learning curve for multiple models (e.g., different regression models) in the same plot?

 

 

5 REPLIES
markbailey

Staff

Joined:

Jun 23, 2011

Yes, it is possible to generate this 'learning curve' with JMP. Scripting this simulation would probably be the best since the sample size is the simulation parameter. You might be able to use a formula with the simulation feature in JMP Pro.

Yes, you should be able to repeat this simulation with any modeling approach. You just capture a measure of the error for the training set and the validation set.

The blog post provides the R script, so you could use it, as is, if you have R installed on the same computer as JMP. A short JMP script is necessary to run R with this script.

Learn it once, use it forever!
UserLightNew

New Contributor

Joined:

Aug 28, 2017

Thanks for your reply, Mark. I see running R script inside JMP would be an option.

I was wondering if you could point me to the general approach to or a brief outline of steps I need to take to use the similuation feature, which appears to be something new.

David_Burnham

Super User

Joined:

Jul 13, 2011

I have some examples in the following blog entries, together with some sample scripts:

 

http://www.pega-analytics.co.uk/blog/logistic-regression-pt-2/ 

http://www.pega-analytics.co.uk/blog/stepping-carefully/

 

 

-Dave
UserLightNew

New Contributor

Joined:

Aug 28, 2017

This is super helpful, Dave. Thank you.

markbailey

Staff

Joined:

Jun 23, 2011

I would only add to Dave's wonderful offer that the approach taken in the blog post that you cited could be used with JMP. The functions are different but there is nothing special or unique about the approach. I would start with Dave's links, though.

Learn it once, use it forever!