BookmarkSubscribe
Choose Language Hide Translation Bar
MAS
MAS
Community Trekker

Tuning Table for estimating Partion Models: Results

An option for partitioning models (tree, bootstrap, boosted) is to use a tuning design table. The results begin with a table that includes the design points as well as the validation R-squared and the validation RMSE for each design point. I need to have that table also show the training R-squared, training RMSE, test R-squared, and test RMSE. Anyone able to help out here--maybe with a script?

 

These statistics, and more, are shown below the table for the chosen model, but I need these statistics for all design points. The method to my madness is this: I want to be able to produce a graph showing, say, both validation R2 and training R2 against learning rate to show the amount of overfitting that occurs with increasing learning rate.

0 Kudos
1 ACCEPTED SOLUTION

Accepted Solutions
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

I change my mind.  It's not too difficult to script up.  Here's an example with boosted trees and the Car Physical Data from the sample data library.  I created a tuning table for this to try out, which I am attaching.  I also put a validation column in the Car Physical Data table.

Here's the script:

dt = Data Table("Car Physical Data");
dt_pars = Data Table("TuningTable");

//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");

for(i = 1, i<= N Row(dt_pars), i++,
	SPT = dt_pars:SplitsPerTree[i];
	LR = dt_pars:Learning Rate[i];
	RSR = dt_pars:Row Sampling Rate[i];
	CSR = dt_pars:Column Sampling Rate[i];
	str = Eval Insert("\[report = (dt << Boosted Tree(
		Y( :Weight ),
		X(
			:Country,
			:Type,
			:Turning Circle,
			:Displacement,
			:Horsepower,
			:Gas Tank Size
		),
		Validation( :Validation ),
		Method( "Boosted Tree" ),
		Splits per Tree( ^SPT^ ),
		Number of Layers( 50 ),
		Learning Rate( ^LR^ ),
		Row Sampling Rate( ^RSR^ ),
		Column Sampling Rate( ^CSR^ ),
		Go,
		invisible
	)) << Report;]\");
	
	//Fit the boosted tree
	Eval(Parse(str));
	
	//Get the Rsq values and insert them into the results table
	R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
	report << Close Window;
	dt_results:Rsq Training[i] = R2[1];
	dt_results:Rsq Validation[i] = R2[2];
);

I did a stack of the R2 columns for the results table to make the following graph:Rsq vs learning rate.PNG

 This should be pretty easy to adapt to the other platforms you mentioned.

 

-- Cameron Willden
9 REPLIES 9
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

I would think you would have to write a script for that, which would effectively need to reproduce the results of the built-in functionality + grab your other desired metrics for the training set.  Certainly do-able, but I'm not sure it's worth the effort.

-- Cameron Willden
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

I change my mind.  It's not too difficult to script up.  Here's an example with boosted trees and the Car Physical Data from the sample data library.  I created a tuning table for this to try out, which I am attaching.  I also put a validation column in the Car Physical Data table.

Here's the script:

dt = Data Table("Car Physical Data");
dt_pars = Data Table("TuningTable");

//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");

for(i = 1, i<= N Row(dt_pars), i++,
	SPT = dt_pars:SplitsPerTree[i];
	LR = dt_pars:Learning Rate[i];
	RSR = dt_pars:Row Sampling Rate[i];
	CSR = dt_pars:Column Sampling Rate[i];
	str = Eval Insert("\[report = (dt << Boosted Tree(
		Y( :Weight ),
		X(
			:Country,
			:Type,
			:Turning Circle,
			:Displacement,
			:Horsepower,
			:Gas Tank Size
		),
		Validation( :Validation ),
		Method( "Boosted Tree" ),
		Splits per Tree( ^SPT^ ),
		Number of Layers( 50 ),
		Learning Rate( ^LR^ ),
		Row Sampling Rate( ^RSR^ ),
		Column Sampling Rate( ^CSR^ ),
		Go,
		invisible
	)) << Report;]\");
	
	//Fit the boosted tree
	Eval(Parse(str));
	
	//Get the Rsq values and insert them into the results table
	R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
	report << Close Window;
	dt_results:Rsq Training[i] = R2[1];
	dt_results:Rsq Validation[i] = R2[2];
);

I did a stack of the R2 columns for the results table to make the following graph:Rsq vs learning rate.PNG

 This should be pretty easy to adapt to the other platforms you mentioned.

 

-- Cameron Willden
MAS
MAS
Community Trekker

Re: Tuning Table for estimating Partion Models: Results

Worked like a charm, Cameron! At the risk of trying your patience, how would I retrieve the RMSEs along with the R2s? Or, more generally for my education, how do I determine the variable names for R2, RMSE, etc. so that I can call them in the script?

0 Kudos
Highlighted
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

The RMSE column will be in numcolbox(2), so you could grab that in the script the same way you grab R2.
To grab values like that out of reports, you just need to figure out where they are in the display tree. You can see how to do that here: https://www.jmp.com/support/help/14/view-the-display-tree.shtml
In my script, I have a variable “report” which holds a reference to the report window from the boosted tree output. Within report, the outline box titled “Overall Statistics” holds the table with the R2s and RMSEs. I get to the numcolbox display element with the R2 values using ‘report[“Overall Statistics”][numcolbox(1)]’
-- Cameron Willden
MAS
MAS
Community Trekker

Re: Tuning Table for estimating Partion Models: Results

All is workinig great now, Cameron. Can't tell you how much I appreciate your wisdom!

0 Kudos
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

Glad I could help!
-- Cameron Willden
0 Kudos
MAS
MAS
Community Trekker

Re: Tuning Table for estimating Partion Models: Results

 Hello, Cameron:

 

I have modified your script (thanks again!) but am stuck on one last issue. Along with the RSq, RMSE, etc results from boosted trees, I would now like to also retrieve the final number of layers. I thought this would be a straightforward modification of your original script. Please see line 13 from the bottom:

NL = report["Specifications"][NumberBox(1)] << Get;

 

This isn't working, apparently. Do you know if there is something different about the "Specifications" portion of the report versus the "Overall Statistics"?

 

dt = Data Table("C.jmp");
dt_pars = Data Table("table 3400.jmp");

//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Seed");
dt_results << New Column("Final N Layers");
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");
dt_results << New Column("Rsq Test");
dt_results << New Column("RMSE Training");
dt_results << New Column("RMSE Validation");
dt_results << New Column("RMSE Test");
dt_results << New Column("N Training");
dt_results << New Column("N Validation");
dt_results << New Column("N Test");

for(i = 1, i<= N Row(dt_pars), i++,
	SED = dt_pars: Seeds[i];
	SPT = dt_pars:Splits Per Tree[i];
	LR = dt_pars:Learning Rate[i];
	RSR = dt_pars:Row Sampling Rate[i];
	CSR = dt_pars:Column Sampling Rate[i];
	MSS = dt_pars:Minimum Size Split[i];
	NL = dt_pars:Number of Layers[i];
	str = Eval Insert("\[report = (dt << Boosted Tree(
		Y( :PR ),
		X(
		:Pe,
		:To,
		:Td,
	),
	Validation( :Name( "Train/Valid/Test" ) ),
	Set Random Seed( ^SED^ ),
		
		Method( "Boosted Tree" ),
		Splits per Tree( ^SPT^ ),
		Number of Layers( ^NL^ ),
		Learning Rate( ^LR^ ),
		Row Sampling Rate( ^RSR^ ),
		Column Sampling Rate( ^CSR^ ),
		Minimum Size Split( ^MSS^ ),
		Go,
		invisible
	)) << Report;]\");
	
	//Fit the boosted tree
	Eval(Parse(str));
	
	//Get the Rsq values and insert them into the results table
	R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
	RMSE = report["Overall Statistics"][NumberColBox(2)] << Get;
	N = report["Overall Statistics"][NumberColBox(3)] << Get;
	NL = report["Specifications"][NumberBox(1)] << Get;
	report << Close Window;
	dt_results:Final N Layers[i] = NL[1];
	dt_results:Rsq Training[i] = R2[1];
	dt_results:Rsq Validation[i] = R2[2];
	dt_results:Rsq Test[i] = R2[3];
	dt_results:RMSE Training[i] = RMSE[1];
	dt_results:RMSE Validation[i] = RMSE[2];
	dt_results:RMSE Test[i] = RMSE[3];
	dt_results:N Training[i] = N[1];
	dt_results:N Validation[i] = N[2];
	dt_results:N Test[i] = N[3];
);

 

0 Kudos
cwillden
Super User

Re: Tuning Table for estimating Partion Models: Results

Looks like NumberBox doesn't recognize "<< Get".  Check the Scripting Index for NumberBox and you'll see all the messages that type of object will recognize.  One of them is << Get Text; however that will return a string.  You can convert it to a number using num().

num(report["Specifications"][NumberBox(1)] << Get Text);
-- Cameron Willden
MAS
MAS
Community Trekker

Re: Tuning Table for estimating Partion Models: Results

I'm learning a lot about the ins and outs of JMP's script, Cameron. Once again, this works just fine. Thanks to you again. I greatly appreciate your help.

0 Kudos