Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Re: Tuning Table for estimating Partion Models: Results

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Mar 23, 2018 12:49 PM
(4750 views)

An option for partitioning models (tree, bootstrap, boosted) is to use a tuning design table. The results begin with a table that includes the design points as well as the validation R-squared and the validation RMSE for each design point. I need to have that table also show the training R-squared, training RMSE, test R-squared, and test RMSE. Anyone able to help out here--maybe with a script?

These statistics, and more, are shown below the table for the chosen model, but I need these statistics for all design points. The method to my madness is this: I want to be able to produce a graph showing, say, both validation R2 and training R2 against learning rate to show the amount of overfitting that occurs with increasing learning rate.

1 ACCEPTED SOLUTION

Accepted Solutions

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

I change my mind. It's not too difficult to script up. Here's an example with boosted trees and the Car Physical Data from the sample data library. I created a tuning table for this to try out, which I am attaching. I also put a validation column in the Car Physical Data table.

Here's the script:

```
dt = Data Table("Car Physical Data");
dt_pars = Data Table("TuningTable");
//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");
for(i = 1, i<= N Row(dt_pars), i++,
SPT = dt_pars:SplitsPerTree[i];
LR = dt_pars:Learning Rate[i];
RSR = dt_pars:Row Sampling Rate[i];
CSR = dt_pars:Column Sampling Rate[i];
str = Eval Insert("\[report = (dt << Boosted Tree(
Y( :Weight ),
X(
:Country,
:Type,
:Turning Circle,
:Displacement,
:Horsepower,
:Gas Tank Size
),
Validation( :Validation ),
Method( "Boosted Tree" ),
Splits per Tree( ^SPT^ ),
Number of Layers( 50 ),
Learning Rate( ^LR^ ),
Row Sampling Rate( ^RSR^ ),
Column Sampling Rate( ^CSR^ ),
Go,
invisible
)) << Report;]\");
//Fit the boosted tree
Eval(Parse(str));
//Get the Rsq values and insert them into the results table
R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
report << Close Window;
dt_results:Rsq Training[i] = R2[1];
dt_results:Rsq Validation[i] = R2[2];
);
```

I did a stack of the R2 columns for the results table to make the following graph:

This should be pretty easy to adapt to the other platforms you mentioned.

-- Cameron Willden

9 REPLIES 9

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

I would think you would have to write a script for that, which would effectively need to reproduce the results of the built-in functionality + grab your other desired metrics for the training set. Certainly do-able, but I'm not sure it's worth the effort.

-- Cameron Willden

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

I change my mind. It's not too difficult to script up. Here's an example with boosted trees and the Car Physical Data from the sample data library. I created a tuning table for this to try out, which I am attaching. I also put a validation column in the Car Physical Data table.

Here's the script:

```
dt = Data Table("Car Physical Data");
dt_pars = Data Table("TuningTable");
//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");
for(i = 1, i<= N Row(dt_pars), i++,
SPT = dt_pars:SplitsPerTree[i];
LR = dt_pars:Learning Rate[i];
RSR = dt_pars:Row Sampling Rate[i];
CSR = dt_pars:Column Sampling Rate[i];
str = Eval Insert("\[report = (dt << Boosted Tree(
Y( :Weight ),
X(
:Country,
:Type,
:Turning Circle,
:Displacement,
:Horsepower,
:Gas Tank Size
),
Validation( :Validation ),
Method( "Boosted Tree" ),
Splits per Tree( ^SPT^ ),
Number of Layers( 50 ),
Learning Rate( ^LR^ ),
Row Sampling Rate( ^RSR^ ),
Column Sampling Rate( ^CSR^ ),
Go,
invisible
)) << Report;]\");
//Fit the boosted tree
Eval(Parse(str));
//Get the Rsq values and insert them into the results table
R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
report << Close Window;
dt_results:Rsq Training[i] = R2[1];
dt_results:Rsq Validation[i] = R2[2];
);
```

I did a stack of the R2 columns for the results table to make the following graph:

This should be pretty easy to adapt to the other platforms you mentioned.

-- Cameron Willden

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

The RMSE column will be in numcolbox(2), so you could grab that in the script the same way you grab R2.

To grab values like that out of reports, you just need to figure out where they are in the display tree. You can see how to do that here: https://www.jmp.com/support/help/14/view-the-display-tree.shtml

In my script, I have a variable “report” which holds a reference to the report window from the boosted tree output. Within report, the outline box titled “Overall Statistics” holds the table with the R2s and RMSEs. I get to the numcolbox display element with the R2 values using ‘report[“Overall Statistics”][numcolbox(1)]’

To grab values like that out of reports, you just need to figure out where they are in the display tree. You can see how to do that here: https://www.jmp.com/support/help/14/view-the-display-tree.shtml

In my script, I have a variable “report” which holds a reference to the report window from the boosted tree output. Within report, the outline box titled “Overall Statistics” holds the table with the R2s and RMSEs. I get to the numcolbox display element with the R2 values using ‘report[“Overall Statistics”][numcolbox(1)]’

-- Cameron Willden

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

All is workinig great now, Cameron. Can't tell you how much I appreciate your wisdom!

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

Glad I could help!

-- Cameron Willden

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

Hello, Cameron:

I have modified your script (thanks again!) but am stuck on one last issue. Along with the RSq, RMSE, etc results from boosted trees, I would now like to also retrieve the final number of layers. I thought this would be a straightforward modification of your original script. Please see line 13 from the bottom:

`NL = report["Specifications"][NumberBox(1)] << Get;`

This isn't working, apparently. Do you know if there is something different about the "Specifications" portion of the report versus the "Overall Statistics"?

```
dt = Data Table("C.jmp");
dt_pars = Data Table("table 3400.jmp");
//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Seed");
dt_results << New Column("Final N Layers");
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");
dt_results << New Column("Rsq Test");
dt_results << New Column("RMSE Training");
dt_results << New Column("RMSE Validation");
dt_results << New Column("RMSE Test");
dt_results << New Column("N Training");
dt_results << New Column("N Validation");
dt_results << New Column("N Test");
for(i = 1, i<= N Row(dt_pars), i++,
SED = dt_pars: Seeds[i];
SPT = dt_pars:Splits Per Tree[i];
LR = dt_pars:Learning Rate[i];
RSR = dt_pars:Row Sampling Rate[i];
CSR = dt_pars:Column Sampling Rate[i];
MSS = dt_pars:Minimum Size Split[i];
NL = dt_pars:Number of Layers[i];
str = Eval Insert("\[report = (dt << Boosted Tree(
Y( :PR ),
X(
:Pe,
:To,
:Td,
),
Validation( :Name( "Train/Valid/Test" ) ),
Set Random Seed( ^SED^ ),
Method( "Boosted Tree" ),
Splits per Tree( ^SPT^ ),
Number of Layers( ^NL^ ),
Learning Rate( ^LR^ ),
Row Sampling Rate( ^RSR^ ),
Column Sampling Rate( ^CSR^ ),
Minimum Size Split( ^MSS^ ),
Go,
invisible
)) << Report;]\");
//Fit the boosted tree
Eval(Parse(str));
//Get the Rsq values and insert them into the results table
R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
RMSE = report["Overall Statistics"][NumberColBox(2)] << Get;
N = report["Overall Statistics"][NumberColBox(3)] << Get;
NL = report["Specifications"][NumberBox(1)] << Get;
report << Close Window;
dt_results:Final N Layers[i] = NL[1];
dt_results:Rsq Training[i] = R2[1];
dt_results:Rsq Validation[i] = R2[2];
dt_results:Rsq Test[i] = R2[3];
dt_results:RMSE Training[i] = RMSE[1];
dt_results:RMSE Validation[i] = RMSE[2];
dt_results:RMSE Test[i] = RMSE[3];
dt_results:N Training[i] = N[1];
dt_results:N Validation[i] = N[2];
dt_results:N Test[i] = N[3];
);
```

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results

Looks like NumberBox doesn't recognize "<< Get". Check the Scripting Index for NumberBox and you'll see all the messages that type of object will recognize. One of them is << Get Text; however that will return a string. You can convert it to a number using num().

`num(report["Specifications"][NumberBox(1)] << Get Text);`

-- Cameron Willden

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Tuning Table for estimating Partion Models: Results