cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
JMP is taking Discovery online, April 16 and 18. Register today and join us for interactive sessions featuring popular presentation topics, networking, and discussions with the experts.
Choose Language Hide Translation Bar
triunk
Level I

Repeated k-fold Cross-Validation with Stepwise Regression (JSL Script?)

Hi all,

Many of us know the variability that k-fold cross validated stepwise regression might have with small samples. It is often suggested to repeat this procedure 50-100 times in order to increase the realibility of the outputs.

I was wondering if there is a way in JMP to repeat this procedure? It will save me TONS of time!

THANK YOU

1 ACCEPTED SOLUTION

Accepted Solutions
sseligman
Staff

Re: Repeated k-fold Cross-Validation with Stepwise Regression (JSL Script?)

Below is an example script that repeats k-fold crossvalidation on the same stepwise model 10 times. Copied from the JMP Scripting Index, the first part of this script runs a stepwise regression model and then performs a k-fold crossvalidation with 5 folds. The second piece makes the table box containing the step history and best model statistics into a data table for each iteration, and concatenates all the data tables together into one K-Fold Results table. You can of course choose to make a different table box of results into an output table, or none at all. 

 

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Fitness.jmp" );
For( i = 1, i <= 10, i++,
obj = dt << Fit Model( Y( :Oxy ), Effects( :Runtime, :Weight, :RunPulse, :RstPulse, :MaxPulse ), Personality( Stepwise ), Run );
obj << Name( "K-Fold Crossvalidation" )(5);
obj << Finish;

kfold = obj << report;
tablebox = kfold[Table Box( 3 )];

output = tablebox << Make Into Data Table( "K-Fold Results" );
If( i != 1,
Data Table( "K-Fold Results" ) << Concatenate( Data Table( "K-Fold Results " || Char( i ) ), Append to first table );
);
);

 

View solution in original post

1 REPLY 1
sseligman
Staff

Re: Repeated k-fold Cross-Validation with Stepwise Regression (JSL Script?)

Below is an example script that repeats k-fold crossvalidation on the same stepwise model 10 times. Copied from the JMP Scripting Index, the first part of this script runs a stepwise regression model and then performs a k-fold crossvalidation with 5 folds. The second piece makes the table box containing the step history and best model statistics into a data table for each iteration, and concatenates all the data tables together into one K-Fold Results table. You can of course choose to make a different table box of results into an output table, or none at all. 

 

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Fitness.jmp" );
For( i = 1, i <= 10, i++,
obj = dt << Fit Model( Y( :Oxy ), Effects( :Runtime, :Weight, :RunPulse, :RstPulse, :MaxPulse ), Personality( Stepwise ), Run );
obj << Name( "K-Fold Crossvalidation" )(5);
obj << Finish;

kfold = obj << report;
tablebox = kfold[Table Box( 3 )];

output = tablebox << Make Into Data Table( "K-Fold Results" );
If( i != 1,
Data Table( "K-Fold Results" ) << Concatenate( Data Table( "K-Fold Results " || Char( i ) ), Append to first table );
);
);