Hi, I have a problem when doing k-fold cross-validation with stepwise regression: I want to run the k-fold cross-validation with stepwise regression for 100 times using same dataset and save the R squares for both the training and the validation sets. The results table of JMP include two columns, “RSquare” and “RSquare K-Fold”. Although I can get different “RSquare K-Fold”, it seems that every time the “RSquares” are the same. That means maybe the “RSquare” is not the R square for the training set and the “RSquare K-Fold” is not the R square for the validation set.
So, how could I get the two values (i.e., R squares for the training and the validation set)? How could I change my script to get these values?
I found that in the option of the "Plot R square history" we can get a graph of the R squares of the training and validation sets. Then how can I get the value? It seems that it only gives a graph (see the Figure).
Here is my script:
names default to here(1);
dt=Current Data Table();
dtb=New Table( "K-Fold Results",
Add Rows( 0 ),
New Column( "StringColBox",
Character,
"Nominal"
)
);
For( i = 1, i <= 100, i++,
obj = dt << Fit Model(
Y( :Std FP ),
Effects(
:Std wl,
:Std on,
:Std hfn,
:Std odc,
:Std cvq,
:Std son,
:lNo,
:wNo
),
Personality( Stepwise),
Run,
invisible
);
obj << Name( "K-Fold Crossvalidation" )(10);
obj << Finish;
kfold = obj << report;
tablebox = kfold[Table Box( 3 )];
dt1=tablebox <<make into data table();
dt1<<set name("K-Fold Results "||char(i));
dt1<<new column("Iteration",formula(i));
dtb << Concatenate( dt1, "Append to first table" );
close(dt1, nosave);
);