Training and Validation R2s in Lasso/Elastic net

[#1]

Dear all,

I use k-fold cross-validation in combination with penalized regression models (lasso, elastic net) and want to examine the (averaged) R2 values of the training/validation sets to check for overfitting and explained variance. I already searched around with the many options, but I can not find that. Is there a way to show this? It would be great, if that could be shown in the overview table, so that I can use it for model selection. 

Thank you, Markus

[#2]

Hi @MarkusJH,

Welcome in the Community !

To answer your question, one option could be to use the platform "Model Screening" available in menu "Analyze", "Predictive Modeling".

Once in the "Model Screening" platform, you can deselect methods other than "Generalized Regression" and "Additional Methods" (here done on the Titanic dataset), and specify if you want to include 2 factors interactions or quadratic terms in the model. You can also specify the number of folds for cross-validation and a random seed if you want reproducible results : 

When the analysis is done, you can see the validation results as a summary across the folds. This summary table can help you select the most appropriate model(s) (you can select the model(s) you want to see in more details (or click on "Select Dominant") and then click on "Run Selected" to open the platform of Generalized Regression to have more details about the selected model(s)):

And you can also have a look at individual results by folds (if needed) by opening the menu "Training" and "Validation" (here on "Training", folds identifier can be seen in the last column for each type of model trained):

I hope  I understood your question and that this answer will help you.
If I misunderstand something and this was not what you expect, could you provide more details on which platform do you currently use, what info do you already have and which ones are missing ?

[#3]

Thank you, but of course I know the model screening platform, but it runs simplified models and it does not provide the options of the full models. For instance, I assume that the neural model in model screening is a rather simple one with only a few hidden units.

When I want to do nested cross-validation with a more complex neural model with 2 layers, for instance, this can not be done with the model screening, isn't it?

There also other things that may be done within in the inner loop only. For instance, imputation is often done within the inner loop only, because it would be unfair to use the whole sample including the independent test sample for imputation.

Therefore, I am looking for a script, with which nested cross-validation can be done. I could then, for instance, insert imputation within the inner loop only. 

[#4]

... or another thing, one wants to do in the inner loop would be hyperparameter selection, e.g. the number of hidden units. 

Of course, at the end of the inner loop, there must be a criterion that defines the best model, which is then submitted to the outer loop cross-validation.