topic Re: Neural networks: validation vs test set in Discussions

Neural networks: validation vs test set

tnad — Tue, 27 Feb 2018 23:56:13 GMT

I know the validation set can be used to inform the model building process and the test set is not touched. For example, in other models, the validation R2 can be used as a stoppage rule. How is the validation set used in neural networks? I can't find any information about this in JMP documentation or online. Thanks.

Re: Neural networks: validation vs test set

txnelson — Wed, 28 Feb 2018 05:01:29 GMT

I typically use the validation results in Neural Nets as a check for over fitting.

Re: Neural networks: validation vs test set

tnad — Wed, 28 Feb 2018 06:41:36 GMT

yes but what makes it different from the test set. It must be used somehow during model building as a check.

Re: Neural networks: validation vs test set

David_Burnham — Wed, 28 Feb 2018 12:53:04 GMT

Some choice to make when building a neural network with JMP Pro:

One or two layers
Number of nodes in each layer
Types of nodes
Quantity and learning rate when applying boosting

All of these choices represent parameters that you choose to tune your model to get the best possible fit. Too highly tuned and you get over-fitting. So the validation set is used to help protect against over-fitting your model as a result of your choice of these tuning parameters.

Since you are using the validation set to help in the choice of model paremeters it is no longer an indepedent assessment of the performance of your model. That's where the test set comes in.

Re: Neural networks: validation vs test set

Mark_Bailey — Wed, 28 Feb 2018 14:08:07 GMT

The training subset of the data is used to fit the neural network model. This model is then used to predict the observations in the validation set. The validation set is in no way used in the parameter estimation. There is no 'stopping rule' because the model does not change (number of terms/parameters) as it would in Partition or Stepwise.

As previously suggested, you compare the performance of the training and validation to assess if you are under- or over-fitting.