cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
JMP is taking Discovery online, April 16 and 18. Register today and join us for interactive sessions featuring popular presentation topics, networking, and discussions with the experts.
Choose Language Hide Translation Bar
tnad
Level III

Neural networks: validation vs test set

I know the validation set can be used to inform the model building process and the test set is not touched. For example, in other models, the validation R2 can be used as a stoppage rule. How is the validation set used in neural networks?  I can't find any information about this in JMP documentation or online. Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions
David_Burnham
Super User (Alumni)

Re: Neural networks: validation vs test set

Some choice to make when building a neural network with JMP Pro:

  • One or two layers
  • Number of nodes in each layer
  • Types of nodes
  • Quantity and learning rate when applying boosting

All of these choices represent parameters that you choose to tune your model to get the best possible fit.  Too highly tuned and you get over-fitting. So the validation set is used to help protect against over-fitting your model as a result of your choice of these tuning parameters.

Since you are using the validation set to help in the choice of model paremeters it is no longer an indepedent assessment of the performance of your model.  That's where the test set comes in. 

 

 

-Dave

View solution in original post

4 REPLIES 4
txnelson
Super User

Re: Neural networks: validation vs test set

I typically use the validation results in Neural Nets as a check for over fitting.

Jim
tnad
Level III

Re: Neural networks: validation vs test set

yes but what makes it different from the test set. It must be used somehow during model building as a check.

David_Burnham
Super User (Alumni)

Re: Neural networks: validation vs test set

Some choice to make when building a neural network with JMP Pro:

  • One or two layers
  • Number of nodes in each layer
  • Types of nodes
  • Quantity and learning rate when applying boosting

All of these choices represent parameters that you choose to tune your model to get the best possible fit.  Too highly tuned and you get over-fitting. So the validation set is used to help protect against over-fitting your model as a result of your choice of these tuning parameters.

Since you are using the validation set to help in the choice of model paremeters it is no longer an indepedent assessment of the performance of your model.  That's where the test set comes in. 

 

 

-Dave

Re: Neural networks: validation vs test set

The training subset of the data is used to fit the neural network model. This model is then used to predict the observations in the validation set. The validation set is in no way used in the parameter estimation. There is no 'stopping rule' because the model does not change (number of terms/parameters) as it would in Partition or Stepwise.

As previously suggested, you compare the performance of the training and validation to assess if you are under- or over-fitting.