turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Discussions
- :
- Neural Networks

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 8:16 AM
(399 views)

How reliable are neural networks? When I didn't set the seed I got different results so I set the seed, but when I change the seed from 1234 to 100 I also get different results. I'm interested in variable importance which completely changes each time I run the model. I've also tried increasing the number of tours but still variable importance changes. I'm using Kfold validation, 3 for hidden layer structure, 10 for number of models, 0.1 for learning rate, transform covariates and robust fit, squared penalty method and 20 number of tours. RSquare is around 0.9 for both training and validation.

Thank you,

Shanice Krombeen

7 REPLIES

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 8:54 AM
(391 views)

Neural networks are reliable predictive models. You are using k-fold cross-validation so, of course, each time you fit the model, you will get different estimates and results. The fact that you are getting practically the same R square for the training and validation sets is a good indication that the model is a good fit.

You can save the fitted model as a column formula. Then you don't have to fit the model again and the estimates won't change when you use the model.

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 9:02 AM
(384 views)

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 9:03 AM
(383 views)

How many total observations do you have in your data set?

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 9:15 AM
(379 views)

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 10:20 AM
(361 views)

So the sample size ranges from 60 to 91? That seems like sufficient data for k-fold CV. Still, if effects are small then the estimates and accompanying variable importance could change a lot with each random assignment to the training and validation sets. Here is a simple case based on the Big Class example and a NN using the default settings except validation uses 5-fold CV.

(First fun)

(Second run)

These two runs use the same data set and settings but involve different random assignments to the five folds. Is this variability similar to what you see?

Why not include **Day** as a predictor and then the data set size will be 209?

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 10:50 AM
(357 views)

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

May 14, 2018 1:08 PM
(339 views)

Fitting the NN by Day means that within each subset, each fold is 12-15 observations, so the stability of the estimates will be adversely affected.

That said, the top predictors are generally the same and they exhibit similar profiles in both runs. You can remove some of these predictors, perhaps as many as half, from the model with little loss in performance but the resulting parsimonious model will be more stable over NN runs.

Also, you are fitting a very complex model. There are 18 hidden nodes. Is the response really that complicated? Over-fitting the model will also affect the stability of the runs.

If the response is time-dependent as you say, then it makes more sense to me to include Day as a predictor and use the larger, combined data set for better stability.

Learn it once, use it forever!