Neural Networks

[#1]

How reliable are neural networks? When I didn't set the seed I got different results so I set the seed, but when I change the seed from 1234 to 100 I also get different results. I'm interested in variable importance which completely changes each time I run the model. I've also tried increasing the number of tours but still variable importance changes. I'm using Kfold validation, 3 for hidden layer structure, 10 for number of models, 0.1 for learning rate, transform covariates and robust fit, squared penalty method and 20 number of tours. RSquare is around 0.9 for both training and validation. 

Thank you, 

Shanice Krombeen 

[#2]

Yes sample size ranges from 60 to 91. The variables are expression of different nutrient transporters in placenta or endometrium on day 70, 90, or 110 of gestation. Day is a significant effect and expression likely differs by day, which is why I fit the model by day. It is true that the effects are small so that may be why variable importance changes a lot. Here is an example of what I'm seeing. 

Neural Network Run 1Neural Network Run 2

[#3]

Fitting the NN by Day means that within each subset, each fold is 12-15 observations, so the stability of the estimates will be adversely affected.

That said, the top predictors are generally the same and they exhibit similar profiles in both runs. You can remove some of these predictors, perhaps as many as half, from the model with little loss in performance but the resulting parsimonious model will be more stable over NN runs.

Also, you are fitting a very complex model. There are 18 hidden nodes. Is the response really that complicated? Over-fitting the model will also affect the stability of the runs.

If the response is time-dependent as you say, then it makes more sense to me to include Day as a predictor and use the larger, combined data set for better stability.