Best Neural Net model

[#1]

Is it possible to get the best neural net model (maybe knowing the seed) and save the script to be used again in the future? 

[#2]

Hi @Matheus_Plana,

Did you try the Neural Network Tuning Add-In by @scott_allen ?

Thanks to this add-in, you have the possibility to automatically tune neural networks hyperparameters for a specific dataset and validation strategy, and check the reproducibility of the results with different random seeds.

Hope this answer will help you, 

[#3]

Thanks for the answer, Victor! 

I didn't try it, but I saw it. The add-in is for JMP Pro, but my client is using the JMP Standard and asked me if it was possible to run it there. 

It was during a JMP Training and I had no answer to that question. 

Do you know if there is something like that for JMP Standard?

Best regards!

[#4]

Hi @Matheus_Plana,

May I ask why you (or your client) choose a Neural Network model ?

There are many other models available, and depending on your dataset size, representativeness/homogeneity and quality, there may be other (Machine Learning or others) models more adapted (in terms of performance & complexity in relation with your objective). See other topics about this question :

Model Screening: Neural network / K-fold crossvalidation 

help with model comparsion: DOE vs ANN boosted 

...

In JMP, you can build Neural Networks, but most of the options require JMP Pro, so you will only investigate a small subset of all possible NN models, so I wonder if it is very relevant to try optimizing a NN on this very narrow hyperparameters space : Launch the Neural Platform

I guess that you could automatize the testing of various NN models with JMP only options (only TanH activation function, one layer only, ...), but it would require some scripting to automatize these steps :

1. Define the min and max levels of the hyperparameters available in JMP-only options,
2. Create a Space-Filling design with the hyperparameters defined as factors,
3. Fit all NN models from this Space-Filling design,
4. Agregate metrics performances to compare models.

Hope this answer will help you,

[#5]

Victor, thanks a lot for sharing valuable ideas! 

In this case I was running a JMP Training, explaining to the customer how to use different models. 

When it came to the Neural Net, the customer was very impressed and during my demonstration he asked this specific question. So I was not sure about how to answer and came here in the community! 

But I TOTALLY agree with you! We have lots of different models and the neural should be used with caution.

All the best!

[#6]

Hey @Matheus_Plana ,

You could try using this script I made for my Non-pro users to try and squeak out some NN tuning.  This allows you to see which model does the best with different Qty of Tanh nodes as well as try different validation percentages.  It also shows what variables are most important across all the models.

Let me know if you have any questions on it!

Steve

[#7]

Thanks for sharing @shampton82 !

I'm impressed by some of the visuals that are very clear and easy to understand, for example this one enabling to evaluate predictive performances and robustness/generalization properties to a validation set : 

A little recoding of the informations from this attached table could help visualizing performances trend depending on the number of TanH activation functions/nodes : 

Maybe some parts could be interesting to consider for updating the Neural Network Tuning add-in @scott_allen ?

Thanks a lot for sharing !

[#8]

These are some nice visuals. Its really challenging to summarize so much information into simple graphs and @shampton82 has some really nice examples.

I am working on a similar add-in for Torch right now. There are even more parameters to screen and tune and I think similar visuals might work well when scanning across model types and activation functions.