topic Re: Model Screening: Neural network / K-fold crossvalidation in Discussions

Model Screening: Neural network / K-fold crossvalidation

Nimaxim — Sat, 27 Apr 2024 16:20:31 GMT

I need to use model screening to fit a model by Neural network. the default setup for NN is 3TanH. but when I change setup in model launch ( For example: 1TanH 1 Linear 1 Gaussian), it shows me an alert: " Note: The data table has changed. The output no longer represents the data. TO synchronize the output with the data, select "sync to data table changes" from the platform menu."

How can I change the Neural network set up and relaunch the model?

Re: Model Screening: Neural network / K-fold crossvalidation

Byron_JMP — Mon, 29 Apr 2024 19:35:29 GMT

Do you know what you did to change the data between when you ran the first model and when you tried to run it again?

If I run a neural net model, then hide and exclude some rows, then open the model launch and re-run the model, I get the same prompt that you are seeing. From the red triangle menu at the very top outline bar, there is an option to sync the data.

After I click the sync option then I can run other models.

Re: Model Screening: Neural network / K-fold crossvalidation

Victor_G — Tue, 30 Apr 2024 09:28:11 GMT

Hi @Nimaxim,

As answered by @Byron_JMP, there may have been some changes on the table between the two platforms launch that could explain the message you see.

Some side-notes remark :

Have you tried other modeling options ? Neural Networks are complex models, able to approximate any function (you can read about "Universal Approximation Theorem" if you are interested to learn more), but they may quickly overfit, demand relatively large computational ressources, and may not be able to generalize well (without precaution). Because of their very high flexibility, you may see very often Neural Network as one of the model top performer from this Model Screening platform, but I would encourage you to look at other options with similar/closer performances.
Do you have a validation and test strategy ? As mentioned earlier, overfitting with Neural Network is very easy to obtain, so a proper validation strategy is essential to avoid overfitting, and a test set is helpful to validate that the predictions from Neural Network could easily be applied on new/unseen data, with similar properties and distributions as the training and validation sets.
As you have seen, depending on the platform used, there are different settings and neural networks architectures proposed. All the models used in Model Screening platform use default options and tuning parameters in model fitting, so for Neural Network, it's a 1-layer with 3 tanH functions network : Launch the Model Screening Platform (jmp.com)
You may want to fine-tune the architecture of the Neural Network to better fit your use case, and to help you with that, you can use the Add-In created by @scott_allen.

I hope this complementary answer will help you,