building multiple neural network models

[#1]

I am using the neural platform to make predictions, and, being new to neural networking, I am only slightly familiar with all the input parameters: number of hidden layers, number of nodes/hidden layer, boosting models, learning rate, and tours. What I want to do is try to minimize RMSE and the validation model misclassification rate. What I've been doing is iteratively changing each parameter one by one, saving the model performance parameters, and pasting them into a new JMP table, but this is going to take days since there are so many combinations of layers, nodes, tours, etc. Would it be possible to write a script to where JMP Pro builds, say, 1,000 models and dumps the data into a table so that I don't have to manually change each model input parameter? 

#Hidden layers: 1 or 2

#Sigmoidal nodes: 0, 1, 2, 3, or 4

#Linear nodes: 0, 1, 2, 3, or 4

#Radial nodes: 0, 1, 2, 3, or 4

#boosting models: no idea, 1-5 maybe?

#learning rate: 0-0.5 in 0.1 increments

#tours: 1, 5, 10, 20, 100

So that would be 2 x 5 x 5 x 5 x 5 x 5 x 5=~30,000. 

I guess I could widdle a few of these down once I am more familiar with the dataset. Of course, having many nodes and a high number of boosting models doesn't make sense (nor do certain other combinations), but we're still talking about potentially hundreds of models worth testing. Surely this sort of model screening/comparing could be scripted, right?

[#2]

Hi @abmayfield ,

  I have to say, that is some strange behavior for it to work with an Air and not a Pro. If I remember correctly, that o_box call is to extract some of the statistics of the fit and it can change depending on which type of validation method you're using: holdback, excluded rows, etc. It doesn't sound like it's a JMP version since you're running 16.1 on both. Are you using the same validation method for both computers? Which method(s) are you using?

  My only guess is that if it is not working from one Macbook model to the next that there must be a change in the tree structure of the report and the JSL code is looking for a value to be in a specific location, but it's coming back empty or NAN, or something and it throws the error. This is maybe a bit beyond my coding skills -- to make it OS-independent might be beyond me. I'm running on a PC, Windows10. I have no doubt there are more skilled programmers at JMP that could figure out how to get around this issue so that it can run on different OSs and so forth.

  If I can reproduce the error, then it's relatively easy for me to fix, but I don't have access to Macbook Pro. In the meantime, can you run it on the Macbook Pro and check the log file to see if it will tell you which line of code it is breaking down at? This might help me to narrow down the location of the bug and maybe see how to fix it.

Thanks!,

DS

[#3]

I am attaching the log and the table with the GUI (the last script). I tried all the validation types that made sense for several of the things I am trying to predict (the Y's). Normally, I would just do everything on my personal Macbook (where everything works), but that computer is weeks away from biting the dust, so I hope i can get everything working on this newer, nicer Macbook Pro! In the mean time, I am going to try a few things. There is a slight chance that I might have just copied and older version of the GUI (that still had bugs), which would be a simple fix, though I don't think that is the issue since, regardless, it worked on the other computer. I'll be curious if you find anything weird, and thanks again for looking into this. 

[#4]

I tried JMP Pro 15 and still got the same error message: 

expected character arg 1 in access or evaluation of 'Outline Box' , Outline Box/*###*/(o_box)

I am using GUI version 3 from around July 10th or so. That was the most recent, right?

When I ran the debugger, it told me: "Object Button Box does not recognize the message send; maybe you mean one of these..." then it lists some alternatives. But the Button Box forms fine, and all the buttons work. 

I would think your tree structure argument COULD be the issue but it's the same OS (Catalina). I'm going to reach out to technical support. 

[#5]

Hi @abmayfield ,

  Thanks for files, it helped me to figure out where the mistakes were coming in. It has to do with my mediocre programming and complete understanding of the tree structures within JMP (I'm all self taught at this). Anyway, the date for the latest version is just about right, I had updated things around early July to account for the fact that the fit statistics change their position in the tree structure based on what kind of modeling data the Y values are, e.g. numerical, ordinal, or categorical. I'm working with purely numerical responses almost 100% of the time, so I didn't see this error on a regular basis. In order to correctly provide information on the fit statistics for other modeling types of data, I needed to modify how it assigns that variable. Thankfully, it was all rather logical from the JMP side, so the modifications weren't that hard.

  Attached is the ver3.0 that should fix this no matter your OS or JMP version you have. I don't think support will be able to help much since this is a "third party" code in a sense. Anyway, please give this ver3.0 a try and let me know if it continues to misbehave.

Thanks!,

DS

[#6]

Awesome! Let me try it now and report back. As it turns out, JMP support DID come to the same conclusion, so I feel confident it will work now. 

[#7]

I just tried on the Macbook Pro and got the same outline box/arg 1 error, but I noticed something: the July version was named "3.0." Do you think you updated it but then accidentally uploaded the old version? 

[#8]

Another thing that might be informative: even if I have a continuous Y, I get the same message (regardless of validation type). Strange!

[#9]

Hi @abmayfield ,

  Sorry to hear that you continue to get the error. The "GenAT_ver3.0.jsl" file I uploaded is the one I'm using. I have tried it on my work (JMP 16.0 Pro) and personal PC (JMP EA17), and it works on both. I don't get the error on either PC when trying to fit your data. It must be something about how the Macbook Pro deals with the fit report window. I am stumped on how to fix it without having access to the specific computer. I'd have to look at the tree structure of the report window and try and figure out what the difference is between a PC, Macbook, and Macbook Pro.

  I went back and took a closer look at your log file and thought maybe it was something about the specific DOE that you generated, but even when I run that DOE and make a tuning table based on it, it works. At least it works on the PCs I'm using.

  That's to bad, as it sounds like you might not be able to demo the GUI for your talk if it's not running on the Macbook Pro. I wish I could be of more help on this issue.

DS

[#10]

You know what I've been learning this week, which is literally the most time I've ever spent on my work Macbook Pro? There are LOTs of small, annoying, but mostly trivial bugs in JMP Pro 16.1 that are completely absent on my Macbook Air. And all have to do with aesthetics. For instance, you know how you can group scripts on the left, then click to expand the grouped folder? That doesn't work. Well, it works sometimes. You need to click the expand arrow, then click somewhere else on the screen, return, click again, and it eventually expands. There are other small, weird things like that they I never noticed on the same OS and same version of JMP Pro on my other Mac, so it must be some issue with JMP and this particular computer communicating with one another, which then affects your script since it seems to have something to do with appearance. In other words, I don't think it's because of your incompetence, haha, and I fully expect to still use the new GUI on my other computer. In fact, it is running through 2,000 NN models as I type this on my work computer. I just need to eventually figure out these Macbook Pro issues given the decaying nature of my Macbook Air. In fact, I may start logging all these little bugs and quirks and try to talk to some developers at the Discovery Summit. I wonder if it's truly MY computer or a general Macbook Pro issue. I don't want to upgrade to the newest Mac OS (Big Sur), but I wonder if this could actually be the solution.

[#11]

Hi @abmayfield ,

  Interesting, well, it certainly seems as if this issue is truly related to the Macbook Pro (and perhaps OS that it's running). It possibly could be OS or maybe even hardware related -- like how JMP uses the graphics processing unit on a Pro is somehow different than how it uses it on an Air. Maybe check to see if Mac made the two computers with the same GPU or if could be a hardware issue. At any rate, I would imagine JMP developers would want to know more about the specifics you're facing, and bringing it to their attention at the Discover Summit would be helpful. At least it's not my incompetence!

  I'm glad you're still able to run it on your Air. I hope your Air continues to work so you can make your video and submit it for the Summit. Let me know you're time slot for the talk, would like to see it.

Thanks!,

DS