topic Re: Graphical User Interface to Automate the Process of Model Tuning in JMP Scripters Club Discussions

Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Fri, 20 May 2022 12:55:04 GMT

Graphical User Interface to Automate the Process of Model Tuning

As part of Innovation Management at Evonik, we are interested in developing predictive models for some of our quality performance parameters. JMP Pro’s predictive modeling platforms provides a direct path to just that for their practical, real-world problems. However, given the multiple different platforms available: from Neural Nets to Support Vector Machines, and the latest in the XGBoost platform, it can be daunting not only to find which model performs the best predictively, but also in tuning a model. To address this and to simplify the process of tuning models for each platform, I developed a graphical user interface that helps to automate the process of model tuning. The code is written entirely in JSL and utilizes different modeling platforms as well as the DOE platform within JMP. The code spans everything from simple calls and references to more complicated tasks like clicking an “OK” button, extracting data from reports, and passing data from one window to another. The code could not have been possible without the generous help from the JMP Community pages.

(view in My Videos)

Re: Graphical User Interface to Automate the Process of Model Tuning

jpol — Mon, 23 May 2022 12:10:36 GMT

Thank you Diedrich for presenting this in last week's JMP Scripter's Club.

I would have a couple of requests:

You mentioned that you had edited the JMP Sample data used in your demonstration.

Would it be possible to get a copy of the edited table posted here.

In your demonstration you showed the results for tuning a NN.

Could you run your analysis , using the same data, on Bootstrap Forest and post your results here.

Many thanks for your contributions to the community.

Philip

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Tue, 24 May 2022 15:27:06 GMT

Hi @jpol ,

Sure, not a problem. Attached to this post is the modified Powder Metallurgy.jmp file from their sample data.

Open up the JMP data table and the JSL file (be sure to be in the data table when running the JSL or alternatively, be sure to select the correct data table used for modeling from the pull-down menu in the JSL window -- see screenshot).

Click on the "Select Data Table" button to select the data table -- this is needed to feed column names to the next window. Select table and click "OK".

Select the modeling method of interest, in this case Bootstrap Forest, then click "OK". Note, the manual XGBoost modeling requires you to generate a manual tuning table and should probably be done only if you are quite familiar in how to do this for XGBoost.

In the next window, you will cast columns into selected roles, set up tuning table parameters, and DOE options. Enter in the values you see in the screen shot below and use 1234 for the random seed for fitting and for the DOE, this should generate the exact same fit table that I have for this post. Be sure to deselect "Informative Missing" and "Ordinal Restricts Order". Then click "OK".

Note, since we are using the Validation Column, the validation method in this window is automatically selected to "Validation Column". If you didn't want to use a validation column, but a holdback portion, leave the validation column selection empty when casting columns into selected roles and enter a number between 0 to 1 when you select the radio button for the validation method. The default is 0.3 (30% of the data) if you accidentally leave it blank.

There's also some notes on setting tuning parameters for each modeling method in their respective tune setup windows. These are just a guide and some "rule of thumb" usage and should not be considered as the only possible options.

Several windows will then popup and disappear -- this is the code generating the DOE tuning table used to pass to the fit platform, and then you should get a JMP window telling you the fit progress -- for the immediate fit. There will also be a small window that pops up giving you the overall fit progress.

Note that even though you put in 10 runs for the DOE, there are 11 total (N+1) because I programmed it to always run a fit using the platform's default settings within JMP. After the fit is done, you should get a window like the one below. You can then use the right/left arrows to cycle through the different runs and tuning parameters while looking at the R^2 fit statistics (and the difference between training R^2 and fit R^2).

Using the table to the right of the graph, select the run that you would like to re-do. If the the "Save results data table" is checked, then you will be prompted to give the table a name and location to save it. The program will re-run just that one fit and display the results. Here, we're re-running Tune 6. I'm also attaching the output data table from the tuning to this post so you can compare with your use.

You should now get a window like such:

Note that the results window from the original tune is still open, this is so you can re-run other tune runs on an individual basis and compare them if desired. You should get the same results as above if you used all the same numbers in the model tuning. After you re-run individual fits, you can of course use the red hot-button to save those scripts to the data table -- it will populate all the parameters for the fit and save the script to re-run the bootstrap forest fit platform to the data table.

Once you are done, you can click "Cancel" to exit the JSL code, or click "Relaunch" to get back to the first window of the JSL code where you can select other data tables and/or fitting platforms.

I'm also attaching the JSL code if you need it.

Hope this helps, and let me know if you have any questions.

Thanks!,

Re: Graphical User Interface to Automate the Process of Model Tuning

jpol — Wed, 25 May 2022 05:29:54 GMT

Thanks DS for the detailed explanation and great examples.

I ran the analysis (DOE) as recommended and got exactly the same results that you provided.

A special thanks for the notes on setting tuning parameters for each modelling method.

Looking forward to tuning on my own data sets.

Philip

Re: Graphical User Interface to Automate the Process of Model Tuning

jpol — Wed, 25 May 2022 06:21:46 GMT

I'd like to bring it to the attention of those who may wish to evaluate this model tuning procedure that it may require JMP Pro as not all modelling methods are available in standard JMP.

- Philip

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Wed, 25 May 2022 12:02:26 GMT

Hi @jpol ,

Yes, that is correct, this JSL code requires the user to run JMP Pro 15 and higher. Version 15 and up is needed as versions before 15 did not have the XGBoost platform as an add-in. Also note that there is a known bug in the JMP Pro 16.0.0 version that causes JMP to crash when trying to model categorical responses with the XGBoost platform. This has been corrected in the JMP Pro 16.1.0 release.

Thanks for bringing it to people's attention about Pro for this script. I thought I mentioned it in the Scripters Club meeting, but might have forgotten to do so.

Thanks!,

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Mon, 29 Aug 2022 02:18:00 GMT

Hello Diedrich, excellent idea... I don't know what I could be doing wrong... why GenAT_v5.0.jsl doesn't work.

I await your kind guidance,

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Mon, 29 Aug 2022 15:53:31 GMT

Hi @Marco1 ,

I'm not really sure why you're getting the error message, other than the script needs to have a data table with that column name in it, which it should make automatically. Are you trying to follow the example that I showed above? If you follow the above example, you should get the same result that both Phillip and I got.

It looks like you're trying to do the above example, but it appears as if you're missing columns in your output data table. The output data table should have 16 columns in it, but it looks like yours only has 7. Do you know what happened to the other columns? If you could share more screen shots at each step, kind of like what I did above, that might help me to figure out what's going wrong. Or, if I could see the embedded log when you try and run it and get the error, I could look in the code to see where the error is occurring. More information from you about how you're trying to use the code will help to figure out what the problem is.

Thanks!,

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Tue, 30 Aug 2022 17:00:52 GMT

Hello Diedrich,

I am pleased to greet you, below I detail the process:

Any additional information you need, let me know.

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

jthi — Tue, 30 Aug 2022 17:27:09 GMT

There is a possibility that because you are using non-english version of JMP that some columns will be generated with wrong names (I have discussed this with JMP Support as I develop scripts to English and Japanese users and it is a mess...).

One example is line 2241 (N Rows column might have different names on different JMP languages):

dt_resultsSummary << Delete Columns( :N Rows, :R² Diff );

Could you maybe try changing JMP's language to English and rerun the script to see if that could be the problem?

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Tue, 30 Aug 2022 21:23:06 GMT

Hi Jarmo,

Smart observation, where can you change the language from Spanish to English?

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Tue, 30 Aug 2022 21:38:13 GMT

Hi Jarmo,

Clever observation, change the language to English and the example PowderMetallurgy_mod.jmp works ... will it be possible to correct that observation about the language?

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Wed, 31 Aug 2022 01:12:29 GMT

Hello Diedrich,

When choosing the 1-layer neural networks along with the 2-layer neural networks to optimize, could the GenAT_v5.0 script list or include in the results the best results of the found 1-layer neural networks and 2-layer neural networks?

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

jthi — Wed, 31 Aug 2022 06:53:38 GMT

It is fairly annoying to take care of those while scripting but it is possible. It depends a lot on the script what must be done and it is a lot of work to manage all cases (for example with Summary() scripter can use indices of the columns instead of names). In the ticket (7613370838) R&D took it into the consideration that user could "force" column creation to always use English column names, but I have no idea if and when that will be implemented.

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Wed, 31 Aug 2022 16:14:42 GMT

Hello Diedrich,

Could the GenAT_v5.0 script optimize multiple outputs at the same time like Model Screening does?

Cheers,

Marco

GenAT_v5.0 script

Model Screening

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Thu, 01 Sep 2022 17:23:19 GMT

Hi @Marco1 ,

**EDIT**: I think I understand what you were asking before -- can the code be changed to allow for more than one input to be modeling at a time? I'm sure it can, but I have not written it that way. I thought about it, but I really wrote this for my organization's needs and we only model one output at a time. I'm sure it could be done, but it would take some re-vamping and re-design of the code to get that to work.

Unfortunately, the script I wrote does not have the ability to perform tuning across all the different platforms in one go, like the model screening does. Keep in mind that the model screening platform in JMP only fits the default values of the models to the data, it does not tune each model. It's a nice feature to get an idea of how some default models compare to others, but I have found that a well-tuned model can outperform the default model that the model screening platform decides is the best. I end up not using the screening platform all that much because of this.

The primary goal of writing this code was to automate and simplify the process of tuning the many different modeling platforms in JMP, especially for those that might not have as much experience in tuning -- hence the tips for tuning with each platform. I still go through each platform in the GenAT_5.0 code and tune each model, then compare them on a holdout set at the very end to see which ones work best for my data. If one wanted to do an automated tuning over all platforms, it probably would be best to write it in the source code of JMP, which is not something I can do.

I think that Jarmo got it right when he noticed the language. Sorry, I have an English operating system and have English set as the language for JMP. I could imagine that trying to write this code for multiple languages would be very challenging. Hopefully by changing the language setting in JMP you can getup and running.

Please reach out if you have more questions, and happy modeling!

Thanks!,

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Sun, 04 Sep 2022 13:52:05 GMT

Hello Diedrich,

I use the google translator, I'm not sure I understand... in any case I confirm my query:

- GenAT_5.0 could predict 2 or more outputs?
- When selecting NN of 1 and 2 layers....Could GenAT_5.0 include in the report the best models found of 1 and layers?

I have observed, that in most cases GenAT_5.0 when re-training the best result lowers its effectiveness... GenAT_5.0 could have an option to choose to re-train or not to re-train?

The models found by GenAT_5.0 of 1 output.... are in most cases....superior to model screening.

Excellent contribution Diedrich!!!

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Tue, 06 Sep 2022 17:00:25 GMT

Hi @Marco1 ,

It is correct that right now, GenAT_5.0 cannot predict 2 or more outputs. Depending on the platform, it can handle continuous, ordinal and nominal (2 or more levels, again depending on the modeling platform). It cannot fit more than one response at a time. To me, I am not so sure one saves all that much time by fitting more than one response at a time anyway, you still have to go back and evaluate each model for each response.

In principle, yes, the code could be rewritten to allow a user to select more than one model from the output table and re-run the selected models, however, you can still do that in the current version -- you just can't run them at the same time. What you do after you've run the tuning design, and you see the output window with the graph and table, you select one to rerun and then after it's performed that fit, you go back and select another model that you'd like to test. You can do this as many times as you want. You might not want to have the check box Save output data table because you don't need to save that each time you re-run a model.

As for the re-run model not having the same effectiveness (I'm guessing you mean like R^2 or RASE), that is likely due to the randomness of the starting point for each model fit. If you always want to have the same starting point even when re-running a model, then you need to enter a random seed value for the fit model. This then forces the fit to start from the same point each time rather than randomly moving around the model space. This will ensure that your model evaluation values like R^2 will be the same for when it trains vs when you re-run the model. That being said, there is a drawback to this: you lose the randomness that is often very important for a model to be able to manage. For the most part, your response has some kind of average value (depending on its inputs of course), but there is also some randomness to the output which you run the risk of not capturing by allowing the training to start from random points in the model space. Allowing your model to also have this randomness provides for a more realistic situation compared to enforcing the same starting point. The flip side to that is that you run the risk of just modeling noise and not your actual output. This is why it's so important to have a hold-out test data set that is not used in training or validating the model so you can compare different models on this test data set to find which one actually performs the best on "unused" data.

Thanks for your feedback and compliments. If there are any updates to this code, I'll be sure to post it here.

Best wishes,

Re: Graphical User Interface to Automate the Process of Model Tuning

Marco1 — Tue, 06 Sep 2022 18:29:00 GMT

Hello Diedrich,

Thanks for your answer!!!... JMP is a good program and GenAT_5.0 is an excellent script that partially covers (1 output) the gap that JMP has in model screening, in most cases GenAT_5.0 finds better models than model screening ... it would be great if GenAT_5.0 could predict 2 or more outputs ... for example to apply it to cases where there are sales (outputs) of multiple products and the best model found by GenAT_5.0 can detect a pattern or relationship in the multiple exits (sales).

Regarding the trained neural network models, will there be a script that exports the formula found by GenAT_5.0 to excel to make predictions? Or is there a script that converts or generates VBA Excel code of the best NN model found by GenAT_5.0 to make predictions?

Cheers,

Marco

Re: Graphical User Interface to Automate the Process of Model Tuning

SDF1 — Wed, 07 Sep 2022 13:03:19 GMT

Hi @Marco1 ,

Well, the whole purpose of writing the code was to automate the process of model tuning rather than necessarily comparing one model to another. As mentioned before, the model screening platform only fits the default values for a particular method. Often the default isn't as good as a well-tuned model, regardless of which method you use to model.

As for modeling multiple outputs (Ys), the code could be rewritten to do this, but one still needs to go back and evaluate each individual model for each Y because each Y will have it's own independent model generated for it. So, I'm not sure you still save time.

In the case that you describe, you might be better off using the Multiple Correspondence Analysis under Analyze > Multivariate Methods to detect patterns between outputs, Ys.

As for your other question about exporting the formula, there is already a way to do that in JMP. Once you've made a model, you can publish the model formula to the Formula Depot platform (see below). Within the Formula Depot Platform, you can use the red hot-button to export the code in different formats, like C, Python, Java, SQL, and so forth. This might get you where you are looking in order to run the formula in Excel. However, if your prediction formula is easy, you might be able to manually "transform" it from a JMP formula to an Excel formula. But, if it's complicated, then you might want to keep it as a JMP file as it might perform faster when evaluating large data tables.