My name is Scott Allen.
I'm a Systems Engineer at JMP.
I'm going to be presenting a neural network tuning add- in
that I've developed really to help develop neural network models
in an automated way to build many models,
find out which parameters are most effective in building the best model.
What I'd like to do is go through just a little bit of background
on motivation for building this add- in.
I'm going to do maybe a walkthrough, the description,
what's going on in the background when you run the add- in
and then show a use case for it. A sample workflow.
Then just really wrap up with some lessons learned.
This was my first big scripting project, so certainly learned a lot along the way.
I'd like to share that and then as well as what's next.
First, just I want to take 3, 4 minutes here
just on the background and motivation.
A lot of other machine learning platforms in JMP have some sort of
parameter tuning function,
whether it's a tuning table that you create outside the platform,
like in bootstrap forest or boosted trees, or an integrated tuning design
like in support vector machines and the new XGBoost platform.
But there's not one for neural networks.
That led to some customers that I work with asking about it.
"Is this something that's available?
Are there scripts out there that we can use?
Is there an add- in?"
There are a few things out there, but maybe not everything in one place.
Then on a personal level, I've never undertaken a large scripting project.
I've done some small JSL scripts to automate workflows,
or to clean up data, but I had never built an application.
This was, I really wanted a project to help me learn.
I used this as an excuse to learn JSL.
The neural network platform in JMP is really powerful,
and so I'll just launch it really quick.
I'm sure if you've seen it...
I will mention I am using JMP PRO version 17.1 .
That's where all the testing has happened for this add- in.
I have not gone backwards many versions.
I would suggest if you are going to use this
you need to have JMP PRO and at least version 17.1.
The JMP neural network platform is really powerful
and I just want to take a few minutes to go through what this looks like.
I think everybody has...
If you've used neural networks, you've gone through this workflow
and it's a really nice flexible platform for developing these models.
You can start off with a baseline model and maybe you're not satisfied
with the performance of your model in this case.
You can just go back to the model launch and you can adjust the parameters here.
You can go and generate another model.
It's easy to start changing these parameters
and finding models that might be the most appropriate for your system.
But there are a lot of options in here.
You've got different number of nodes, you've got one or two layers,
there's boosting options as well as a lot of different fitting options.
You don't always know...
It's not always clear which of these parameters you need to change
in order to get the best model for the data table that you're working in
and the predictors that you're using.
It's not always clear if adjusting a parameter
is going to lead to a better model.
You just adjust up a nd down, maybe take big steps
to figure out what's going on.
It does require a lot of manual adjustments by the user
in order to generate these models.
They are pretty fast, so it's not time- consuming to create these
but it d oes require a lot of clicks.
My goal here and some of my inspiration was really to provide a single platform
for tuning and evaluating those neural network models.
I was also inspired there were some tuning scripts out there.
Nick Shelton had a script that would help you build that table
and run it and evaluate those models.
Mark Bailey had developed an add- in for this as well,
and there were s ome community posts that were trying to f igure out how to do this.
Those were all some starting points for this add- in.
Then I really like the XGBoost add- in interface.
It's really clean, it's got some graphical outputs
and it's really easy to navigate and lets you auto tune those networks.
The add- in that I'm developing isn't auto tuning,
it's really more of a brute force method.
Just build lots of models and then have a graphical interface there
to help y ou find the best one for your system.
Let's go on and move on t o just what the add- in does and how to use it.
To launch the add- in, just like most add- ins,
you go to the add- in menu, and it's called Neural Network Tuning.
That takes you to the launch dialog.
This gives you a column role interface that's very similar
to many other platforms.
We can just select all o f our factors here and our response.
Currently, it allows a single Y response,
and it can be either continuous or categorical.
The add- in currently only has the two validation methods,
so you can specify a validation column with both validation and testing.
Settings if you have that, if you don't, it's just going to do a random holdback,
the default for t he neural network.
Then you can specify and toggle or toggle the informative missing as well.
But we'll just go with this for now.
We click OK, and that takes you to the tuning launch.
This tuning dialog, first, gives you a little bit of information
about what's in the data table and what you specified as column roles.
Right now we've got the data table here, the model validation method,
as well as whether or not we have informative missing on or off.
Then we've got the DOE option.
What this add- in will do, is look at all the parameter ranges that you've set
and create a fast, flexible, space- filling design.
Really the only design specification you need
is to set the number of trials in this design.
In this case we've got by default it's going to be set to 20,
and there are some rules of thumb.
Maybe you want to specify at least 10 treatments in this design
for every factor that you're looking at, but it's completely up to the user,
and so we'll come back and adjust that if we need to.
You can also replicate this design with a different random seed.
In some cases if you want to see how robust you might be
to certain parameters, you can run the same neural network model 20 times,
but all with different random seeds to see how robust the model is to that.
Then you can toggle whether or not you want to see the DOE dialog that shows up.
Once you've gone through the DOE options here,
you can come down to the neural network tuning.
I tried to stay pretty true to the original platform
in the neural network by using similar language and options here.
This is where you go, a nd instead of specifying a single value
for one of these parameters, we're going to specify a range.
Also, I'm limiting a little bit the available options
for the type of neural network that you're going to create.
In this case you can generate a single hidden layer,
you can generate two hidden layers, or one hidden layer with boosting.
Currently, the neural network platform doesn't allow
a second layer with boosting.
If you try to do that in the neural network platform,
it will give you a warning saying that it's going to ignore that second layer.
In this add- in, I just make it so that you zero out all those second layers
when you activate boosting.
In this case we're going to e xplore this data table a bit
and maybe we're going to set some wide range of tuning parameters.
I'm going to go from 0-9 on each of those activation function number of nodes.
Over on the fitting options, you don't have to necessarily
know what might be best for your system, you can just select them
to be part f the DOE, or in the case
of these transform covariates and robust fit options.
You can toggle them to always be off, always be on, or include in the DOE.
We'll include both of those in the DOE for this example.
Penalty methods.
You can test any or all the different methods.
If you don't specify any penalty method, it will run
the default penalty method squared.
But we're going to check all of those.
You can also specify the number of tours per model.
In retrospect, I probably didn't need to add this
but because you could do something similar by just replicating all of the designs
and seeing the results from each of those models.
But it's here in case you want to adjust the number of tours.
You can also specify a random seed.
In this case, I've got, 3,4,5,6. Six different factors in this design,
and so I'm just going to maybe turn t his up to 60.
I'm going to create 60 models.
Before I click Run, I'm g oing to talk a little bit about
what's going on when you click Run.
The first thing that happens when y ou click Run,
is it's going to create a factor table with only the selected parameters.
We select it.
In this case it'll create a factor table with single- layer nodes,
all of the different fitting options.
It's not going to put the second l ayer or any of the boosting parameters
into the factor table.
Then I'm going to create a response table where the training, validation,
and testing R- squared values are specified
a nd then it's going to combine those into a single table.
This is what the sample DOE table will look like.
It's just going to have the responses as the R- squared values
and then all the different parameters.
Then these just get passed into the neural network platforms row- by- row,
and the results show up in the data table.
Just like we're saying, each of those runs in the design is going to go into the
neural network model platform sequentially and then we have a little dialog box
that's going to indicate the overall process.
In the case of really long- running neural networks,
it's good to just see how quickly the whole design is going to go.
I would recommend before specifying a large number of trials in this design
that you run one or two ahead of time just to see
how long they're going to take.
If you have fairly small data set, then they go pretty quickly, but if you have
a complex data set, they can take sometimes quite a while to run.
In this case I'm just using a sample data set from the sample data set library.
This tablet production.
It's not the best data set to run for neural networks just because
there aren't a whole lot of observations, but I'm using it just b ecause
the neural networks run really quick.
In this case, if we're satisfied with all of the specifications
that we have in our model, we can just go ahead and click Run.
What this has done is it's created that design table
and now it's passing through each of those trials
into t he neural network platform and giving an output.
If we look at the home window, all this has done is it's taken
that tuning result table and I've hidden it
and I just bring i t out into the same window here.
Let's just go on to the next one.
We can work through that data table here, so it's interactive.
What you first see up here is graph builder report.
T his shows you all of your different parameters,
so the different activation functions as well as the number of nodes
is the scale, and how the validation R- squared changes
with respect to each of those individual parameters.
In the case of categorical parameters, you g et a box plot showing the difference
in the values this way.
We can see in this case maybe for robust fit,
it's got a wider range, but on average it's g oing to be better.
The median value is better.
Is higher R- squared.
We can also look down here at the data table,
it's going to sort the data table by the validation R- squared
and it just shades those a deeper purple depending on the value.
Let's see.
We can use...
One reason I just kept this data table as it is,
is we can explore it just like we explore any data table in JMP,
so you can look at column headers or you can launch graph builder
or do various analyses on this data table as it is right here.
There are a few additional data columns or columns that are added
to this data table.
One, it records the random seed.
If you want to reproduce this, you know what random seed was used
in generating the n eural network models, shows you the informative missing,
shows you the elapsed time for each of the models.
If you are building large models or you might want to minimize the r untime,
then you can get an estimate o f how long it might take for you to run
additional models.
nDOE just records what DOE was run.
This was the first, so it gets a 1.
As you'll see we can run a second and third, and you can then use those
as some graphing tools.
Also tells you the type of neural network that was run.
Whether it was single- layered, double- layer,
or a single- layer with boosting.
There's a couple of other
columns that are added.
This factor column just lets you keep track of the factors
that were used in the design in case you want to compare models
that have different factors that were used instead of just parameters,
and then also a row column.
The next steps so we can continue building out.
Maybe in this case we're exploring the parameters.
Now we've got a baseline of a single neural network
and maybe we want to see how this compares to a two- layer neural network.
We can go back to the tuning launch which was just minimized.
You can see that all of this report, the graph builder report
as well as the table are just a ppended to the tuning launch.
We can go back up to t his tuning launch and maybe I want to look at
two hidden layers.
We'll go to two, and now I'll put in a range of
those results or those parameters and keep everything else the same.
You can see it's going to preload the random seed that I generated
in the first DOE just to make sure everything's using the same random seed.
We'll keep it at 60, even though we might want to increase it
by a little bit since we're a dding some more factors,
but keep it simple, we'll just go with 60.
One other thing just to mention is, in the case it's probably pretty
low likelihood t hat all of these would be set to zero in a design,
but if they are it'll just go to the default with JMP,
which is a single 10H layer.
You don't have to worry about having a neural network with zero layers
or zero nodes.
If we're satisfied with this, we're just going to click Run.
While this is running, I'm just going to bring up the home window.
You can see what it's doing.
We've got our original tuning result table here,
and it's built a second t uning result table,
and when it's done, i t just appends that to our original table.
Now I can see in orange, is all of those second- layer models.
We can see in this case, maybe having two layers
was not very beneficial in this case.
Looks like in pretty much all cases, those neural networks are not as good.
Then maybe just to round this out, we're going to go
and buld some neural networks with boosting.
Boosting is a really nice way to increase the prediction power of our model.
We're going to specify some boosting levels from 1-10, as well as
the learning rate from 0.1 to 1.
Once again our random seed is the same, and these are going to take
a little bit longer, so I'm just going to decrease the number of runs
just in the interest of time.
We can click Run here, we get our dialog box that tells us
our overall progress, and we can see these in the cases
where we've probably got lots of boosting, they might be taking a little bit of time.
In this case boosting neural networks, or neural networks that had boosting
are now this purple, and we can see in pretty much all the cases
purple is higher, especially when it looks like
we can see by these box plots as well.
Then we can also see in t his case over here our boosting.
Maybe we haven't found the maximum number of boosts,
it looks like it continues going up,
so we might want to add some additional boosting layers or maybe
our learning rate is best at this middle to high value
instead of at the lowest learning rate values.
The next thing we might w ant to do once we have this initial view
of our system, is to run a few of these neural networks and compare them.
We can do that a few ways.
We can look in our table here a nd run or and select some neural networks
based on the graph, and you can see they're highlighting down below
in our data table as well.
Maybe what we want to do is run the top four or five.
I want to run the top five.
I don't have to go back into the neural network platform to do this,
I can just run it by clicking this button, run selected models.
Now it's going to run those back into the neural network platform
and give us a little output that shows us the row in our data table that was run.
This is just the standard neural network output now.
We pass through the same random seed, so if we take a look at R-squared values,
we're going to get the same neural network that was developed
because we're using that, passing through that same seed.
Then we can come through and we can take a look at more...
Either the more summary statistic values or the actual versus predicted,
or get the profilers.
Whatever we might want to do to compare these neural networks.
One other thing we might want to do at this stage
is create some additional visualizations.
We can go through and we can show this tuning table.
If we don't want to work on i t here, we can just unhide it,
and now we've got a data table that we can use,
and we can go into graph builder and maybe we want to
take a look at validation versus training R- squared
just to see how that's looking, or we can go through.
Because we've got the number of DOE, we can track how our neural networks
have improved with our tuning.
I can go into graph builder and I can use that nDOE,
and maybe we want to look at training and validation R-squared,
we can turn on box plots and we can see how our tuning has progressed over time
with each subsequent design that we ran.
Other option is if you want to do some more analytical type analysis here,
we can go into screening and we can run predictor screening.
We can take all of those parameters, load them in as predictors,
and look at our training and validation R- squared.
Click OK, and now we can see which factors might be most influential
in the training or validation.
Not surprisingly, those learning rates and boosting a re good at this validation.
There's no end to what analysis you might want to do,
on here it's just what your goal of the neural network was
and where you want to go with it.
But you can do that all in t his data table which now has 160 neural network models.
Maybe these neural network models take a little bit of time,
so another thing you can do is we can just save this.
I want to go file, and I can save this.
I'm just g oing to save it to my desktop as T uning Results.
I can come back to this later.
Now that I've got it saved, I'm going to close this.
One thing that you'll get a warning, when you close it.
You should just hide this data table.
If you close it, it's going to take it out of the neural network platform
and you won't be able to continue your work,
but you can hide it.
When you get back to the add- in, when you close this window,
it's just going to remind you .
You can close it without saving or you can cancel to go back and save it.
We're just going to click OK.
When I save this tuning table for later, so what I can do is,
is now I can relaunch that add- in and I can go into the tuning add- in,
click OK.
But instead of resetting any parameters and clicking Run, I can go to Load.
What I can do is I can load a previously saved tuning table,
and what this does is let me in the case of neural networks
that take a long time, you don't have to start over from scratch.
You can just keep building on the table that you wanted, or maybe you do
some additional analyses offline and want to come back in
and build some more models, so you can go through and when you specify
all the different parameters and number of tuning runs you want to have,
it's just going to continue appending it to this table.
With that let's minimize this.
That's quick run through of a ll the different options
in this tuning table.
It's really a brute force way.
You can see there's no auto tune. Right now it's just running
all those different parameter combinations giving you an idea
of where your optimal model might be .
How you might work this into your workflow so you can explore those parameters,
this is essentially what we did today.
We looked at single- layer models, we looked at models with two layers.
Single- layer with boosting.
Another thing you might do is go back and say,
"W ell, where are my optimal models?"
They're here.
I can go back to the tuning launch and maybe I want to recreate those models
and go more towards this range.
It looks like I'm in the middle range here, is optimal,
so instead of going from 0-9, maybe I go from 5-7 .
Linear models also might be best in the middle,
and maybe the gaussian, there are some good ones on the left,
so we'll go from 1-8 or something like that, seven or eight.
Then we can look at which... In this case, there wasn't an advantage
to having it on or off, so we could just have those off,
or off or on.
But it looks like maybe in this case the weight decay was the best method,
so we could just include that one and then go through
and build some more models with a narrow range of those parameters.
We can also check parameter sensitivity to a random seed.
In this case we can pass through either constant values
o r a narrow range.
You can pass through a constant set of parameters
just by setting each parameter equal to each other,
and it'll pass that through the neural network as five every time,
or you can just specify the min value, and that will pass it through
as a constant value.
If we wanted to we could look at
maybe this was a neural network that was one that we identified as optimal,
but I want to see how robust it is to the random seed.
What we can do is we can just say just run it once,
but let's replicate it with 20 random seeds.
That might actually take a little bit of time,
so let's go down to 10.
Then we can...
Actually, I don't want to do that on this one,
because I've got all those others, but we'll just run this.
You might want to do this on a clean data table
where you're not adding it to the design, but we can use b ecause we have
the number of DOEs we could find where that random seed was.
We can see here, all the different random seeds
and I can go into graph builder and I could just put in
the training and validation R- squareds, and then I can use local data filter now
on my random seed to show everything except the one where we had all those.
Now I can see how robust my model is to that random seed.
That's the same design now run 10 times,
and so on average we can see what the median value is
or the range of expected results.
Then finally, w e can load those saved data tables.
Those are really beneficial for neural networks that might have
really long computing times, so you can save it off
and come back to it later.
Those are some workflows that might b e helpful to you as you build
your neural network models.
I just want to wrap up with a few lessons learned.
This was the first scripting project that I went through
that developed an application,
and so I think this is pretty obvious to anybody that's developed any JSL
or built any applications, the JMP community is really
an invaluable resource.
I would say 95% of all my questions, I would just go to the community
and someone else had a similar issue.
Really recommend if you are interested in building add- ins,
building applications, the community is really the first place to go.
Really, your imagination and your targets are the limiting factor
in developing these applications.
A ligned with that is because there are so many possibilities,
there was a need to keep this focused.
In this case, I just have... I think in one iteration
I had many different outputs, different graphs,
but they weren't really the heart of the application.
Because there's so many possibilities, I really tried to keep this focused
on just giving some graphical output to navigate the different models
and run them and identify which might be most appropriate.
What's next?
I really want feedback.
It's selfish, but it's what I really want.
I would love to hear how people can use this, how it's benefiting them.
You can find this in the community, in the add- in section of the community.
Please feel free to make a post here, tell me how you're using it.
I do have a few known issues, and as I gather more,
I'm sure it's not 100% bug- proof, but this will be the place to go
to get the current version of the add-in as well as understand any known issues.
But I'd love to hear how you're using it,
and the community is a great place to do that.
I would also like to hear ideas on additional reports.
I kept this pretty simple, but if you find that there's a report
that you're generating every time you run this,
I'd like to know what that is a nd maybe we can build that into the add- in.
I'm also going to continue debugging.
There are a few known issues that I'm working through right now,
and then maybe a larger goal is to try to do that auto tune.
Instead of doing this brute force method, maybe start out with
a small number o f parameters, figure out which ones
are a ffecting the response the most, and then having a directed tuning.
With that, I'd like to thank you for listening to this presentation.
This is a really fun project and I hope you find it useful,
and would love to hear about it.
Thanks.
