Model Screening in JMP Pro 16: Streamlining the Predictive Modeling Workflow (2021-US-45MP-846)
Mia Stephens, JMP Principal Product Manager, SAS
Predictive modeling is all about finding the model, or combination of models, that most accurately predicts the outcome of interest. But, not all problems (or data) are created equal. For any given scenario, there several possible predictive models you can fit, and no one type of model works best for all problems. In some cases, a regression model might be the top performer, in others it might be a tree-based model or a neural network.
In the search for the best performing model, you might fit all of the available models, one at a time, using cross-validation. Then, you might save the individual models to the data table, or to the Formula Depot, and then use Model Comparison to compare the performance of the models on the validation set to select the best one. Now, with the new Model Screening platform in JMP Pro 16, this workflow has been streamlined. In this talk, you learn how to use Model Screening to simultaneously fit, validate, compare, explore, select and then deploy the best performing predictive model.
Auto-generated transcript...
Speaker | Transcript |
| Mia Stephens | it's probably won't take 45 minutes. |
| All right, we're good. | |
| damn it, why did that text just come on. | |
| Jason Wiggins, JMP | Did you see, I was just gonna ask that I had started recording but that somebody's going to get in and and clip the beginnings in the ends of the so. |
| Mia Stephens | I would say you're recording. |
| Jason Wiggins, JMP | yeah. |
| I can stop if you want. | |
| um. | |
| Jason Wiggins, JMP | Alright recording started. |
| Mia Stephens | hi my name is MIA Stevens, and I am a JMP product manager. |
| And today we're going to talk about a new feature that first appeared in JMPed 16 or JMP pro 16 model screen screening. | |
| Actually, let me start that over. | |
| Jason Wiggins, JMP | Recording started again. |
| Mia Stephens | Hi, I'm Mia Stephens. |
| And I am a JMP product manager. And today we're going to talk about model screening. Model screening is a new platform in JMP Pro 16, and it allows you to fit several models at the same time, which enables you to streamline your predictive modeling workflow. | |
| So we're going to start by talking a little bit about what we mean by predictive modeling, and this is in contrast to more classical modeling or explanatory modeling. | |
| We'll see the types of models that you can build in JMP Pro, and we'll talk a little bit about the predictive modeling workflow. And then we'll see how to use model screening to streamline this workflow. | |
| We'll see several different metrics for comparing competing models and we'll use two different examples in JMP Pro, and I'll be using JMP Pro 16.1. | |
| So first, what do we mean by predictive modeling? | |
| Most of us have had statistics courses at the university or college or through your company, and typically, when we...when we learn statistical modeling it's explanatory modeling. | |
| And what we mean by this is that you're interested in studying some Y variable, some response of interest, as a function of several Xs or input variables. | |
| So we might be interested in identifying important variables, maybe we're in a problem-solving sort of situation where we want to try to identify the most important Xs, the most important potential causes. | |
| So, as a result of explanatory modeling, we might be able to identify, say, that X1, X3 and X6 are potential causes of variation in Y, and this might lead you to study these variables further using a designed experiment. | |
| We might also be interested in quantifying the nature of the relationship so, for example, we might be interested in understanding how Y, our response, changes on average, as a function of the Xs. | |
| a one unit change in X is associated with a five unit change in Y. | |
| So this is what we typically think about when we talk about modeling or statistical modeling or regression analysis. | |
| Now, in contrast, predictive modeling is about accurately predicting or classifying future outcomes. | |
| So in classical modeling, we're typically making statements about the mean, the overall mean. In predictive modeling, we want to know what's going to happen next. So, for example, will this customer churn? Will this customer | |
| take the credit card offer? Will the machine break down? So we're interested in predicting what's going to happen next. So we're interested in accuracy, accuracy of our predictions. | |
| In predictive modeling, we tend to fit and compare many different possible models and this involves or can involve some very advanced techniques. | |
| And some of these techniques may not be very easy to interpret. So if you think of neural networks, neural networks are typically considered to be somewhat of a black box. | |
| And with these more complex models, overfitting can be a problem, and what I mean by overfitting, is we tend...we can fit a model that is far more complex than we need. | |
| So in predictive modeling, we tend to use model validation. We use model validation to protect against overfitting and also against underfitting. And what model validation involves is splitting our data, and a typical way of splitting our data is to have some of our data held out... | |
| well, some of our data is used to train our model, some of our data held out, and we might have a third set that's also held out. And the way we use these data is that we fit our models using the training data | |
| and then we see how well the model performed on the validation data. And the third step might be used, and this is often called the test set, | |
| because validation...the validation data in JMP is often used to control how big the model gets. So we'll see this in action in a few minutes. | |
| I won't go into very many details around validation but if this is a topic that's relatively new to you, there are some slides at the end of this deck that I borrowed from our free online course, STIPS, and it illustrates why validation is important. | |
| In JMP Pro there are lots of different types of predictive models that you can fit. | |
| Some of these are available from fit model, so if you have a continuous response, you can fit a linear regression model. If you have a categorical response, you can fit | |
| a logistic regression model. This can be an ordinal or multinomial or binomial regression model. You can fit generalized linear models or penalized regression models like elastic net and Lasso and ridge regression. | |
| There's a predictive modeling menu with many items, so for example, neural networks. And you can fit neural networks in JMP Pro with | |
| two layers, with multiple nodes, multiple activation functions, with with boosting. And generally in JMP, | |
| you can fit models for categorical responses or continuous responses using the same platform. You can fit a variety of different types of trees, classification trees if you have a | |
| categorical response, regression trees if you have a continuous response. You can also fit some more complicated types of trees, like bootstrap forest and boosted trees. And you can see that there are other methods available as well. | |
| And there are a couple of predictive modeling platforms under multivariate methods, so discriminate analysis and partial least squares. So this is by no means an exhaustive list, and I will point out that partial least squares is also available from fit model. | |
| The question is, why do we need so many models? In classical explanatory modeling, we're typically learning about regression analysis, but there are clearly many, many more types of models that we can fit. | |
| Well, the reason is that no single type of model is always the best. The best model often depends on the structure of your data. | |
| How many variables you have, whether you've got purely continuous Xs or if you've got a mixture of continuous and categorical variables, if you've got non linear behavior. | |
| So, so no single type of model is always the best model at predicting accurately. So we generally try several different models and we pick the model or the combination models that does the best job. | |
| And we think about predictive modeling in the context of the broader analytic workflow. | |
| Of course there's some type of problem that we're trying to solve, a business problem or an engineering problem. | |
| But we compile data, and this could be data from variety different sources, and we need to pair these data, we need to curate the data. | |
| We explore and we visualize the data to learn more about what we have available to us and additional data needs. And the predictive modeling aspect of this is under analyze data and build models and, in my mind, this is this is where the fun really begins. | |
| So the classical workflow within JMP is that you fit a model with validation. | |
| And then you save the prediction formula to the data table, or you publish it to the formula depot and we'll see this in a few moments. | |
| And then you fit another model. So you might start with a regression model and then fit a tree-based model. And then you repeat this and you fit several different types of models. | |
| And then use model comparison to evaluate the performance of each model on the validation data. So remember we use the training data to fit the model, but to compare models to see which one prediction most accurately, we use the validation or the test data. | |
| And we choose the best model or the best combination of models. | |
| And then we deploy model, so this is kind of a typical workflow. | |
| Now, with model screening, what model screening does is it streamlines this. So you can fit all of the desired models within the model screening platform, | |
| compare those models using validation data, select the best model. You can explore the models, you can even fit new models from model screening, and then you can easily deploy the best model. | |
| So, to take a closer look at this, we're going to use a couple of examples, and the first example is one that you might be familiar with, | |
| the diabetes data, and these are in the sample data directory. So the scenario is that researchers want to be able to predict the rate of disease progression | |
| one year after baseline measurements were taken. So there are 10 baseline variables and 442 patients, and there are two types of responses that we're going to look at here. The response Y is a quantitative measure that measures how much the disease has progressed in one year. | |
| And there's also a binary response, high/low, so high means there is a high...high rate of disease progression and low means that there was a low rate of progression. | |
| And the modeling goal here is to predict patients most likely to have a high rate of disease progression so that corrective actions can be taken. | |
| So, ultimately, what we'd like to be able to do is build a model that allows us to classify future patients, based on their demographic variables and the baseline information that we have here. So we want to be able to | |
| accurately predict whether they're going to have a high rate of disease progression or not so that we can then take action. Okay, so that's that's the scenario. | |
| And before we get into JMP, I want to talk a little bit about how we decide which model predicts the best. | |
| And I mentioned that we use the validation or test set for this, and there's several different types of measures that we can use. | |
| For continuous responses, we use RMSE, which is the root main square error, or RASE, which is a related measure, the root average squared error, and this is the prediction error, and here the lower is better. | |
| We can also use AAE (average absolute error), MAD (mean absolute deviation) or MAE and for these also, lower is better. So these are...these are...these are kind of average error measures. | |
| And then, R square, and for R square, which is really measure of goodness of fit, higher is better. | |
| So we tend to use these for continuous responses; there are many other types of measures available. | |
| For categorical responses were often interested in the misclassification or error rate or accuracy rates, we might be interested in precision. | |
| There's other measures, area under the curve, which we see in an ROC curve, sensitivity and specificity, and these go by several different types of names. | |
| And there are a couple of other measures that we've added in model screening that are good when we have unbalanced data. There's an F1 score and MCC, which is Matthews correlation coefficient, and this is a value that takes on... | |
| or a metric that takes some values between minus one and plus one, and it's good at measuring when you've got categorical data, how well the data are classified in all four of the possible categories, the true positive, true negative, false positives and false negatives. | |
| So let's...let's go on into to JMP. | |
| And I'll just open up the data. | |
| So these are...and I'll make a little bit bigger, these are again the diabetes data. | |
| And if you want to play around with these data, there's a blog that uses these data, and you can also find them under help and then sample data. | |
| And again, we have a response Y, let me make it a little bit bigger, which is our continuous measure of disease progression. And then we have a binary Y with outcomes of low or high, and we'll take it...we'll take a look at each of these. | |
| So to start, let's take a look at if you...if you had JMP 15 or if you're not using model screening, how would you fit predictive models to predict Y, the continuous response, if you have JMP Pro? | |
| Notice that we we have several demographic variables. | |
| So there are 10 of these, and there's a validation column, and the validation column partitions our data into training and validation data. | |
| So we're going to fit our model using the training data, and then we're going to see how well the model performs when we apply it to the validation data. | |
| To create this column, there's utility under analyze, predictive modeling, make validation column, and this allows us to create a validation column. | |
| You might have variables that you want to use as stratification columns so, for example, you want to make sure that the data is balanced across the | |
| validation sets. I'm just going to click OK here. | |
| And this allows us to assign a certain percentage of our data into the different sets. | |
| And there's some additional options down here that are useful, so random seed is really nice if you're teaching and you want all of your students to have the same partitioning. | |
| So this was used prior to | |
| analysis, so let's start by just taking a look at this column Y, the continuous response. So if I were fitting predictive models, a natural starting point is to fit a regression model, and regression in JMP is under fit model. | |
| My response is Y, and I'll just fit...start by fitting a main effects only model. | |
| I'll input validation here in this validation field and, by default, the personality is standard least squares. But if you're familiar with JMP and JMP Pro, you know that there are a lot of different options available under personality. | |
| When I click run, | |
| one change in JMP 16 is that if we look at the actual by predicted plot or any of the other plots, notice that there are a points...there are some points here with a V, instead of a dot. | |
| This is to indicate that these points were actually assigned to the validation set, so we're basically looking at, | |
| you know, how well the model predicts in the random noise around that, and you can see that there's some some degree of scatter. And | |
| remember, we're fitting our model to the training data and we're seeing how well it performs on the validation data. So there's some statistics down below. So anytime we've got validation, you'll see some extra statistics for the training and validation sets. | |
| And we can see that there are a couple of measures here. There's R Square and there's also RASE. | |
| And generally when you fit the model to one set of data and then you you you apply that model to new data, it's not going to perform quite as well on the new data. So we can see that R square is lower for the validation set and the RASE, the error, is a little bit higher. | |
| Once...the one thing we see down here under the prediction profiler | |
| is that notice...notice that total cholesterol and LDL really kind of fan out at the ends. | |
| And as I drag | |
| the value of cholesterol, notice how wide this this confidence interval for the mean gets. So as I drag it to the low end, | |
| it gets super wide and the same thing as I drag it to the high end. In fact, all these intervals for each of the predictors gets super wide. | |
| And this is kind of beyond the scope of this talk, but this is because we don't actually have data or a lot of data out in this region. If I took...take a look and I'm just going to | |
| grab these two variables, total cholesterol and LDL as is an example...for the cholesterol and LDL. | |
| Notice that there's a pretty strong correlation between these. I don't have any data in this region and I don't have any data in this region. | |
| So if you ever fit a predictive model or any type of model where you see these bands really fanning out, | |
| it's an indication that you your predictions aren't going to be very precise out in that region. And there's a new feature in JMP Pro, | |
| called extrapolation control, that can warn you if this is a problem. So if I drag this out, | |
| notice that it's telling me hey, there is possible extrapolation; your predictions might be problematic here. And if I turn extrapolation control on, | |
| notice that it truncates these bands, and it's basically saying, you can't make predictions out beyond those bounds. It's just not valid. | |
| So this is a little bit beyond the scope of this, but if you ever see bands in your prediction profiler that are fanning out, | |
| it's often because you're predictors are correlated to one another, and it's...it could be that you just don't have a lot of data to make predictions out in those regions. So it's good to be aware of that. | |
| Alright, so getting back to the task at hand, so let's see, I fit this model and reduced the model and comfortable that this is, this is my final model. | |
| I'm going to save this model out to my data table by saving the prediction formula. And you'll see that it adds a new column to the data table. And I just built a linear model, so it saved that formula to the data table. | |
| And an alternative is to publish this, and I'm going to publish it to the formula depot. | |
| So, ultimately, what I'm going to do is I'm going to fit several different types of models (and I'll go ahead and close this) and I want to compare those models so I can use that... | |
| I can do that either by saving the formula and going to analyze, predictive modeling, model comparison, and I'll do this in a moment, or by saving it out to the formula depot. | |
| And the advantage of saving it to the formula depot is that now I've got scoring code. So, for example, if this were my final model that I wanted to apply to new data, | |
| I could copy this script and I can apply it to score new data | |
| (and I won't do that here), or I can create code in a variety of different languages, if I want to be able to use this model in a production environment. | |
| So that was a simple linear model. What if I want to fit a different type of model? | |
| I could fit a more complex model here in fit model. | |
| So I can fit a generalized regression model and maybe I want to fit a Lasso model, so I'll click go here. | |
| And there's some model comparison right here within generalized regression, but let's say that I want to save this model out, right. | |
| And I'm not I'm not paying too much attention to the details here, but I'm going to save this one. I'm going to save the prediction formula out. Maybe I'll go ahead and publish that one as well. | |
| Go here, publish this one. | |
| And I might repeat this, I might fit a model that's got interactions. I might then proceed to fit a neural network model or partition model, a tree-based model and, if I have a categorical response, this will fit | |
| a classification tree. If I have a continuous response, that will fit a regression tree. I might fit a bootstrap forest, which is a collection of smaller trees or boosted tree, so there's several different types of models I can fit. | |
| And once I get my models and save the formulas out or published to the formula depot, I want to find which model predicts most accurately. | |
| So I'm going to use model comparison. | |
| And I'm simply going to add validation as my by variable, | |
| because I want to be able to see how well the model performed on the validation data. And just using these two models and looking at the validation data (I'll tuck away the training data), | |
| I can see that I have lower average error for the least squares model and my RASE is a little bit higher for least squares model. | |
| And my R square is a little bit better for the generalized Lasso. | |
| So this is the typical way we might do predictive modeling if we have JMP 15 Pro or JMP Pro 15, or if we're not using model screening. And what model screening does is allows us to do all of that within one step. So I'm going to go into predictive modeling and launch model screening. | |
| And model screening, if you look down in the corner here, allows us to basically launch each of the platforms that makes sense for the type of data we have | |
| and automatically run those models. So we don't have to go in and select the model from the menu, we can run it automatically from here. So I'm going to fit the same type of model, so it's a main effects only model here. | |
| These are my Xs. I'm going to use the same validation. | |
| And there are a bunch of options here that we'll talk about in a moment, and I'm going to click OK. And notice I've got a decision tree, bootstrap forest, boosted tree, K nearest neighbors. | |
| By default Naive Bayes is off, I can fit a neural model. | |
| There's others at the bottom here. Notice that got XGBoost, so XGBoost is available because I have installed an add-in, the XGBoost add-in. | |
| So this is a free add-in that accesses open source libraries, and if you've got it installed in JMP Pro, then you can launch it from the model screening platform. So I'm going to leave that off and just click OK. | |
| So it's running the models and there it is. So it's gone out and it's actually skipped a step, right. It's run each of these models for me automatically. | |
| And, and there are some simulations you use to determine the best starting values for the tuning parameters, so we fit all of these models. | |
| And it's reporting out validation training and validation statistics for each of the models. And now to figure out which of these models is the best, | |
| Select dominant. And what select dominant basically does is a create afrontier across the available metrics, so I hit select dominant here. | |
| And finds that for these data, the best model is the neural boosted model, and then the details for that model are up here under neural. | |
| So all of the details it's really just launching the platform and capturing all the information, right here within this this platform. | |
| So all the details are here for neural, so it's basically run a TanH, with three nodes that is boosted. And I can see the training statistics and the validation statistics and if I want to be able to see the model that was actually fit, | |
| here's the underlying model. | |
| Right, so this is...so this is my best performing model. If I want to be able to take a closer look at these models, I can explore them from here and there are some options here, I can look at the actual by predicted plot. | |
| And I can compare these models, so as expected, there's a little less variation in our...in our training data than there is in the validation data and the actual by predicted plot. I can turn off certain models if I want to be able to explore this. | |
| And I can also launch the profiler and if you've got 16.1, the profiler is an option. It's actually a hidden option if you have 16.0, and again I'm in Pro for this. The profiler is nice because it allows us to explore the models that we fit, so I've selected three models. | |
| And let me make this a little bit bigger. | |
| Maybe not that big. | |
| So I've got a bootstrap forest, which is a tree-based model, where it where it basically adding the results of a bunch of different trees together. | |
| I've got a boosted model, and neural models are really nicely picking up nonlinear behaviors. And I've got the Lasso model, which is essentially just a linear model. | |
| And you can see these models pretty clearly. Notice with generalized Lasso, right, I've got this linear relationship between BMI | |
| and the response. With neuro boosted, I can see that there's some curvature. I can see it out here with with LTG, and with bootstrap forest, I can see that I've got this little step function, right, so it's actually approximating this this non linear relationship that we have here. | |
| So if we decide that the neural boosted model is the best, | |
| then we can basically launch the model, launch it from here, and we can either save the formulas out to the data table, or we can publish them to the formula depot for deployment. | |
| So it saves a lot of time and still gives us the ability to explore our models. | |
| And what if we have a categorical response? So remember, we have Y binary. We can use Y binary in the same way, so I'm going to go to analyze, and again call model screening, and I'll just hit recall and instead of Y I'm going to use Y binary. | |
| We can run the same types of models in it. If a certain type of model doesn't make sense, it won't run it. So, for example, least squares regression doesn't make sense here, so it won't actually run it. | |
| There are a few options here. I if I select set random seed, like 1, 2, 3, 4, 5, this will give me repeatability and if I launch a model from the details, | |
| it'll...it'll have the same results. I've got some additional options here for validation. | |
| So it turns out that in many cases, using the classical partitioning of training, validation, and test | |
| isn't actually the best, particularly in small data sets. So I might want to use K fold cross validation instead and we've also got nested cross validation. | |
| And you can do repeated K fold validation. So this is a nice option, particularly if you have a small data set. | |
| And there's some additional options at the bottom. So you might want to add two-way interactions or quadratics. | |
| If you know you're missing values, then you might want to use informative missing. And additional methods applies to additional generalized regression methods. And if you're in the JMP 17 early adopter program, you'll notice that this part of the menu dialogue | |
| or the the model launch dialogue has been reorganized a little bit to make it flow a little bit better. Right, so so I'm going to just use the default settings here with a random seed and click OK. | |
| So it's running all these models. Again, it's very, very quick. Well and now, because I have a binary response, I've got some additional metrics. I've got two | |
| R square measures, I've got the misclassification rate, that area under the curve, RASE. Remember this is related to root mean square error. Again, I'll click select dominant and again it found neural boosted, and that's not always the case. | |
| If I select what I think are the top models, the the options under the red triangle, you can look at the ROC curves for these and, again, there are check marks that I can turn on and off models. Let me | |
| select this. And one of my favorite new features, which is also available from the model comparison platform and in JMP 17 is being added to any any modeling platform that does classification modeling, is this thing called a decision threshold. | |
| I think this is one of the...one of...one of the most powerful things that we added in JMP 16. | |
| So with these three models selected, it's given me a summary of the classifications for those three models. | |
| And I'm going to make this a little bit bigger, so we can see a little bit better. | |
| So the way we read this is that, if we just take a look at neural boosted, it's showing us that two categories for neural...neural boosted and we're predicting the probability of high. So these are fitted probabilities for high. | |
| And the cutoff is .5, so anything below .5 is classified as low; anything above .5 is classified as high. | |
| So, if we look at this red block right here, these are all the points that were classified as low, but were actually high, and we can see that number here. There were 29 of those, so these were the false negatives. | |
| The lower red box, these were classified as high, | |
| but they were actually lows. So these are the false positives, these guys here. | |
| And across the bottom are some metrics, some graphs to help us make sense of this, and makes...and helps us to make some decisions. And there's also some metrics, so we can see those same values here, the 29 false negatives, 17 false positives. | |
| And we can also see the true positives and the negatives. And what I really like about this is that it allows me to explore different cut offs. | |
| So, depending on your modeling goals, for example, we might be interested in having a high sensitivity rate, but we really don't want to have a low specificity rate. | |
| Sensitivity rate, and this key over here is really helpful, sensitivity is the true positive rate; specificity is the true negative rate. | |
| You know, can we find a cut off that actually does a little bit better at both of those? And by clicking and dragging, you can change the cutoff for classification | |
| and try to find a cut off that does a better job. So for example at .26, and I'll just put in a nice rounded number here, at a cut off of .25, the sensitivity is higher, while still maintaining a high specificity. | |
| The graphs that I tucked away are really useful in in interpreting these numbers as well, so they give us an overall picture of false classification | |
| by threshold, by portion, and then true classification. So if I'm trying to find a point that balances specificity and sensitivity, I can look at this graph. | |
| So these top lines are the sensitivity, the dashed lines are the specificity and is there a point where I'm still above a certain value for sensitivity where... | |
| sensitivity, where my specificity stays high? And we can...we can balance those out. So if I look right here, I can see this is the point at which both of those lines cross. | |
| And this becomes really important if we have a case where we have unbalanced data. So let's take a look at another example. So I'm going to close this out and | |
| not save this. And let me close the formula depot, and by the way, formula depot is standalone, so basically it's not tied to the data set that we were using. So it makes it really easy to take a model and then apply it to new data. I'm going to just go ahead and close this. | |
| So the scenario here for this next data set, the data called credit card marketing, and the scenario is that | |
| that we're doing some market research on the acceptance of credit card offers. And the response is offer accepted. And a caveat is that only 5.5% of the offers, | |
| using historical data, have been accepted. This is actually a designed experiment where we're looking at different rewards, different mailer types, | |
| different financial information, but the designed experiment is is in reward and mailer type. We're going to focus on this particular modeling goal. We want to identify customers that are most likely to accept the credit card offer. | |
| And by doing this we're going to try to | |
| apply the model to new customers and, based on the classification for these new customers, we're going to make a decision whether to send them the offer or not. | |
| So if customers are not very likely to accept the offer, we're not going to send it to them. If they're highly likely, then we are going to send it to them. | |
| So that's the scenario, and returning to JMP, we've got offer accepted. And I'm just going to go ahead and run a saved script here while I'm talking about the data. | |
| We've got 10,000 observations, so this is going to take a little bit longer, and right now it's running those models, and I'm just using the default settings here. So I've got offer accepted, | |
| the reward type is air miles, cashback, or points. The mailer type is letter or postcard. And then the rest of these are banking...historical banking information and some demographic information. | |
| You see that it's running support vector machines. Support vector machines can be a little slow, particularly in big data sets, and in fact, it won't run, by default, if we have over 10,000 observations. | |
| We can see we've got average balance. And the data are petitioned into training and validation, and JMP will automatically recognize the zeros as training and the ones as validation. Right, so it's run all of the models. | |
| And if I select dominant, we can see that it selected fit stepwise and decision tree. | |
| But I want to point out this misclassification rate. | |
| So this is...this is really something we see very often when we're trying to look for the needle in the haystack, right. If I take a look at these data, and I'm just going to run a distribution of these data... | |
| Let me make this a little bit bigger, so we can see it a little bit better, right. So I'm just looking at the validation data, notice that | |
| 6.25, 6.2, 6.3% | |
| of the values were yes, they responded to the offer, and that's exactly what we're getting here for the misclassification rate. | |
| And to better understand this, I'm going to turn on the decision threshold. | |
| And make this a little bit bigger. | |
| Notice that the predicted yes is all zero, and this is because the predicted probabilities were all relatively low. | |
| Remember that that in a classical confusion matrix, the cutoff is .5, and if you have unbalanced data, it's very unlikely that you're going to classify very many of the observations as yes, in this case. | |
| As I click and drag, | |
| and I get to this first set of points, then I start to see some predicted yeses. In fact, I'll turn the metrics on. | |
| When you've got highly unbalanced data, the predicted probabilities might be very low. | |
| So, having this slider allows you to explore cut offs | |
| that can impact both your sensitivity and specificity. So in a case like this, we might say something like, well, if we were to send... | |
| I'll just put in .05 here...if we were to send a credit card offer to the predicted probabilities, the top 5% of the predicted probabilities of yes, right, our sensitivity | |
| between the two models is about .72, .75 and our specificity is still pretty high. So having this...this cut off, this slider, allows you to get more useful information out of your model. | |
| So, so if...at .5, I can see I'm getting some true positives. I'm also getting a lot of false positives, right, so there's a trade off here. | |
| Now, in a case like this, you may also have some financial impacts. right. So it makes sense to think about this... | |
| profit matrix. And there's a button. | |
| I click on this. | |
| The profit matrix is a column property for a particular column, and we can set | |
| that value, that cut off from the probability threshold, as a column property, or we can also plug in values. You know, in classical scenario is maybe it costs $5 | |
| if we choose to send somebody an offer. | |
| And if they respond to the offer, | |
| maybe it's worth $100. So if we choose to send somebody an offer, and they don't respond, we lose $5. | |
| If we choose to send somebody an offer, and they do respond, maybe we make on average $100. And maybe there's some lost opportunity, maybe we say that we lose $100 | |
| if they don't respond. And oftentimes the cost of this is, this is just left blank, we just we just plug in values for the yeses. If I apply this and click OK, | |
| then some new values are added, right. So, so now we have a profit matrix and if I look at our metrics, then we can see the predicted profits. So | |
| on average, right, for each customer in this data set, it's saying that will make $1.50 if we apply the decision tree model and a little bit less if we apply the stepwise model. | |
| So, in a situation like this with unbalanced data when there are some underlying costs, it makes sense to use the profit matrix. If we don't have underlying costs, | |
| then this slider helps us to pick a cut off that will allow us to meet our modeling goal, okay. It could be to maximize sensitivity, maximize specificity, or something else. | |
| So that's that's essentially model screening. | |
| Let me close this and just go back to the slide deck. | |
| So we've talked a little bit about predictive modeling and what we mean by predictive modeling. | |
| And we've seen that you can easily fit and compare many models using validation, and model screening is all about streamlining this workflow. | |
| You can still go out and fit individual models, if you want, but model screening allows you to fit all the models at the same time in one platform and then compare these models. | |
| For more information, there are lots of different types of metrics available and many, many metrics for classification data. There's a really nice | |
| page in Wikipedia about sensitivity and specificity and all of the...all of the various metrics, and I found this really useful as I was trying to dig in and understand how to interpret the metrics that are provided. | |
| There are a lot of resources about predictive modeling in general, and also model screening in JMP Pro | |
| in our...in our user Community. Again I'll reference STIPS. STIPS is our statistical thinking for industrial problem solving course. | |
| And the seventh module is an introduction of predictive modeling, so if you're brand new to predictive modeling, I'd recommend checking this out. And there's also a blog that uses the diabetes data that's posted on our Community. And finally, you know, any new feature | |
| will tend to go through iterative improvements. Model screening was introduced in JMP Pro 16. We're already working on JMP Pro 17, and there are a lot of improvements already in the works. | |
| So if you're interested in taking a look at what's coming, it gives you an opportunity to provide feedback, | |
| you know, reach out to us about joining the early adopter program and we'll get you set up. Thank you for your time and hope you enjoy the rest of the conference. |
