Research Statistician Developer Elizabeth Claassen kicks off JMP On Air’s Predictive Modeling Day with an introduction to machine learning. Machine Learning, according to Elizabeth is the latest, current buzz-phrase meant to encompass the computer algorithms used to make decisions, predictions, or classification based on data. During this presentation, Elizabeth introduces the different types of Machine Learning algorithms – including supervised and unsupervised machine learning – and the possible trade-off of accuracy vs. interpretability. After discussing the most impactful aspects of machine learning, Elizabeth pulls up JMP to show you how well the software performs as a machine learning platform. Spoiler Alert: JMP can do machine learning very well. ... View more

My pool table set for 9-ball In a previous post, I discussed definitions of machine learning and whether JMP does it (yes, it does), and here I'm going to apply machine learning to some real data I collected. Two of my favorite machine learning algorithms, because they are easy to understand, are K Nearest Neighbors and Naïve Bayes. An algorithm doesn’t have to be complex to achieve good results, and for some problems these two algorithms perform quickly and accurately. K Nearest Neighbors is pretty much exactly what it sounds like. The algorithm calculates which of the observations in the data set are the “K” nearest to the point you want to predict. Then the prediction is either the average of the response for those nearest points (if your response variable is continuous) or is the most frequent category (if your response variable is categorical). Typically, you fit a range of Ks and, using cross-validation, choose which K provides the best predictive performance. I’m often asked why Naïve Bayes is called “Naïve.” The name comes from the fact that the key assumption that is made when using this algorithm is that all of the predictor variables are independent of each other. In many cases, that is a naïve assumption. Therefore, this method tends to perform better when that assumption is closer to true than if you have predictor variables that are highly correlated with one another. To make predictions, the probability of a category of the response (say, win or loss) given the observed predictor variables is calculated based on the probability of observing those values in the training data where that category is the response. Whichever category results in the higher calculated probability is the predicted category. Let’s take a look at these two methods in action. Who's Better? My husband and I are avid pool players and play nearly every day when we get home from work. After a year or so of playing, we thought we were pretty evenly matched, with maybe me being slightly better. I said, “We can answer this question!” So beginning Jan. 1, 2018, we began tracking our results. I keep a spreadsheet (only because the computer in the game room at home doesn’t have JMP installed on it!) that tracks the month, day of the week, date, how many games I won, how many games we played, whether I won that gaming session, and which game of pool was played (8-ball or 9-ball). Because this was in a spreadsheet first, there are also columns calculating the overall game win percentage as well as session win percentage. Thankfully, I can just drag my Excel file into my JMP Home Window, and the Excel Import Wizard works its magic to create a JMP data table. A few tweaks to make the Session Win? indicator a nominal variable and the number of games won and played to ordinal variables, and I’m ready to predict my next night of pool domination!   Imported data Except that I’m not quite ready. Remember that I mentioned with K Nearest Neighbors that cross-validation is used to determine the best fitting K? I don’t currently have a column for training and validation. While both K Nearest Neighbors and Naïve Bayes have an option to use a percentage of the data as validation in their platform launches, if I want to compare the two methods, I should use the same training set for both. Fortunately, JMP has me covered here, too, with the Make Validation Column utility. There are several options in the utility for how to create the validation column. If my response variable was highly unbalanced, which happens in cases where failures are uncommon or looking for fraudulent transactions, I would want to use the Stratified Random option to be sure I had data with both responses in the Training and Validation set. In this case, I know that all of my columns are balanced fairly well, so it’s unlikely that a category will be completely missing from my Training or Validation set. Therefore, I’ll just create my Validation Column using the Fixed Random option. I could set a seed for the random generation so I can reproduce this particular column again in the future, but I don’t think it’s necessary in this case. Analysis Time! Now, validation column in hand, I can launch the K Nearest Neighbors platform in the Analyze > Predictive Modeling menu. Session Win? is my response variable and what I would like to predict, namely for my birthday, Saturday, Oct 13. [Note: I started writing this post before my birthday; read to the end to find out whether the predictions made herein were correct!] Month, Day, Game, and Played (the number of games played) are my factors. I don’t think Date will be helpful, mostly because each Month-Date combination occurs once (except for a few weekend days when we played multiple sessions). I’m also going to have the platform calculate the nearest neighbors up to 25 rather than the default 10.     KNN Launch Dialog   The Model Selection portion of the report traces the misclassification rate calculated on each of the Training and Validation sets for the models with 1 through 25 neighbors. The selection line is set at K = 20 – which has the smallest misclassification rate on the validation set. The table below shows that just over 44% of the observations in the validation set were misclassified. The Confusion Matrix shows that roughly the same number of observations that were actually wins or losses were misclassified. The Mosaic Plot gives a visual representation of the confusion matrix.   KNN Report The misclassification rate isn’t great, but I’m not sure that my predictors are really all that predictive. If I’d been thinking about writing this post back on Jan. 1, I might have tracked things like how stressful the day had been, whether or not we were enjoying an adult beverage with our game, and maybe how much sleep I’d had the night before! So I’m not going to fault the algorithm for not giving me a great-fitting model. I’m going to save the prediction formula to the data table, so I can make my prediction about my birthday after I’ve fit these same data using Naïve Bayes.   NB Launch Dialog The first thing we notice is that the misclassification rate on the validation set is a bit smaller at just over 40%. The confusion matrix shows us that we’ve correctly classified seven more observations in the validation set with the Naïve Bayes model than the K Nearest Neighbors model. Finally, we have an ROC curve for each set. On the validation set, our area under the curve (another measure of the quality of the fit of the model) is 0.5676. While this isn’t great, it’s still better than random guessing, which would give us an AUC ≈ 0.5. NB Report I’ll save the prediction formula to the data table again to make my prediction about my birthday pool performance. Because we play a “best of 9” series when we play 9-ball, I don’t know ahead of time how many games we will play. Will I win quickly in a 5 – 0 rout? Or will it take all 9 games to determine a winner? I’ll add a row for each of the potential number of games played (5 through 9) and see what our models predict.   Added rows for prediction For this particular day, both models agree that I should be the winner unless the series goes to 9 games. In that case, K Nearest Neighbors predicts that I will lose. It’s going to be a happy birthday! A few days later, my birthday arrived and armed with the knowledge that I will win, I entered the game room. An hour or so later, I exited, dejected, having lost three games to five. At least I got cake! What Happened? Things looked so good for me according to the models! How could I have lost? The Naïve Bayes model gives me a chance to look at the estimated probability of winning in addition to the ultimate prediction. It looks like I should have taken a look at those, too, rather than get my hopes up so high. I saved the probability formulas to the data table, and it becomes clear that while I was predicted to win, the highest probability was just under 75% and the lowest was just over 60%. That means my husband actually had a pretty good chance of winning. And by looking at just the final prediction, I misjudged my actual chances of a birthday win. I’ll take that as a word of caution to look at all of the information I have available before placing my bets next time!   Win probabilities. [Note: some data table columns hidden for space.] Parting Shot A few final words on these models: Even the most advanced algorithms don’t perform well when the variables in the model aren’t very predictive of the response. We see that in action with these data and the high misclassification rates they have. As I stated earlier, this is not the fault of the algorithms, but a fault in the data I collected. Most models with reasonable predictors are much more successful than these were here. Maybe in 2019 I’ll add some more information to my data collecting and revisit these models. In addition, pool is a sporting event where the human element plays a larger role in the outcome than many (maybe even most) usual applications of machine learning. You can see this play out nearly every weekend in the fall in collegiate and professional football games where an underdog pulls off an upset. This is sometimes referred to as “any given Sunday”. However, in the long term over many trials given these probabilities, I would win my bet; just like most of the time the favored football team wins, and the sports bettors make their money. And, in fact, if I look at subsequent days after my birthday, I did win about 60% of the time. ... View more

Machine learning is the current buzz-phrase meant to encompass the computer algorithms used to make decisions, predictions, or classifications based on data. When I joined JMP in 2014, I came fresh from completing a PhD in statistics at a school with a long history of excellence in agricultural experiments and mixed modeling. This was the world I was used to: small experiments designed to maximize the information received through judicious planning, almost always some experiment design feature that required mixed model techniques to analyze (the simplest being a split-plot design), and the goal of determining what treatment(s) result in the best outcomes. This was true whether the scientist I worked with was in animal science, entomology, food science, or education. Shortly after arriving at JMP, the buzz around Big Data reached its peak. A couple of years after that came the rise in Machine Learning. It’s not surprising that Machine Learning’s rise trailed that of Big Data. What were people supposed to do with all the data they’d collected? Analyze it using the tools in the Machine Learning toolbox, of course. My formal education hadn’t particularly set me up for either of these things, but as I began my career as a Statistician Developer, these topics were what I needed to get up to speed on and quickly! So I did what every student does these days when taking first steps into a new topic: I visited Wikipedia. “Machine learning is an interdisciplinary field that uses statistical techniques to give computer systems the ability to "learn" (e.g., progressively improve performance on a specific task) from data, without being explicitly programmed.” This was a bit stunning. How is it possible for a computer system to “learn” anything without being explicitly programmed? We haven’t achieved Skynet, yet, that I’m aware! Time to keep digging. Encyclopaedia Britannica says, “Machine learning, in artificial intelligence (a subject within computer science), discipline concerned with the implementation of computer software that can learn autonomously.” Now I’ve lost my connection to statistics in the definition, but at least it’s more apparent that we’re interested in implementing software that can learn autonomously. The “without being explicitly programmed” refers to the fact that once the data are passed off to the algorithm, the algorithm does the rest without any further instruction from us. Based on these definitions and my perception of the common usage of the phrase, I’ve come to think of machine learning as being the current buzz-phrase meant to encompass the computer algorithms used to make decisions, predictions, or classifications based on data. Supervised vs. Unsupervised Learning So does JMP do machine learning? Absolutely! And we have, technically, from the very first version! Were we that ahead of our time back in 1989? I’d like to think so. But let’s look into further detail about those algorithms that people are thinking of when they say Machine Learning. Typically, Machine Learning algorithms are classified into different types – supervised and unsupervised learning. With supervised learning, the algorithms work with both the predictors (the X columns) and a response (Y column). The training data used “supervises” the algorithm by telling it what the response should be for the given values of the predictor variables. With unsupervised learning, only the predictors are used by the algorithms, and any patterns are determined solely by them. There is no “truth” provided – and in fact we may not know the “truth.”   What machine learning algorithms in JMP fall into which categories? Supervised Unsupervised Decision Trees * Clustering Neural Networks * Self-Organizing Maps Bootstrap Forest * Association Analysis Regression Singular Value Decomposition K Nearest Neighbors *   Naïve Bayes *   * Some features of these platforms are JMP Pro only. Many people are surprised to see regression on the list, and it is certainly not among what’s commonly thought of as machine learning, but it is the most straightforward form of supervised learning. And Fit Y by X was in JMP 1! Clustering was added in JMP 3, and the Machine Learning options kept coming after that. For those of you who are familiar with one or more of these algorithms, you may notice a similarity between them. Many models produce amazing predictive accuracy, but if you tried to explain what the coefficients or cut-offs mean, you would likely be hard pressed to do so. I can follow a decision tree through its cut points to arrive at a prediction, but there’s no inherent meaning behind where those cut points are. I have no idea what the parameter estimates in a neural net model mean in the “real world”, but neural nets are some of the best predictive performing Machine Learning models out there. By contrast, a regression model provides parameter estimates that have inherent meaning for users. If a coefficient is positive, we know that increasing the value of that predictor by one unit will increase the response by the coefficient value. This ties in to what Dr. Galit Shmueli refers to as “To Explain or To Predict?” One of the best compromises in this trade off is in the Generalized Regression personality of the Fit Model platform. As my colleague Clay Barker has discussed, using the Lasso or Elastic Net improves predictive performance, but the model can still be explained. But does the lack of interpretability matter? Maybe, maybe not. If the use case is strictly focused on achieving the most accurate predictions of future events, then an interpretable model isn’t necessary. On the other hand, if stakeholders want more concrete ways to “improve their score”, an explainable model may be preferred. The key for any model is can you put the output into action. Humans Are Still Needed This brings up my final point. Even with all of the amazing things that the computer algorithms can do, a human analyst is still needed to deal with the higher-level questions. What data should be included or excluded from the analysis? Some algorithms have the capability of choosing important variables, but if a variable hasn’t been given to the computer in the beginning, it won’t be picked even if it could be highly predictive! Where is the point of diminishing returns? If a model is too complex to run in the time needed, it’s not going to be useful. Perhaps a less complex model gives almost as good results and can be run quickly. A well-known example of this is the winner of the Netflix prize. The winning model performed better than almost anyone expected, but the amount of data required and the processing made it impossible to run in the real-time environment Netflix needed to suggest what users should watch next. Are there inherent biases in the model? Obviously, a computer algorithm won’t have the biases that humans have, but it’s possible for an algorithm to result in a model that highly correlates with characteristics that are biased. In mortgage lending there are characteristics that are illegal to use in making lending decisions. However, there are several legal factors which can correlate highly with illegal ones. An algorithm may generate a model that almost perfectly rejects a particular ethnicity even without ethnicity being directly part of the data. That model would likely have legal consequences. When should the model be updated? Because the point of modeling is to produce some sort of output on which we can take action, once that action is taken the model will need to be updated. This update is needed because the changes that are implemented will affect the future inputs into the model. If we continue to use the old model, we are at risk of extrapolating beyond the bounds of the original data. The point at which a model should be updated is a question only humans can answer. In my next post, I’ll tell you about two of my favorite machine learning algorithms: K Nearest Neighbors and Naïve Bayes. And we’ll take a look at these two methods in action on some fun data I collected myself. ... View more