Level: Intermediate
Laura Lancaster, JMP Principal Research Statistician Developer, SAS
Jeremy Ash, JMP Analytics Software Tester, SAS
Chris Gotwalt, JMP Director of Statistical Research and Development, SAS
Uncontrolled model extrapolation leads to two serious kinds of errors: (1) the model may be completely invalid far from the data, and (2) the combinations of variable values may not be physically realizable. Using the Profiler to optimize models that are fit to observational data can lead to extrapolated solutions that are of no practical use without any warning. JMP Pro 16 introduces extrapolation control into many predictive modeling platforms and the Profiler platform itself. This new feature in the Prediction Profiler alerts the user to possible extrapolation or completely avoids drawing extrapolated points where the model may not be valid. Additionally, the user can perform optimization over a constrained region that avoids extrapolation. In this presentation we discuss the motivation and usefulness of extrapolation control, demonstrate how it can be easily used in JMP, and describe details of our methods.
Speaker | Transcript |
| Hi, I'm Chris Gotwalt. My co | |
| presenters, Laura Lancaster and | |
| Jeremy Ash, and I are presenting | |
| an useful new JMP Pro | |
| capability called Extrapolation | |
| Control. Almost any model that | |
| you would ever want to predict | |
| with has a range of | |
| applicability, a region of the | |
| input space, where the | |
| predictions are considered to be | |
| reliable enough. Outside that | |
| region, we begin to extrapolate | |
| the model to points far from the | |
| data used to fit the model. Using | |
| the predictions from that model | |
| at those points could lead to | |
| completely unreliable | |
| predictions. There are two | |
| primary sources of | |
| extrapolation | statistical |
| extrapolation and domain based | |
| extrapolation. Both types are | |
| covered by the new feature. | |
| Statistical extrapolation occurs | |
| when one is attempting to | |
| predict using a model at an x | |
| that isn't close to the values | |
| used to train that model. | |
| Domain based extrapolation | |
| happens when you try to evaluate | |
| at an x that is impossible due | |
| to scientific or engineering | |
| based constraints. The example | |
| here illustrates both kinds of | |
| extrapolation in one example. | |
| Here we see a profiler from a | |
| model of a metallurgy production | |
| process. The prediction reads | |
| out says -2.96 with no | |
| indication that we're evaluating | |
| at a combination of temperature | |
| and pressure that is impossible | |
| in a domain sense to attain for | |
| this machine. We also have | |
| statistical extrapolation as it | |
| is far from the data used to fit | |
| the model as seen in the scatter | |
| plot of the training data on the | |
| right. In JMP Pro 16, Jeremy, | |
| Laura and I have collaborated to | |
| add a new capability that can | |
| give a warning when the profiler | |
| thinks you might be | |
| extrapolating. Or if you turn | |
| extrapolation control on, it | |
| will restrict the set of points | |
| that you see to only those that | |
| it doesn't think are | |
| extrapolating. We have two types | |
| of extrapolation control. One is | |
| based on the concept of leverage | |
| and uses a least squares model. | |
| This first type is only | |
| available in the Pro version of | |
| Fit Model least squares. The | |
| other type we call general | |
| machine learning extrapolation | |
| control and is available in the | |
| Profiler platform and several of | |
| the most common machine learning | |
| platforms in JMP Pro. Upon | |
| request, we could even add it to | |
| more. Least squares | |
| extrapolation control uses the | |
| concept of leverage, which is | |
| like a scaled version of the | |
| prediction variance. It is model- | |
| based and so it uses information | |
| about the main effects | |
| interactions in higher order | |
| terms to determine the | |
| extrapolation. For the general | |
| machine learning extrapolation | |
| control case, we had to come up | |
| with our own approach. We | |
| wanted a method that would be | |
| robust to missing values, linear | |
| dependencies, faster compute, | |
| could handle mixtures of | |
| continuous and categorical input | |
| variables, and we also | |
| explicitly wanted to separate | |
| the extrapolation model from the | |
| model used to fit the data. So | |
| when we have general | |
| extrapolation control turned on, | |
| there's only one supervised | |
| model that is...that fits the | |
| input variables to the responses | |
| that we see in the profiler | |
| traces. The profiler comes | |
| up with a quick and dirty | |
| unsupervised model to describe | |
| the training set axes, and this | |
| unsupervised model is used | |
| behind the scenes by the | |
| profiler to determine the | |
| extrapolation control | |
| constraint. So I'm having to | |
| switch because PowerPoint and my | |
| camera aren't getting along | |
| right now for some reason. We | |
| know that risky extrapolations | |
| are being made every day by | |
| people working in data science | |
| and are confident that the use | |
| of extrapolations leads to poor | |
| predictions and ultimately to | |
| poor business outcomes. | |
| Extrapolation control places | |
| guardrails on model predictions | |
| and will lead to quantifiably | |
| better decisions by JMP Pro | |
| users. When users see an extrapolation | |
| occurring, the user must make a | |
| decision about whether the | |
| prediction should be used or not | |
| used based on their domain | |
| knowledge and familiarity with | |
| the problem at hand. If you | |
| start seeing extrapolation | |
| control warnings happen quite | |
| often, it is likely the end of | |
| the life cycle for that model in | |
| time to refit it to new data | |
| because the distribution of the | |
| inputs has shifted away from | |
| that of the training data. We | |
| are honestly quite surprised and | |
| alarmed that the need for | |
| identifying extrapolation isn't | |
| better appreciated by the data | |
| science community and have made | |
| controlling extrapolation as | |
| easy and automatic as possible. | |
| Laura, who developed it in JMP | |
| Pro, will be demonstrating the | |
| option up next. Then Jeremy, who | |
| did a lot of research on our | |
| team, will go into the math | |
| details and statistical | |
| motivation for the approach. | |
| Hello, my name is Laura | |
| Lancaster and I'm here to do a | |
| demo of the extrapolation | |
| control that was added to JMP | |
| Pro 16. I wanted to start off | |
| with a fairly simple example | |
| using the fit model least | |
| squares platform. I'm gonna | |
| use some data that may be | |
| familiar; it's the Fitness data | |
| that's in sample data and I'm | |
| going to use Oxygen Uptake as | |
| my response and Run Time, Run | |
| Pulse and Max Pulse as my | |
| predictors. And I wanted to | |
| reiterate that in fit model, | |
| fit least squares the | |
| extrapolation metric that's | |
| used is leverage. So let's go | |
| ahead and start to JMP. | |
| So now I have the fitness data | |
| open in JMP and I have a script | |
| saved to the data table to | |
| automatically launch my fit | |
| least squares model. So I'm | |
| going to go ahead and run that | |
| script, it launches the least | |
| squares platform. And I have the | |
| profiler automatically open. And | |
| we can see that the profiler | |
| looks like it always has in the | |
| past, where the factor boundaries | |
| are defined by the range of each | |
| factor individually, giving us | |
| rectangular bound constraints. | |
| And when I change the factor | |
| settings, because of these bound | |
| constraints, it can be really | |
| hard to tell if you're moving | |
| far outside the correlation | |
| structure of the data. | |
| And this is why we wanted to add | |
| the extrapolation control. So | |
| this has been added to several | |
| of the platforms in JMP Pro | |
| 16, including fit least squares. | |
| And to get to the extrapolation | |
| control, you go to the menu under | |
| the profiler menu. So if I look | |
| here, I see there's a new option | |
| called Extrapolation Control. | |
| It's set to off by default, | |
| but I can turn it to either | |
| on or warning on to turn on | |
| extrapolation control. If I | |
| turn it to on, notice that | |
| it restricts my profile | |
| traces to only go to values | |
| where I'm not extrapolating. | |
| If I were to turn it to warning | |
| on, I would see the full profile | |
| traces, but I would get a | |
| warning when I go to a region | |
| where it would be considered | |
| to be extrapolation. | |
| I can also turn on extrapolation | |
| details, which I find really | |
| helpful, and that gives me a | |
| lot more information. First of | |
| all, it tells me that my | |
| metric that I'm using to | |
| define extrapolation is | |
| leverage, which is true in the | |
| fit least squares platform. | |
| And the threshold that's being | |
| used by default initially is | |
| going to be maximum leverage, | |
| but this is something I can | |
| change and I will show you that | |
| in a minute. Also, I can see | |
| what my extrapolation metric | |
| is for my current settings. | |
| It's this number right here, | |
| which will change as I change | |
| my factor settings. | |
| Anytime this number is greater | |
| than the threshold, I'm going to | |
| get this warning that I might be | |
| extrapolating. If it goes below, | |
| I will no longer get that | |
| warning. This threshold is not | |
| going to change unless I change | |
| something in the menu to adjust | |
| my threshold. So let me go ahead | |
| and do that right now. So I'm going | |
| to go to the menu | |
| and I'm going to go to set | |
| threshold criterion. So | |
| in fit least squares, you have two | |
| options for the threshold | |
| initially,it's set to maximum | |
| leverage, which is going to keep | |
| you within the convex hull of | |
| the data, or you can switch to a | |
| multiplier times the average | |
| leverage or model terms over | |
| observations. And I want to | |
| switch to that threshold. So it's | |
| set to 3 as the multiplier | |
| by default. So this is going to | |
| be 3 times the average leverage | |
| and I click OK, and notice that | |
| my threshold is going to change. | |
| It actually got smaller, so this | |
| is a more conservative | |
| definition of extrapolation. | |
| And I'm going to turn it back to | |
| on to restrict my profile traces. | |
| And now I can only go to | |
| regions where I'm within 3 | |
| times the average leverage. | |
| Now we have also | |
| implemented optimization | |
| that obeys the | |
| extrapolation | |
| constraints. So now if I | |
| turn on set desirability | |
| and I do the optimization, | |
| I will get an optimal value that | |
| satisfies the extrapolation | |
| constraint. Notice that this | |
| metric is less than or equal to | |
| the threshold. So now when I go | |
| to my next slide, which is going | |
| to compare in a graph, a scatterplot | |
| matrix, the difference | |
| between the optimal value with | |
| extrapolation control turned on | |
| and with it turned off. | |
| So this is the scatterplot | |
| matrix that I created with JMP, | |
| and it shows the original | |
| predictor variable data, as well | |
| as the predictor variable values | |
| for the optimal solution using | |
| no extrapolation control, in | |
| blue, and the optimal solution using | |
| extrapolation control in red. | |
| And notice how the unconstrained | |
| solution here in blue, | |
| right here, violates the | |
| correlation structure for the | |
| original data for run pulse and | |
| Max pulse, thus increasing the | |
| uncertainty of this prediction. | |
| Whereas the optimal solution | |
| that did use extrapolation | |
| control is much more in line | |
| with the original data. | |
| Now let's look at an example | |
| using the more generalized | |
| extrapolation control method, | |
| which we refer to as a | |
| regularized T squared method. As | |
| Chris mentioned earlier, we | |
| developed this method for models | |
| other than least squares models. | |
| So we're going to look at a | |
| neural model for the Diabetes | |
| data that is also in the sample | |
| data. The response is a measure | |
| of disease progression, and the | |
| predictors are the baseline | |
| variables. Once again, the | |
| extrapolation metric used for | |
| this example is the | |
| regularized T square that | |
| Jeremy will be describing in | |
| more detail in a few minutes. | |
| So I have the Diabetes data open in | |
| JMP and I have a script saved | |
| of my neural model fits. I'm | |
| going to go ahead and run that | |
| script. It launches the neural | |
| platform, and notice that I am | |
| using validation method, random | |
| hold back. I just wanted to note | |
| that anytime you use a | |
| validation method, the | |
| extrapolation control is based | |
| only on the training data | |
| and not your validation | |
| or test data. | |
| So I have the profiler open and | |
| you can see that it's using the | |
| full traces. Extrapolation | |
| control is not turned on. Let's | |
| go ahead and turn it on. | |
| And I'm also going to | |
| turn on the details. | |
| You can see that the traces have | |
| been restricted and the metric | |
| is the regularized T square. The | |
| threshold is 3 times the | |
| standard deviation of the sample | |
| regularized T squared. Jeremy is | |
| going to talk more about what | |
| all that means exactly in a few | |
| minutes. And I just wanted to | |
| mention that when we're using | |
| the regularized T squared | |
| method, there's only one choice | |
| for threshold, but you can | |
| adjust the multiplier. So if you | |
| go to extrapolation control, set | |
| threshold, you can adjust this | |
| multiplier, but I'm going to | |
| leave it at 3. And now I | |
| want to run optimization using | |
| extrapolation control. So I'm | |
| just going to maximize and | |
| remember. Now I have an | |
| optimal solution with | |
| extrapolation control turned | |
| on. And so now I want to look | |
| at our scatterplot matrix, just | |
| like we looked at before, with | |
| the original data, as well as | |
| with the optimal values with | |
| and without extrapolation | |
| control. | |
| So this is a scatterplot matrix | |
| of the Diabetes data that I | |
| created in JMP. It's got the | |
| original predictor values, as | |
| well as the optimal solution | |
| using extrapolation control in | |
| red, and optimal solution without | |
| extrapolation control in blue. | |
| And you can see that the red | |
| dots appear to be much more | |
| within the correlation structure | |
| of the original data than the | |
| blue, and that's particularly | |
| true when you look at this LDL | |
| versus total cholesterol. | |
| So now let's look at an example | |
| using the profiler that's under | |
| the graph menu, which I'll call | |
| the graph profiler. It also uses | |
| the regularized T squared method | |
| and it allows us to use | |
| extrapolation control on any | |
| type of model that can be | |
| created and saved as a JSL | |
| formula. It also allows us to | |
| have extrapolation control on | |
| more than one model at a time. | |
| So let's look at an example | |
| for a company that uses powder | |
| metallurgy technology to | |
| produce steel drive shafts for | |
| the automotive industry. | |
| They want to be able to find | |
| optimal settings for their | |
| production that will minimize | |
| shrinkage and also minimize... | |
| minimize failures due to bad | |
| service conditions. So we have | |
| two | responses shrinkage (which is |
| continuous and we're going to | |
| fit a least squares model for | |
| that) and surface condition (which | |
| is pass/fail and we're going to | |
| fit a nominal logistic model for | |
| that one). And our predictor | |
| variables are just some key | |
| process variables in production. | |
| And once againm the extrapolation | |
| metric is the regularized T square. | |
| So I have the powder | |
| metallurgy data open in JMP | |
| and I've already fit a least | |
| squares model for my shrinkage | |
| response, and I've already fit a | |
| nominal logistic model for the | |
| surface condition pass/fail | |
| response, and I've saved the | |
| prediction formulas to the data | |
| table so that they are ready to | |
| be used in the graph profiler. | |
| So if I go to the graph menu | |
| profiler, I can load up the | |
| prediction formula for shrinkage | |
| and my prediction formula is for | |
| the surface condition. | |
| Click OK. And now I have | |
| both of my models launched into | |
| the graph profiler. | |
| And before I turn on | |
| extrapolation control, you | |
| can see that I have the full | |
| profile traces. Once I turn on | |
| extrapolation control | |
| you can see that the traces | |
| shrink a bit, and I'm also going | |
| to turn on the details, | |
| just to show that indeed I am | |
| using the regularized T square | |
| here in this method. | |
| So what I really want to do is I | |
| want to find the optimal | |
| conditions where I minimize | |
| shrinkage and I minimize | |
| failures with extrapolation | |
| control and I want to make sure | |
| I'm not extrapolating. I want to | |
| find a useful solution. And | |
| before I can do the optimization, | |
| I actually need to set my | |
| desirabilities. So I'm going to | |
| set desirabilities. It's already | |
| correct for shrinkage, but I | |
| need to set them for the service | |
| condition. I'm going to try to maximize | |
| passes and minimize failures. | |
| K. | |
| And now I should be able to do | |
| the optimization with | |
| extrapolation controls on. | |
| Do maximize and remember. | |
| And now I have my optimal | |
| solution with extrapolation | |
| control on. So now let's look | |
| once again at the | |
| scatterplot matrix of the | |
| original data, along with the | |
| solution with extrapolation | |
| control on in the solution, | |
| with the extrapolation control | |
| off. | |
| So this is a scatterplot matrix | |
| of the powder metallurgy data | |
| that I created in JMP. And it | |
| also has the optimal solution | |
| with extrapolation control as a | |
| red dot, and the optimal | |
| solution with no extrapolation | |
| control as a blue dot. And once | |
| again you can see that when we | |
| don't enact the extrapolation | |
| control, the optimal solution | |
| is pretty far outside of the | |
| correlation structure of the | |
| data. We can especially see | |
| that here with ratio versus | |
| compaction pressure. | |
| So now I want to hand over | |
| the presentation to Jeremy | |
| to go into a lot more | |
| detail about our methods. | |
| Hi, so here are a number of | |
| goals for extrapolation control | |
| that we laid out at the | |
| beginning of the project. We | |
| needed an extrapolation metric | |
| that could be computed quickly | |
| with a large number of | |
| observations and variables, and | |
| we needed a quick way to assess | |
| whether the metric indicated | |
| extrapolation or not. This was | |
| to maintain the interactivity of | |
| the profiler traces and | |
| we needed this to | |
| perform optimization. | |
| We wanted to be able to | |
| support the various variable | |
| types available in the | |
| profiler. These are | |
| essentially continuous, | |
| categorical and ordinal. | |
| We wanted to utilize | |
| observations with missing cells, | |
| because some modeling methods | |
| will include these observations | |
| in ???. | |
| We wanted a method that was | |
| robust to linear dependencies in | |
| the data. These occur when the | |
| number of variables is larger | |
| than the number of observations, | |
| for example. And we wanted | |
| something that was easy to | |
| automate without the need for a | |
| lot of user input. | |
| For least squares models, we | |
| landed on leverage, which is | |
| often used to identify outliers | |
| in linear models. The leverage | |
| for new prediction point is | |
| computed according to this | |
| formula. There are many | |
| interpretations for leverage. | |
| One interpretation is that it's | |
| the multivariate distance of a | |
| prediction point from the center | |
| of the training data. Another | |
| interpretation is that it is a | |
| scaled prediction variance. So | |
| as prediction point moves | |
| further away from the center | |
| of the data, the uncertainty | |
| of prediction increases. And we | |
| use two common thresholds in | |
| the statistical literature for | |
| determining if this distance | |
| is too large. The first is | |
| maximum leverage, prediction | |
| points beyond this threshold | |
| or outside the convex hull of | |
| the training data. | |
| And the second is 3 times the | |
| average of the leverages. It | |
| can be shown that this is | |
| equivalent to three times the | |
| number of model terms divided | |
| by the number of observations. | |
| And as Laura described | |
| earlier, you can change the | |
| multiplier of these | |
| thresholds. | |
| Finally, when desirabilities | |
| are being optimized, the | |
| extrapolation constraint is a | |
| nonlinear constraint, and | |
| previously the profiler allowed | |
| constrained optimization with | |
| linear constraints. This type of | |
| optimization is more | |
| challenging, so Laura implemented | |
| a genetic algorithm. And if you | |
| aren't familiar with these, | |
| genetic algorithms use the | |
| principles of molecular | |
| evolution to optimize | |
| complicated cost functions. | |
| Next, I'll talk about the | |
| approach we used to generalize | |
| extrapolation control to models | |
| other than linear models. When | |
| you're constructing a predictive | |
| model in JMP, you start with a | |
| set of predictor variables and a | |
| set of response variables. Some | |
| supervised model is trained, and | |
| then a profiler can be used to | |
| visualize the model surface. | |
| There are numerous variations in | |
| the profiler in JMP. You can | |
| use the profiler internally in | |
| modeling platforms. You can | |
| output prediction formulas and | |
| build a profiler for multiple | |
| models. As Laura demonstrated, | |
| you can construct profilers for | |
| ensemble models. We wanted an | |
| extrapolation control method | |
| that would generalize all these | |
| scenarios, so instead of | |
| tying our method to a | |
| specific model, we're going | |
| to use an unsupervised | |
| approach. | |
| And we're only going to flag a | |
| prediction point as | |
| extrapolation if it's far | |
| outside where the data are | |
| concentrated in the predictor | |
| space. And this allows us to | |
| be consistent across | |
| profilers so that our | |
| extrapolation control method | |
| will plug into any profiler. | |
| The multivariate distance | |
| interpretation of leverage | |
| suggested Hotelling's T squared as | |
| a distance for general | |
| extrapolation control. In fact, | |
| some algebraic manipulation will | |
| show that Hotelling's T squared is | |
| just leverage shifted and | |
| scaled. This figure shows how | |
| Hotelling's T squared measures | |
| which ellipse an observation | |
| lies on, where the ellipses are | |
| centered at the mean of the | |
| data, and the shape is defined | |
| by the covariance matrix. | |
| Since we're no longer in | |
| linear models, this metric | |
| doesn't have the same | |
| connection to prediction | |
| variance. So instead of | |
| relying on thresholds used | |
| back in linear models, we're | |
| going to make some | |
| distributional assumptions | |
| to determine if T squared | |
| for prediction point should | |
| be considered extrapolation. | |
| Here I'm showing the formula for | |
| Hotelling's T squared. The mean and | |
| covariance matrix is estimated | |
| using the training data for the | |
| model. If P is less than N, | |
| where P is the number of | |
| predictors, N is the number | |
| of observations and if the | |
| predictors of multivariate | |
| normal, then T squared for | |
| addiction point has an F | |
| distribution. However, we wanted | |
| a method to generalize the | |
| data sets with complicated data | |
| types, like a mix of continuous | |
| and categorical data sets where P | |
| is larger than N, data sets with | |
| missing values. So instead of | |
| working out the distributions | |
| analytically in each case, we | |
| used a simple conservative | |
| control limit that we found | |
| works well in practice. This is | |
| a three Sigma control limit | |
| using the empirical distribution | |
| of T squared from the training | |
| data and, as Laura mentioned, you | |
| can also tune this multiplier. | |
| One complication is that when P | |
| is larger than N, Hotelling's T | |
| squared is undefined. There are | |
| too many parameters in the | |
| covariance matrix to estimate | |
| with the available data, and | |
| this often occurs in typical use | |
| cases for extrapolation control | |
| like in partial least squares. | |
| So we decided on a novel | |
| approach to computing Hotelling's T | |
| squared, which deals with these | |
| cases, and we're calling it a | |
| regularized T squared. | |
| To compute the covariance | |
| matrix we use a regularized | |
| estimator originally | |
| developed by Schafer and | |
| Strimmer for high | |
| dimensional genomics data. | |
| It's just a weighted | |
| combination of the full | |
| sample covariance matrix, | |
| which is U here and a | |
| constraint target matrix | |
| which is D. | |
| For the Lambda weight | |
| parameter, Schafer and Strimmer | |
| derived an analytical | |
| expression that minimizes the | |
| MSE, the estimator | |
| asymptotically. | |
| Schafer and Strimmer proposed | |
| several possible target | |
| matrices. The target matrix we | |
| chose was a diagonal matrix with | |
| the sample variances of the | |
| predictor variables on the | |
| diagonal. This target matrix has | |
| a number of advantages for | |
| extrapolation control. First, we | |
| don't assume any correlation | |
| structure between the variables | |
| before seeing the data, which | |
| works well as a general prior. | |
| Also, when there's little data | |
| to estimate the covariance | |
| matrix, either due to small N or | |
| a large fraction missing, the | |
| elliptical constraint is | |
| expanded by a large weight on | |
| the diagonal matrix, and this | |
| results in a more conservative | |
| test for extrapolation control. | |
| We found this was necessary to | |
| obtain reasonable control of the | |
| false positive rate. To put this | |
| more simply, when there's | |
| limited training data, the | |
| regularized T squared is less | |
| likely to label predictions as | |
| extrapolation, which is what you | |
| want, because you're more | |
| likely to observe covariances | |
| by chance. We have some | |
| simulation results | |
| demonstrating these details, | |
| but I don't have time to go | |
| into all that. Instead on | |
| the Community webpage, we put a | |
| link to a paper on archive and | |
| we plan to submit this to the | |
| Journal of Computational | |
| Graphical Statistics. | |
| This next slide shows some other | |
| important details we needed to | |
| consider. We needed to figure | |
| out how to deal with categorical | |
| variables. We are just | |
| converting them into indicator- | |
| coded dummy variables. This is | |
| comparable to a multiple | |
| correspondence analysis. Another | |
| complication is how to compute | |
| Hotelling's T squared when | |
| there's missing data. Several | |
| JMP predictive modeling | |
| platforms use observations with | |
| missing data to train their | |
| models. These include naive | |
| Bayes and Bootstrap forest. And | |
| these formulas are showing the | |
| pairwise deletion method we | |
| used to estimate the covariance | |
| matrix. It's more common to use | |
| row wise deletion. This means | |
| all observations with missing | |
| values are deleted before | |
| computing the covariance matrix. | |
| And this is simplest, but it can | |
| result in throwing out useful | |
| data if the sample size of the | |
| training data is small. With | |
| pairwise deletion observations | |
| and deleted only if there are | |
| missing values in the pair of | |
| variables used to compute the | |
| corresponding entry and that's | |
| what these formulas are showing. | |
| Seems like a simple thing to do. | |
| You're just using all the data | |
| that's available, but it | |
| actually can lead to a host of | |
| problems because there are | |
| different observations used to | |
| compute each entry. This can | |
| cause weird things to happen, | |
| like covariance matrices with | |
| negative eigenvalues, which is | |
| something we had to deal with. | |
| Here are a few advantages of | |
| the regularized T squared we | |
| found when comparing to other | |
| methods in our evaluations. One | |
| is that the regularization | |
| works the way regularization | |
| normally works. It strikes a | |
| balance between overfitting the | |
| training data and over biasing | |
| the estimator. This makes the | |
| estimator more robust to noise | |
| and model misspecification. | |
| Next, Schafer and Strimmer | |
| showed in their paper that | |
| regularization results in a | |
| more accurate estimator in | |
| high dimensional settings. | |
| This helps with the cursive | |
| dimensionality which plauges | |
| most distance based methods | |
| for extrapolation control. | |
| Then in the fields that have | |
| developed the methodology for | |
| extrapolation control, | |
| often they have both high | |
| dimensional data and highly | |
| correlated predictors. For | |
| example in cheminformatics and | |
| chemometrics, the chemical | |
| features are often highly | |
| correlated. Extrapolation control | |
| is often used in combination | |
| with PCA and PLS models, where | |
| T squared DModX are used to | |
| detect violations of correlation | |
| structure. This is similar to | |
| what we do in model driven | |
| multivariate control chart. | |
| Since this is a common use case, | |
| we wanted to have an option that | |
| didn't deviate too far from | |
| these methods. Our regularized T | |
| squared provides the same type | |
| of extrapolation control, but it | |
| doesn't require projection step | |
| which has some advantages. | |
| We found that this allows us to | |
| better generalized other types | |
| of predictive models. Also, in | |
| our evaluations we observed that | |
| if a linear projection doesn't | |
| work well for your data, like | |
| you have nonlinear relationships | |
| between predictors, the errors | |
| can inflate the control limits | |
| of projection based methods, | |
| which will lead to poor | |
| protection against | |
| extrapolation, and our approach | |
| is more robust than this. | |
| And then another important point | |
| is that we found the | |
| single extrapolation metric | |
| was much simpler to use and | |
| interpret. | |
| And here is a quick summary of | |
| the features of extrapolation | |
| control. The method provides better | |
| visualization of feasible | |
| regions in high dimensional | |
| models in the profiler. | |
| A new genetic algorithm has | |
| been implemented for flexible | |
| constrained optimization. | |
| Our regularized T squared | |
| handles messy observational | |
| data, cases like P larger | |
| than N, and continuous and | |
| categorical variables. | |
| The method is available in most | |
| of the predictive models in JMP | |
| 16 Pro and supports many of | |
| their idiosyncracies. It's also | |
| available in the profiler in | |
| graph, which really opens up its | |
| utility because you can operate | |
| on any prediction formula. | |
| And then as a future direction, | |
| we're considering implementing | |
| a K-nearest neighbor based | |
| constraint that would go beyond | |
| the current correlation | |
| structure constraint. Often | |
| predictors are generated by | |
| multiple distributions resulting | |
| in clustering in the predictor | |
| space. And a K-nearest neighbors | |
| based approach would enable | |
| us to control extrapolation | |
| between clusters. | |
| So thanks to everyone who | |
| tuned in to watch this and | |
| here are our emails if you have | |
| any further questions. |