Using Auto-Validation to Analyze Screening DOEs (2020-US-30MP-541)
Peter Hersh, JMP Global Technical Enablement Engineer, SAS
Phil Kay Ph.D, Learning Manager Global Enablement, SAS institute
In the process of designing experiments often many potential critical factors are identified. Investigating as many of these critical factors as possible is ideal. There are many different types of screening designs that can be used to minimize the number of runs required to investigate the large number of factors. The primary goal of screening designs is to find the active factors that should be investigated further. Picking a method to analyze these designs is critical, as it can be challenging to separate the signal from the noise. This talk will explore using the auto-validation technique developed by Chris Gotwalt and Phil Ramsey to analyze different screening designs. The focus will be on group orthogonal supersaturated designs (GO-SSDs) and definitive screening designs (DSDs). The presentation will show the results of auto-validation techniques compared to other techniques to analyze these screening designs.
Supplementary Materials
- A link to Mike Anderson's Add-in To Support Auto-Validation Workflow.
- JMP data tables for examples 1 and 2 from the presentation.
- Journal attached for an interactive demonstration of how auto-validation works for screening experiments.
- Other Discovery Summit papers about auto-validation from: Europe 2018 , US 2018, Europe 2019, US 2019 and US 2020.
- Recorded demonstration of how auto-validation works for screening experiments:
Auto-generated transcript...
|
Speaker |
Transcript |
| Peter Hersh | All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening |
| All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening | |
| designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself? | |
| All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening | |
| designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself? | |
| designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself? | |
| phil | Yes. So I'm in the global technical enablement team as well. I'm learning manager. |
| Peter Hersh | Perfect. |
| So we're going to start out at the end, show some of the results that we got while we were working through this so we | |
| did some experiments with using auto validation with some different screening DOEs and we found it a very promising technique. We were able to find more active factors than some analysis techniques. | |
| And really when we're looking at screening DOEs, what we're trying to find as many active factors as we can. And Phil, maybe you can talk a little bit about why that is. | |
| phil | Yeah. So the objective of a screening experiment is to find out which of all of your factors are actually important. So if we miss any factors |
| from our analysis of the experiment that turned out to be important, then that's a big problem. You know, we're not going to fully understand our process or our system | |
| because we're we're neglecting some important factor. So it's really critical. The most important thing is to identify which factors are important. | |
| And if we occasionally add in a factor that turns out not to be important. That's, that's less less of a problem but we really need to make sure we're capturing all of the active factors. | |
| Peter Hersh | Yeah, great, great explanation there, Phil, and I think |
| if we look at this over here on the right-hand side, our table, we we've looked at 100 different simulations of these different techniques where we looked at different signal-to-noise ratios in screening design and we found that out of those seven different techniques, | |
| we did a fairly good job when we had a higher signal-to-noise ratio, but as that dropped a little we | |
| struggled to find those less | |
| large effects. So this top one was the auto validation technique, and and we only ran that once, and we'll go into why that is, and what that running that auto validation technique did for us. But I think this was a very promising result. | |
| And that now typically, when we do a designed experiment, we don't hold out any of the data. We want to keep it intact. Phil, can you talk a little to why we wouldn't do that? | |
| phil | Yeah. |
| When we design an experiment, we are looking to find the smallest possible number of runs that give us the information that we need. So we deliberately keep the number of rows of data really as small as possible. | |
| Ideally, you know, in machine learning what you can do is you hold back some of the data to...as a way of checking how good your models are and whether you need to | |
| improve that model, whether you need to simplify that model. | |
| With design of experiments, you don't really have the luxury of just holding back a whole chunk of your data because all of it's critically important. | |
| You've designed it so that you've got the minimum number of rows of data, so there isn't really that luxury. | |
| But it would be really cool if we could find some way that we could use this this validation in some in some way. And I guess that's that's really the trick of what we're talking about today. | |
| Peter Hersh | Yeah, excellent lead in, Phil. So |
| here, here, this auto validation technique has been introduced by Chris Gotwalt, who is our head developer for pretty much everything under the analyze menu in JMP, | |
| and a university professor, Phil Ramsey. And I have two QR codes here for two different Discovery talks that they gave and | |
| if you're interested in those Discovery talks, I highly recommend them. They go into much more technical details than Phil and I are planning | |
| to go into to day about why the technique works and what they came up with. We're just trying to advertise that it's it's a pretty interesting technique and it's something that might be worth checking out for for you and show some of our results. | |
| The basic idea is we start with our original data, | |
| And then we resample that data so the data down here in gray is a repeat of this data up here in white and that is used as validation data. And how we get away with this is we used a fractional weighting system. | |
| And this has really been...it's really easy to set up with an add in that that Mike Anderson developed and | |
| there's the QR code for finding that but you can find that on the JMP user community. And it just makes this setup a lot more simple and we'll go through the setup and the use of the add in when we walk through the example in JMP. But | |
| the basic idea is it creates this validation column, this fractional waiting column, and a null factor, and we'll talk about those in a little bit. | |
| Alright, so we have a case study that both Phil and I used here and we're trying to maximize a growth rate for a certain microbe. And we're adjusting the nutrient combination. | |
| And for my example I'm looking at 10 different nutrient factors. And this nutrient factors, we went in everywhere from not having that nutrient up to some high level of having that. And | |
| this is based on a case study that you can read about here, but we just simulated the results. So we didn't have the real data. | |
| And the case study I'm going to talk to is a DSD where we have five active effects. Actually it's four and the intercept that are active. | |
| And we did a 25-run DSD and I am...I'm just looking at these 10 nutrient factors and I'm adjusting the signal-to-noise ratio for the simulated results. So that's, that's my case study and, Phil, do you want to talk to yours, a little bit? | |
| phil | Yeah, so in mine, I look to a smaller number of the factors, just six factors, in a smaller experiment, so a 13-run definitive screening design. And what I was really interested in looking at was how well this method could identify active effects when we've got |
| as many active effects as we have runs in the design of experiments. So we've got 13 rows of data and we've got 13 active parameters when we include the intercept as well. | |
| That's a really big challenge. Most the time we're not going to be able to identify the active effects using standard methods. So I was really, really interested in how this auto validation method might do in that situation. | |
| Peter Hersh | Yeah, great. So we're gonna |
| duck out of our PowerPoint. I'm going to bring in my case study here, and we'll, we'll talk about this. So here is my 25-run DSD. I have | |
| I have my results over here that are | |
| simulated. And so this is my growth rate which is impacted by some of these factors and we're in a typical screening design. We're just trying to figure out which of these factors are active, which are not active. | |
| And we might want to | |
| follow up with an augmented design or at least some confirmation runs to to make sure that our, our initial results are confirmed. So | |
| how we would set up this auto validation? So for now, in JMP 15, this is that add in that I mentioned that that Mike Anderson | |
| created and it's just called auto validation setup. And in JMP 16 this is going to be part of the product, but in JMP 15, it's an add in. | |
| And so when that I run that add in, what happens is it creates... | |
| resamples that data. So it created 25 runs that are identical to | |
| those top 25 and they're in gray. And then it added this partially... | |
| this fractional weighting here and then it added the validation and the null factor. So basically, what we're going to do is we're going to run a model on this using | |
| validation and you can use any modeling technique; generalized regression | |
| is a good one. You can use any of the variable selection techniques. You want to make sure that it can do some variable selection for you. So just to give you an idea, I'm going to go under analyze, fit model. | |
| We'll take our growth rate which is our response. We're going to take that weighting. | |
| Actually I'll | |
| change this to generalize regression. I'm going to put that weighting in as our frequency. I'm going to add that validation column that was created. | |
| This null factor that's created,and we'll talk a little bit more about that null factor. And then I'm going to just add all those 10 factors. Now in Phil's example, he's going to look at interactions and quadratic effects. I could do that here as well, but this is just to show | |
| the capability. | |
| And I'll hit Run. We'll go ahead and launch this again. I'll use lasso, you could use forward selection or anything like that. But I'll just use a lasso fit. Hit go. And then I'm going to come down here and I'm going to look at | |
| these estimates. So I what I want to do is simulate these estimates and I want to figure out which of these estimates get zeroed out | |
| most often and least | |
| often. So | |
| I would go in here and I'd hit simulate, and I could choose my number of simulations. | |
| In this case I had, I have done 100 and I won't make you sit here and watch it simulate. | |
| I can go right over here to my simulated results. So I've done 100 simulations here and I'm looking at | |
| the results from those hundred simulations and when I run the distribution, which automatically gets generated there, we can see | |
| some information about this. | |
| Now the next thing that I'm going to do is hold down control, and I'm going to customize my summary statistics. | |
| And all I want to do is remove everything except for the proportion non zero. So what that's going to do is it's going to allow me to just see the factors that were | |
| that were zeroed out or how often a factor...a certain factor was zeroed out and how often it was kept in there. So when I hit okay, all of these are changed to proportion non zero. | |
| And when I and then when I | |
| right click on here, I can make this combine data table, which I've, I've already done. | |
| And the combined data table is here. | |
| And the reason I I'm kind of going quick on this is because we can... | |
| I've added a a factor type row and just just showing, have a saved script in here, but this is...you would get these three columns out of that. | |
| Make a combined data table, so it would have your all of your factors and then how often that factor was non zero. So the higher the number, | |
| the more indicative that it's an active factor. So the last thing I'm going to do is run this Graph Builder and this shows how often | |
| a factor is added to the model. That null factor is kind of our, our line...reference line, so it has no correlation to | |
| the response. And so anything that is lower than that. we probably don't need to include in the model and then things that are higher than that, we might want to look at. And so these green ones | |
| from the simulation were the act of factors, along with the intercept and then the red ones were not active. So it did a pretty good job here. | |
| It flagged all of the good ones. | |
| phil | Yeah. |
| Peter Hersh | And we got one extra one but like Phil, you mentioned, that's not as big of a problem, right? |
| phil | Yeah, I mean, that's not the end of the world. I mean, |
| it's more, it's much more of a problem if you miss things that are active and your method tells you that they're not. And it's really impressive how it's picked out some factors here which had really low signal-to-noise ratios as well. | |
| Peter Hersh | Yes, yeah. So just to give you an idea, this was citric acid was two, EDTA was one, that was half...so half the signal to noise, and potassium nitrate was a quarter, so very low signal and it was still able to catch that. |
| Yeah, so I'm gonna pass the ball over | |
| to Phil and have him | |
| present his case study. Yeah. | |
| phil | Thanks, Pete. |
| Yeah. | |
| Well, in my case study, as I said, it's a six factor experiment and we only have 13 runs here. And I've simulated the response here so that, such that | |
| every one of my factors here is active, and the main effects are active, and the quadratic effects of each of those are active. So we've got 12 active effects, plus an intercept, to estimate or to find. | |
| And I've made them, you know, just for simplicity, I made them really high signal-to-noise ratio. | |
| So there's a signal-to-noise ratio of 3 to 1 for every one of those effects. So these are big, important effects basically as the... | |
| is what I've simulated. So what we want to find out is that all of these effects, all of these factors are really important. Now if you look to analyze this using fit DSD, which would be a standard | |
| method for this, it doesn't find the | |
| active factors. It only finds ammonium nitrate as a as an active factor. I think fit DSD struggles when there are a lot of active effects. It's very good when there are only a small number. | |
| And actually, you know, we probably wouldn't want to run a 13-run DSD and expect to get a definitive analysis. We would recommend adding some additional runs in this kind of situation. | |
| Even if we knew what the model was, so if we somehow we knew that we had six active main effects and six active quadratic effects | |
| plus the intercept, we really can't fit that model. So this is just that model fit using JMP's fit model platform, the least squares regression. And you know there's...we've got as many parameters to estimate as we have rows of data, so we've got nothing left to estimate error. | |
| So this is really all just to illustrate that this is a really big challenge, analyzing this experiment and getting out from it what we need to get out from it | |
| is a real problem. | |
| So I followed the same method as Pete. I generated this auto validation data set where we've got the repeated runs here with the fractionally weighted... | |
| fractional weightings | |
| Ran that through gen reg, so using the lasso as a model selection and then again resimulating. So simulating and each time changing out the | |
| fractional weighting and around about 250 simulations, which again I won't show you actually doing that. These are the simulation results that we got, the distributions here, and you can see that it's | |
| picking out citric acid. So the some of the times the models had a zero for the parameter estimate for citric acid, but a lot of the time | |
| it was getting the parameter estimate to be about three, which is what it was simulated as originally, and what it should be getting. And you can see that for then, some of these other effects, which was simulated to not be active then, by and large, they are | |
| estimated to have a parameter size of zero, which is what we want to see. And just looking at the proportion, nonzero as Pete did there. | |
| And I've added in all the, the effect types here because here I was looking at the main effects, the quadratic and the interactions. And what the method should find is that the main effects and the quadratics are all active, but the two factor interactions were not. | |
| And when we look at that, | |
| just plotting that proportion non zero for each of these effects, you can see, first of all, the, the null factor that we've added in there. | |
| And anything above that, that's with a higher proportion non zero was suggesting, that's an active effect. And you can see, well, first of all, the intercept, which is always there. | |
| We've got the main effects, which | |
| we're declaring as active using this method. They've got a higher proportion nonzero than the null factor and the quadratics. | |
| And we can see the all of the two factor interactions, the proportion nonzero was was much, much lower. So it's done an amazingly good job of finding the effects that I'd simulated to be active | |
| in this very challenging situation, which I think is, is really very exciting. | |
| That's just one little exploration of this method. To me that that's a very exciting result and it makes me very excited about looking at this more. So I just wanted to finish with | |
| some of the concluding remarks. And I think, Pete, it's fair to say we're not saying that everybody should go and throw away everything else that they've done in the past and only use this method now. | |
| Peter Hersh | Yeah, absolutely. We've seen some exciting results. I think, Chris, Chris is seeing exciting results, but this is not the end all, be all to always use auto validation, but it's a new tool in your tool belt. |
| phil | Yeah, I mean I think I'll certainly use it every time, but I'm not saying only use that. I think there's always...you always want to look at things from different angles using all the available tools that you've got. |
| And so it clearly shows a lot of promise, and we focused on the screening situation where we're trying to identify the active effects from screening experiments and we've looked at DSDs. I've looked be briefly other screen designs like group orthogonal super saturated designs, | |
| and it does a good job there from my quick explorations. I'd see no reason why it won't do well in fractional factorial, custom screening designs. | |
| And it seems to be working in situations where the standard methods just fall down. The situation that I showed was a very extreme example, it's probably not a very realistic example. | |
| But it really pushes the methods and the standard methods are going to fall down in that kind of situation. Whereas this auto validation method, | |
| it seems to do what it should do. It gives you the results that you you need to get from that kind of situation. And so it's very exciting. I think we're waiting for some the results of more rigorous simulation studies that are being done by | |
| Chris Gotwalt and Phil Ramsay and a PhD student that are supervising. | |
| But it really does open up a whole | |
| a whole load of new opportunities. I think, Pete, it's just very exciting, isn't it? | |
| Peter Hersh | Absolutely. Really exciting technique and thank you everyone for coming to the talk. |
| phil | Yeah, thank you. |
