Choose Language Hide Translation Bar

Using Auto-Validation to Analyze Screening DOEs (2020-US-30MP-541)

Level: Intermediate

 

Peter Hersh, JMP Global Technical Enablement Engineer, SAS
Phil Kay Ph.D, Learning Manager Global Enablement, SAS institute

 

In the process of designing experiments often many potential critical factors are identified. Investigating as many of these critical factors as possible is ideal. There are many different types of screening designs that can be used to minimize the number of runs required to investigate the large number of factors. The primary goal of screening designs is to find the active factors that should be investigated further. Picking a method to analyze these designs is critical, as it can be challenging to separate the signal from the noise. This talk will explore using the auto-validation technique developed by Chris Gotwalt and Phil Ramsey to analyze different screening designs. The focus will be on group orthogonal supersaturated designs (GO-SSDs) and definitive screening designs (DSDs).  The presentation will show the results of auto-validation techniques compared to other techniques to analyze these screening designs.

 

 

 

Supplementary Materials

 

  1. A link to Mike Anderson's Add-in To Support Auto-Validation Workflow.
  2. JMP data tables for examples 1 and 2 from the presentation. 
  3. Journal attached for an interactive demonstration of how auto-validation works for screening experiments.
  4. Other Discovery Summit papers about auto-validation from: Europe 2018 , US 2018, Europe 2019, US 2019 and US 2020
  5. Recorded demonstration of how auto-validation works for screening experiments:

Auto-generated transcript...

 


Speaker

Transcript

Peter Hersh All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening
  All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening
  designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself?
  All right, well thanks for tuning into watch Phil and I's Discovery presentation. We are going to be talking today about a new ??? technique in JMP that we are both really excited about and that's using auto validation to analyze screening
  designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself?
  designs for for DOEs. So my name is Peter Hersch. I'm a senior systems engineer and part of the global technical enablement team, and my co-author who's joining me is Phil. Do you want to introduce yourself?
phil Yes. So I'm in the global technical enablement team as well. I'm learning manager.
Peter Hersh Perfect.
  So we're going to start out at the end, show some of the results that we got while we were working through this so we
  did some experiments with using auto validation with some different screening DOEs and we found it a very promising technique. We were able to find more active factors than some analysis techniques.
  And really when we're looking at screening DOEs, what we're trying to find as many active factors as we can. And Phil, maybe you can talk a little bit about why that is.
phil Yeah. So the objective of a screening experiment is to find out which of all of your factors are actually important. So if we miss any factors
  from our analysis of the experiment that turned out to be important, then that's a big problem. You know, we're not going to fully understand our process or our system
  because we're we're neglecting some important factor. So it's really critical. The most important thing is to identify which factors are important.
  And if we occasionally add in a factor that turns out not to be important. That's, that's less less of a problem but we really need to make sure we're capturing all of the active factors.
Peter Hersh Yeah, great, great explanation there, Phil, and I think
  if we look at this over here on the right-hand side, our table, we we've looked at 100 different simulations of these different techniques where we looked at different signal-to-noise ratios in screening design and we found that out of those seven different techniques,
  we did a fairly good job when we had a higher signal-to-noise ratio, but as that dropped a little we
  struggled to find those less
  large effects. So this top one was the auto validation technique, and and we only ran that once, and we'll go into why that is, and what that running that auto validation technique did for us. But I think this was a very promising result.
  And that now typically, when we do a designed experiment, we don't hold out any of the data. We want to keep it intact. Phil, can you talk a little to why we wouldn't do that?
phil Yeah.
  When we design an experiment, we are looking to find the smallest possible number of runs that give us the information that we need. So we deliberately keep the number of rows of data really as small as possible.
  Ideally, you know, in machine learning what you can do is you hold back some of the data to...as a way of checking how good your models are and whether you need to
  improve that model, whether you need to simplify that model.
  With design of experiments, you don't really have the luxury of just holding back a whole chunk of your data because all of it's critically important.
  You've designed it so that you've got the minimum number of rows of data, so there isn't really that luxury.
  But it would be really cool if we could find some way that we could use this this validation in some in some way. And I guess that's that's really the trick of what we're talking about today.
Peter Hersh Yeah, excellent lead in, Phil. So
  here, here, this auto validation technique has been introduced by Chris Gotwalt, who is our head developer for pretty much everything under the analyze menu in JMP,
  and a university professor, Phil Ramsey. And I have two QR codes here for two different Discovery talks that they gave and
  if you're interested in those Discovery talks, I highly recommend them. They go into much more technical details than Phil and I are planning
  to go into to day about why the technique works and what they came up with. We're just trying to advertise that it's it's a pretty interesting technique and it's something that might be worth checking out for for you and show some of our results.
  The basic idea is we start with our original data,
  And then we resample that data so the data down here in gray is a repeat of this data up here in white and that is used as validation data. And how we get away with this is we used a fractional weighting system.
  And this has really been...it's really easy to set up with an add in that that Mike Anderson developed and
  there's the QR code for finding that but you can find that on the JMP user community. And it just makes this setup a lot more simple and we'll go through the setup and the use of the add in when we walk through the example in JMP. But
  the basic idea is it creates this validation column, this fractional waiting column, and a null factor, and we'll talk about those in a little bit.
  Alright, so we have a case study that both Phil and I used here and we're trying to maximize a growth rate for a certain microbe. And we're adjusting the nutrient combination.
  And for my example I'm looking at 10 different nutrient factors. And this nutrient factors, we went in everywhere from not having that nutrient up to some high level of having that. And
  this is based on a case study that you can read about here, but we just simulated the results. So we didn't have the real data.
  And the case study I'm going to talk to is a DSD where we have five active effects. Actually it's four and the intercept that are active.
  And we did a 25-run DSD and I am...I'm just looking at these 10 nutrient factors and I'm adjusting the signal-to-noise ratio for the simulated results. So that's, that's my case study and, Phil, do you want to talk to yours, a little bit?
phil Yeah, so in mine, I look to a smaller number of the factors, just six factors, in a smaller experiment, so a 13-run definitive screening design. And what I was really interested in looking at was how well this method could identify active effects when we've got
  as many active effects as we have runs in the design of experiments. So we've got 13 rows of data and we've got 13 active parameters when we include the intercept as well.
  That's a really big challenge. Most the time we're not going to be able to identify the active effects using standard methods. So I was really, really interested in how this auto validation method might do in that situation.
Peter Hersh Yeah, great. So we're gonna
  duck out of our PowerPoint. I'm going to bring in my case study here, and we'll, we'll talk about this. So here is my 25-run DSD. I have
  I have my results over here that are
  simulated. And so this is my growth rate which is impacted by some of these factors and we're in a typical screening design. We're just trying to figure out which of these factors are active, which are not active.
  And we might want to
  follow up with an augmented design or at least some confirmation runs to to make sure that our, our initial results are confirmed. So
  how we would set up this auto validation? So for now, in JMP 15, this is that add in that I mentioned that that Mike Anderson
  created and it's just called auto validation setup. And in JMP 16 this is going to be part of the product, but in JMP 15, it's an add in.
  And so when that I run that add in, what happens is it creates...
  resamples that data. So it created 25 runs that are identical to
  those top 25 and they're in gray. And then it added this partially...
  this fractional weighting here and then it added the validation and the null factor. So basically, what we're going to do is we're going to run a model on this using
  validation and you can use any modeling technique; generalized regression
  is a good one. You can use any of the variable selection techniques. You want to make sure that it can do some variable selection for you. So just to give you an idea, I'm going to go under analyze, fit model.
  We'll take our growth rate which is our response. We're going to take that weighting.
  Actually I'll
  change this to generalize regression. I'm going to put that weighting in as our frequency. I'm going to add that validation column that was created.
  This null factor that's created,and we'll talk a little bit more about that null factor. And then I'm going to just add all those 10 factors. Now in Phil's example, he's going to look at interactions and quadratic effects. I could do that here as well, but this is just to show
  the capability.
  And I'll hit Run. We'll go ahead and launch this again. I'll use lasso, you could use forward selection or anything like that. But I'll just use a lasso fit. Hit go. And then I'm going to come down here and I'm going to look at
  these estimates. So I what I want to do is simulate these estimates and I want to figure out which of these estimates get zeroed out
  most often and least
  often. So
  I would go in here and I'd hit simulate, and I could choose my number of simulations.
  In this case I had, I have done 100 and I won't make you sit here and watch it simulate.
  I can go right over here to my simulated results. So I've done 100 simulations here and I'm looking at
  the results from those hundred simulations and when I run the distribution, which automatically gets generated there, we can see
  some information about this.
  Now the next thing that I'm going to do is hold down control, and I'm going to customize my summary statistics.
  And all I want to do is remove everything except for the proportion non zero. So what that's going to do is it's going to allow me to just see the factors that were
  that were zeroed out or how often a factor...a certain factor was zeroed out and how often it was kept in there. So when I hit okay, all of these are changed to proportion non zero.
  And when I and then when I
  right click on here, I can make this combine data table, which I've, I've already done.
  And the combined data table is here.
  And the reason I I'm kind of going quick on this is because we can...
  I've added a a factor type row and just just showing, have a saved script in here, but this is...you would get these three columns out of that.
  Make a combined data table, so it would have your all of your factors and then how often that factor was non zero. So the higher the number,
  the more indicative that it's an active factor. So the last thing I'm going to do is run this Graph Builder and this shows how often
  a factor is added to the model. That null factor is kind of our, our line...reference line, so it has no correlation to
  the response. And so anything that is lower than that. we probably don't need to include in the model and then things that are higher than that, we might want to look at. And so these green ones
  from the simulation were the act of factors, along with the intercept and then the red ones were not active. So it did a pretty good job here.
  It flagged all of the good ones.
phil Yeah.
Peter Hersh And we got one extra one but like Phil, you mentioned, that's not as big of a problem, right?
phil Yeah, I mean, that's not the end of the world. I mean,
  it's more, it's much more of a problem if you miss things that are active and your method tells you that they're not. And it's really impressive how it's picked out some factors here which had really low signal-to-noise ratios as well.
Peter Hersh Yes, yeah. So just to give you an idea, this was citric acid was two, EDTA was one, that was half...so half the signal to noise, and potassium nitrate was a quarter, so very low signal and it was still able to catch that.
  Yeah, so I'm gonna pass the ball over
  to Phil and have him
  present his case study. Yeah.
phil Thanks, Pete.
  Yeah.
  Well, in my case study, as I said, it's a six factor experiment and we only have 13 runs here. And I've simulated the response here so that, such that
  every one of my factors here is active, and the main effects are active, and the quadratic effects of each of those are active. So we've got 12 active effects, plus an intercept, to estimate or to find.
  And I've made them, you know, just for simplicity, I made them really high signal-to-noise ratio.
  So there's a signal-to-noise ratio of 3 to 1 for every one of those effects. So these are big, important effects basically as the...
  is what I've simulated. So what we want to find out is that all of these effects, all of these factors are really important. Now if you look to analyze this using fit DSD, which would be a standard
  method for this, it doesn't find the
  active factors. It only finds ammonium nitrate as a as an active factor. I think fit DSD struggles when there are a lot of active effects. It's very good when there are only a small number.
  And actually, you know, we probably wouldn't want to run a 13-run DSD and expect to get a definitive analysis. We would recommend adding some additional runs in this kind of situation.
  Even if we knew what the model was, so if we somehow we knew that we had six active main effects and six active quadratic effects
  plus the intercept, we really can't fit that model. So this is just that model fit using JMP's fit model platform, the least squares regression. And you know there's...we've got as many parameters to estimate as we have rows of data, so we've got nothing left to estimate error.
  So this is really all just to illustrate that this is a really big challenge, analyzing this experiment and getting out from it what we need to get out from it
  is a real problem.
  So I followed the same method as Pete. I generated this auto validation data set where we've got the repeated runs here with the fractionally weighted...
  fractional weightings
  Ran that through gen reg, so using the lasso as a model selection and then again resimulating. So simulating and each time changing out the
  fractional weighting and around about 250 simulations, which again I won't show you actually doing that. These are the simulation results that we got, the distributions here, and you can see that it's
  picking out citric acid. So the some of the times the models had a zero for the parameter estimate for citric acid, but a lot of the time
  it was getting the parameter estimate to be about three, which is what it was simulated as originally, and what it should be getting. And you can see that for then, some of these other effects, which was simulated to not be active then, by and large, they are
  estimated to have a parameter size of zero, which is what we want to see. And just looking at the proportion, nonzero as Pete did there.
  And I've added in all the, the effect types here because here I was looking at the main effects, the quadratic and the interactions. And what the method should find is that the main effects and the quadratics are all active, but the two factor interactions were not.
  And when we look at that,
  just plotting that proportion non zero for each of these effects, you can see, first of all, the, the null factor that we've added in there.
  And anything above that, that's with a higher proportion non zero was suggesting, that's an active effect. And you can see, well, first of all, the intercept, which is always there.
  We've got the main effects, which
  we're declaring as active using this method. They've got a higher proportion nonzero than the null factor and the quadratics.
  And we can see the all of the two factor interactions, the proportion nonzero was was much, much lower. So it's done an amazingly good job of finding the effects that I'd simulated to be active
  in this very challenging situation, which I think is, is really very exciting.
  That's just one little exploration of this method. To me that that's a very exciting result and it makes me very excited about looking at this more. So I just wanted to finish with
  some of the concluding remarks. And I think, Pete, it's fair to say we're not saying that everybody should go and throw away everything else that they've done in the past and only use this method now.
Peter Hersh Yeah, absolutely. We've seen some exciting results. I think, Chris, Chris is seeing exciting results, but this is not the end all, be all to always use auto validation, but it's a new tool in your tool belt.
phil Yeah, I mean I think I'll certainly use it every time, but I'm not saying only use that. I think there's always...you always want to look at things from different angles using all the available tools that you've got.
  And so it clearly shows a lot of promise, and we focused on the screening situation where we're trying to identify the active effects from screening experiments and we've looked at DSDs. I've looked be briefly other screen designs like group orthogonal super saturated designs,
  and it does a good job there from my quick explorations. I'd see no reason why it won't do well in fractional factorial, custom screening designs.
  And it seems to be working in situations where the standard methods just fall down. The situation that I showed was a very extreme example, it's probably not a very realistic example.
  But it really pushes the methods and the standard methods are going to fall down in that kind of situation. Whereas this auto validation method,
  it seems to do what it should do. It gives you the results that you you need to get from that kind of situation. And so it's very exciting. I think we're waiting for some the results of more rigorous simulation studies that are being done by
  Chris Gotwalt and Phil Ramsay and a PhD student that are supervising.
  But it really does open up a whole
  a whole load of new opportunities. I think, Pete, it's just very exciting, isn't it?
Peter Hersh Absolutely. Really exciting technique and thank you everyone for coming to the talk.
phil Yeah, thank you.
Comments
Amir_H

Hi @Peter_Hersh. I appreciate it if you can answer this question: after we pick the important variables by comparing them to the null factor, do we need to go back to the original data set prior to auto-validation and use the chosen variables (main or interaction) and run the model again to find the predictive equation? or we can use the data set with weighted validation column and no null factor to do so? or we can use the simulated table? If the latter can be used, how to get rid of the null factor in the final equation?

I am kind of stuck here or am missing a point.

Thanks.

@Amir_H,  Great question.  For the auto-validation technique, Phil and I show off here the idea is variable selection.  If you would like to make a predictive model using this technique you should use Self-Validating Ensemble models (SVEM).  The basic idea with that is that you make many different models each time you re-calculate the weights and save the model results.  After making many of these models you take the average of all of the models and typically get good results.  Below is a link to a talk we gave that covers that technique and has an example script.

 

Re-Thinking the Design and Analysis of Experiments? (2021-EU-30MP-776) 

 

Amir_H

Thanks, @Peter_Hersh. As far as I am learning about this technique, it's very useful when the dataset is very small and for variable selection there as you said. It has not helped me much yet because my dataset is not too small. Also, based on my experience, sometimes you may have to use other techniques to get rid of large VIFs and GenReg may not help. I am still trying to learn how to make the best out of either the Add-in (which only uses GenReg) or method (script).

Article Tags
Contributors