Avoid Wasting Time Investigating False Effects (2021-EU-30MP-771)

[#1]

Level: Intermediate

Jose Goncalves, Upstream Scientist, Process R&D Department, Oxford Biomedica
Rebecca Clarke, Upstream Senior Scientist, Process R&D Department, Oxford Biomedica
George Pamenter, Downstream Process R&D Scientist, Oxford Biomedica
Thomas Chippendale, Upstream Senior Scientist, Process R&D Department, Oxford Biomedica

A significant growth in the generation of cell culture bioprocess data has been observed at Oxford Biomedica in recent years. This increase in data collection is not only driven by intensive process development and characterization studies, but also due to incorporation of high throughput production systems. Throughout a cell culture process, many online and offline process parameters are recorded. However, due to the complexity of biological systems, one of the main challenges is the identification of genuine factors that can influence process performance or product yield. Holdback validation is often used to avoid overfitting and prevent the inclusion of non-genuine terms in a model, but the use of a single validation column may not be effective in the prediction of the best model. This presentation will demonstrate how refitting models using multiple validation columns can allow the identification of the simplest and most frequently occurring model. Data from 67 bioreactor production runs containing 20 candidate predictors were used for analysis. The simulate function in JMP® Pro was used to randomly generate 100 validation columns.

Auto-generated transcript...

Speaker	Transcript
Rebecca Clarke	Okay hi everyone, and my name is Rebecca and I'm joined today by Jose and George and today we're going to talk to you about how we use the JMP Pro validation methods to help us to stop wasting time investigating false effects.
	So firstly I'm just going to introduce us as a company and what we do so. We work for Oxford Biomedica.
	We're pioneers in the gene and cell therapy business, and we have a leading position in lentiviral vector research, development and bioprocessing.
	So we have over 20 years experience in the field. We were the first company to treat humans with a lentiviral vector(?) based therapy. Some of the
	local partners that we work with are listed here in the bottom right, so we work with Novartis, Sonofi, Boehringer Ingelheim, and Orchard Therapeutics and some of the conditions that we treat include cancer, central nervous system disorders, and cystic fibrosis just to name a few.
	So last year, Oxford Biomedia also joined the fight against COVID-19 when we signed up to the UK vaccine consortium.
	And the consortium was run by the Jenner Institute of Oxford University and the goal was to rapidly develop a COVID-19 vaccine candidate.
	And by joining this consortium, Oxford Biomedica allowed for this scaled up manufacturing of the vaccine by allowing the consortium access to our state of the art ??? suites. So following on from that, we then signed a clinical and commercial supply agreement with AstraZeneca.
	And, with this agreement, we agreed to supply the AstraZeneca COVID vaccine should it prove to be successful in the clinical trials. And as I'm sure you're all aware, the vaccine was indeed successful and Oxford Biologica continue to manufacture the COVID-19 vaccine to this day.
	So I also just want to briefly introduce the processes that we use here and then at OXB.
	So a bioprocess is a process that uses a living cell or their components to obtain a desired product, so a well known
	example of such a process would be the fermentation of alcoholic beverages.
	And during this process, the yeast is the living component and the yeast plus all the other ingredients needed to brew beer are put into a bioreactor and the yeast initiates a chemical reaction called fermentation. During fermentation glucose is converted into ethanol.
	And then, once this fermentation reaction is complete, the contents of the bioreactor is then harvested and purified via filtration and then we're left with the final product that's beer.
	So the process that we utilize here in Oxford Biomedica is similar to fermentation. So the living component of our process are
	cell cultures, so with the bioreactors we put into cell cultures and also the other components needed to make the lentiviral factors.
	At the end of this production process, the contents of the bioreactor are passed on for downstream processing, where the product is purified and then it gets bottled and shipped out for use in patients.
	So this slide is just showing the life cycle of drug development. So on the left hand side at the very bottom is the research stage.
	So it's at this stage where we identify target genes or proteins that are important in disease mechanisms.
	And, and we also develop drugs against these targets and we show whether the drug is successful in its ability of targeting the disease.
	And then, at the very end of the life cycle, we have commercial manufacturing, where the drug is made and bottled for use in patients.
	So our group, the Process, Research and Development group, sit in between these two stages. It's our job to design, optimize, and scale up the production process
	and move drug product from the research bench through to manufacturing. Some of the processes that we use include small scale shake flasks and we work right the way up to 50 liter bioreactors.
	So, as you can imagine, we generate a lot of data during the course of our work and the bioreactors have many input parameters that we need to monitor and optimize.
	With the sheer volume of the data that we generate, it makes it quite difficult to discern which of the parameters we should focus on during our optimization.
	And just because of the sheer number of parameters that we have to work with, it's not commercially viable for us to optimize each one individually, so at this point, we look to statistical modeling.
	And we use the models to try and narrow down the list of potential parameters that we should optimize.
	And so, while these models are useful, we do generate a lot of random noise during our work, and this is down to the living nature of bio processing, so no two cultures are the same.
	So this random noise obviously gets incorporated into our statistical models and this can lead to us wasting time looking at non genuine effects during our optimization experiments.
	So it was with this in mind that we looked to use JMP Pro, specifically the validation features,
	which will hopefully help us to further narrow down the list of parameters that we'll look at, remove the non genuine effects from the models and also
	to avoid us overfitting our models as well. So now I'm going to hand over to Jose who's going to talk you through the whole data validation method.
Jose Goncalves	Right, my name is Jose and I'm going to speak about the whole optimization method, so this is a method, this is a modeling tool only available on JMP Pro. We were interested to explore how to use...how useful it could be at OXB for...
	I saw how useful it could be
	to OXB for modeling of bioprocess data. So as Rebecca was saying,
	we use these steritank(?) reactors in our process and basically we grow cells into these bioreactors and we produce these viral vectors, which are our
	main product. It's a...it's what we want to maximize and we want to model, so this is our response variable really.
	And bioreactors require many control inputs and produce a lot of data during one production process, so having a tool to help determine which inputs outputs are worthy of investigation,
	it would be of great value for us. It will allow us to save time, free up resources, and allow for a more targeted approach to process investigations.
	So before jumping into model building we we had to first collect as much data as we could from historical batches.
	And so, this data gives us indications on how our production process is running and any abnormalities in the data, might indicate a detrimental effect on our vector titers. So here on the left side, you can see that
	we collected the data from 20 variables, unfortunately, unfortunately we couldn't show the actual names of the variables and we had to anonymize the data.
	So we list them here from X1 to X20 and since our our process is quite a dynamic process, we cannot expect the degree of colinearity between predictors. So
	in summary, we have a data set that contains 20 variables or 20 candidate predictors to our model, and we, and we have collected the data from 67 historical batches.
	So, since this was part of JMP Pro trial, we thought it would be a great thing to do in the beginning to use standard JMP to build the model and then use it as a comparison to to to to to another model built using the hold out formulation method.
	Here, to build the model without the validation method, we use the standard modeling platform on JMP and
	we selected here response surface model because we wanted to allow all main effects, interaction terms and square terms to enter into the model. And here, in the personality, we chose to do a stepwise progression. So here on the right hand side,
	we have the stepwise regression control panel and we use the stopping rule to be our P value threshold, so the remaining parameters, we left them by default.
	And then, after running the stepwise regression, this was the final model that was generated.
	So here, you can see that we have eight terms into the model and seven of them, they are actually significant. So I would like to bring your attention just for these first three three factors here, because they have...they are highly significant in our model, and they will be
	very important in in a couple of slides when I'm going to show
	the results...the results
	when we approach using the hold out validation method, so the old model here is significant and can explain about 44% of the variation in our data set.
	And so, this was kind of our initial approach.
	We were happy with with with this first model that was generated, but we were more interested to know more about this about validation methods so.
	Because to investigate all of these predictors it would take a considerable amount of time and laboratory resources so that was the approach we took next.
	So here is the an overview on how this holdout validation method works.
	And, essentially, we have our data set and our data set will be randomly split into two groups. So on the left hand side, 70% of the data
	will be, it will be assigning into a training set, which is the proportion of data that will be used to exclusively build the model.
	And here on the right hand side we have 30% of the data that we will be assigning to a validation set, and this is the proportion of data that will be used
	to stop our model building process when this R squared of the validation set, which is its maximum. So here at the bottom,
	we can see, this is a screenshot of the stepwise regression platform, which is essentially the same as we used before, but there was as a stopping rule, which shows this to be,
	the maximum validation R squared instead of our p value threshold. So and here, this graph at the bottom, it shows
	we have the training set and we have the validation set here, which is labeled excluded.
	And so, as you can see, the training set, as we start entering terms into the model,
	because we have this rule here, this forward direction rule, we start with zero terms into the model, and then we go in a stepwise fashion to add terms into the model.
	As we start adding those terms, the R squared increases quite fast, up to when we have five terms into the model and afterwards the there is not much of an increase in the improvement in the R squared
	when we start...when we keep adding more and more terms. And so one of the important things when we use this method is also what happens into the validation set,
	because this is related to our stopping rule. And so what happens is essentially the same as we start adding terms into the model,
	the R squared increases quite fast, up to when we reach the maximum R squared and then after that point, as we keep adding more and more terms the R squared decreases.
	And so, this is exactly what what this validation set does is to...what...we want to use this validation set to stop the model building process at this stage, so we can include all these these terms here that explain most of the variation in our data set and avoid
	the inclusion of all these terms in the model that might create model overfitting or or they might not actually be genuine at all.
	So after running the stepwise regression, so this was the final model that was generated and, as you can see, we have we had a reduction from eight or seven significant terms into only three. And, as you recall, these terms here they were highly significant when we didn't use the
	the the the hold out validation method and so here they don't appear to be significant at all.
	And actually this model is only able to explain about 4% of the of the variation in our data set.
	And so, this was quite...we we thought this was quite a dramatic decrease in the number of terms in the model and we start thinking, okay, maybe we were a bit unlucky with the validation set that was generated for for
	for building this particular model and we thought we... One of the great things in JMP Pro is that we can use this generalized regression platform, which is essentially...the output is essentially it looks the same as in a stepwise regression, but in this
	generalized regression platform, we are able to use this simulation function. And what this basically does is to rerun the model over and over again, as many times as we want.
	And then use it for each model that is generated, it uses a different validation set. so this was the approach we took next.
	And because
	we couldn't list all of the all of the models that we generated, essentially we just...we tried to generate 100 models, or we did 100 simulations of models. So we couldn't list them all in here.
	And the way we we wanted to to summarize the results is to...we tried to count the number of times that a particular factor was picked up to be included in the model after those 100 simulations.
	And so, this is exactly what this graph is showing here. So on the right...on the left hand side
	we have just...these are just the parameter estimates. This is good just to observe the scatter of the data. And here,
	as you can see, we have the X13, X8 and it's this interaction terms that were actually significant when we didn't use this method. And here if we take as an example, X13, it was only...it was only picked up 46 times
	to be included in the model or 46% of the time to include in the model, and so, which is quite a low number, as we were expecting since
	these factors here, they were highly significant when we didn't use this method. So
	the conclusion that we take on using the the hold out validation method is that it's definitely a good tool to have for scientists and especially when we when we build models and
	we have these variables that come out as significant and they don't quite fit in our scientific understanding. So another one...another conclusion that we take is that
	specifically for this study is that we can safely assume that the process attributes that we studied do not significantly impact our vector titers. So I'm going to hand over to George now. He's going to speak about the autovalidation.
George Pamenter	Okay hi everyone, my name is George. I work in the purification group here Oxford Biomedica. I'm going to be talking a little bit about this autovalidation
	method that was put to us from JMP and its specific use in design of experiments, like data structure. And I'm going to talk a little bit about how we've used that methodology to actually remove a variable from an optimization model of viral vector purification.
	Okay, so just to start with a little bit of background on a viral vector purification for those that don't know.
	So essentially we receive this cell culture material from the bioreactors.
	And that material is quite crude material, so includes our therapeutic of interest are very viral vectors, that also includes thousands of other species, contaminant species.
	And they can be anything from unwanted proteins to bacterial DNA or sometimes even ineffective viruses.
	And so the aim of purification really is to separate our pure viral vector product from all these different contaminants.
	But these can be incredibly complicated processes, and you know, involving chromatography or enzymatic treatments and a number of different factors that affect their performance. And so we need a way of efficiently selecting the best conditions to reach the best purity.
	And the way we do that a lot of the time is by use of design of experiments. And design of experiments that are common throughout the bioprocess development timeline,
	we essentially use them to screen, to optimize, and to fully characterize our bioprocesses.
	Essentially, as we move through the drug development timeline, as we move towards commercial manufacturing, the size of our experiments goes up.
	The number of experiments we can do actually goes down, and it becomes harder and harder for us to change things the closer we get to a commercial product.
	And so we need to be able to make smart decisions early on in the development timeline and the way we do this is by
	developing effective models. So if we have an effective model early on, it sort of affords us the confidence that we've chosen the best conditions as we go through our scale of procedure.
	So I'm just going to detail how we've used this autovalidation technology on a specific example. So we conducted the response surface design of experiments
	on a viral vector purification step and again we're not able to share the actual data with you but we've had to normalize it here.
	So this was a three factor design space and our...and our output or response variable of interest is this impurity level. So this impurity level could be the amount of contaminating DNA protein, something of that description.
	So the goal of this really was to optimize these models for the lowest impurity level possible.
	And this is how we sort of looked initially in our JMP setups, there are three input variables and this response impurity.
	So we initially built some models, using the stepwise regression platform and also the Lasso regression, that was a JMP Pro feature.
	And actually at the beginning, we were we were quite pleased with what we were seeing so, we were explaining quite a lot of the data.
	We seemed to be explaining it quite well and we started seeing the variables we would expect to see crop up.
	But there was one variable that kind of made us less sure, and that was this X1 X3 interaction, you can see I've highlighted here.
	If you draw your attention to the top right of the screen, you can just see that this X1 profile completely changes depending on
	the value of X3, so at low values of X 3, we've got this kind of no real effect to be honest, and then a high values of X3, we've got this kind of positive correlation.
	And that actually really confused us and it was a little bit concerning because actually this X1 X3 interaction really didn't fit with what we would understand scientifically.
	So that kind of made us a little bit confused and it was also a little bit like, were our models really explaining the data we'd seen properly.
	It also had a significant real world implications for process development in the sense that this X3 variable was actually known
	to change quite a bit. So the analogy I would use here would be that if we were developing a drug dosing of two drugs, so X1 and X2,
	and we discovered that actually the dosing depends on how old the patient is, say X3, that's something that you really have to dig down and characterize. So exploring this interaction would have actually required significant extra experimentation costs and in time, as well.
	So we wanted to run some validation methodology on that, so we consulted our colleagues at JMP,
	and they essentially said to us, unfortunately, they wasn't really possible to use this holdout validation on the design of experiment data structure.
	And the reason for that being is, if you can imagine, I assigned 70% of a validation set, that sort of training sets to my data here to the design space,
	that we're actually building the model, but actually completely missing part of design space. So it's not really applicable to use this hold out validation on DOE data sets.
	So our colleagues at JMP came to us with this autovalidation.
	And the way it was explained to me was essentially that you would resample your entire data set. So you use your whole data set for training and your whole data set for validation.
	And that might appear at first glance kind of like cheating, it certainly did to me, but I'm told the way in which this is achieved fairly is
	is by the application of these three extra columns, and really by this pared fractionally weighted bootstrap weight.
	And, essentially, I think the the idea here is that if you use a piece of data in the training of your model, you would then wight that so that you didn't use that same piece of data as much in the validation, and that's how you sort of get around this idea of double counting your data.
	So we we used this methodology. We ran this on our models. Here you can see the setup in JMP and we use this again generalized regression platform, which is a JMP Pro tool.
	And here it's pretty much the same as what Jose was explaining, but instead of simulating for different
	training and validation assignments, we actually simulate this partial weighting. So we simulate that a number of times to generate different models and we actually went through this 500 times.
	Again, so the output of our data now like, you can see, with Jose's before, on the right here is a histogram of how many times a particular variable
	was listed in our model, and on the left here, you've got this these parameter estimates from various models.
	I've actually just drawn this red line here and I've called this this the null factor line. So this null factor variable kind of operates sort of like a fake variable.
	So we know this to be a sort of a nonsense variable because, if we look at the null factor on the left here, you can see its parameter estimates are quite widely distributed around zero, so it's not really knowing where to go.
	And so we draw this line and we say that any...the first time that null factor appears, anything below that we would consider quite likely to be a non genuine effect.
	So this is kind of a range of the non genuine terms. And I've highlighted in blue, as I'm sure you can see, this this X1 X3 interaction, which was the one of concern.
	And you can see here that it's it's come up less times and a lot of the null factor effects, so this was sort of the first indication we had that this might not be a genuine effect.
	So we didn't want to stop there. We ran multiple iterations of this with different
	autovalidation setups, and I've just sort of generated three examples here. But you can see, in in every example, highlighted in blue, this X1 X 3 interaction was consistently coming lower than this this red line, this null factor line.
	So, based on that, we were pretty confident that what we were seeing probably wasn't a genuine effect, and so we were able to eliminate that from the models that we desired.
	Now just a comparison of kind of final outputs of what we got. So on the left here was the initial model we built that included this X1 X3 interaction.
	And then on the right, this autovalidated model where we removed that.
	And I suppose, the first thing we noticed was actually this reduction in R squared. So at first it appears like you're probably explaining less of the data, but actually now we're a lot more confident that what we're explaining was actually genuine.
	And you can see here the difference in the minimization optimization conditions that it's output.
	And this actually had a number of kind of benefits for us, actually, in terms of processing. So remember these are all real life variables, they have real things behind them.
	So this X1 variable actually, that's a very expensive thing for our processes. So we were very...it was good to see that we could reduce that that variable.
	Again, also as this X2 variable goes down, that's actually also, for scientific reasons, quite a benefit to our process, so we're able to reduce that as well, which was of benefit to us.
	And these are kind of happy chance coincidences, I guess, but the one thing that we were really pleased with really was that actually,
	you can see the removing this X1 X3 interaction, we were seeing the actually the profiles of X1 and X2 were constant and that actually really fit with
	the scientific understanding that we expected, so we were quite happy to observe this. And obviously we've gone away now, and we've tested both of those models with follow up experiments just to see which one was performing the best.
	And on all occasions, this actually...this autovalidated model has agreed with the observations we've seen in our final final experiments.
	I'll just hand it back now, Rebecca.
Rebecca Clarke	I'm just going to do a quick summary of what we were talking about. So we know that non true effects have real world implications for both the development timelines and experimental costs for bioprocessing.
	And here we looked at two validation methods to help us with our bioprocesses, to highlight a potential non true effects that we can eliminate from our optimization experiments.
	So the hold out validation tool we used successfully to remove a number of parameters from our optimization list, and then we also use the autovalidation during our DOE type experiments.
	And, and in these cases the models that were generated are also then later confirmed by experimental work. So overall, we were very pleased with how we could use JMP Pro,
	particularly, the validation methods within our bioprocesses to help save us a lot of time and resources and not focus on non true effects.
	And just to finalize, we'd like to acknowledge Robert Anderson and Anna Roper at JMP who helped us throughout this presentation, and that is it for us, and thank you for listening.