Biopharmaceuticals' New Frontier: Modeling Patient/Donor Variability for Gene/Ce...

Americas 2021

Biopharmaceuticals' New Frontier: Modeling Patient/Donor Variability for Gene/Cell Therapy in JMP (2021-US-30MP-887)

Andrew Karl, Senior Management Consultant, Adsurgo, LLC

Heath Rushing, Principal Consultant, Adsurgo, LLC

The most novel, innovative and promising therapeutics in biopharmaceuticals are gene and cell therapies. Gene therapies modify a patient’s existing genes, while cell therapies transfer human cells into a patient to treat disease. These cells either come from the patient themselves or a healthy (cell) donor. The first such therapy approved in the United States was Kymriah in 2017. Since that time, there have been countless companies exploring these novel approaches to treating patients. One of the main barriers to regulatory approval is the development of a process that consistently meets the needs of patients (in both efficacy and safety). One obvious challenge is modeling either patient-to-patient or donor-to-donor variability that can (and usually does) account for a large portion of the variability.

This talk focuses on different methods of modeling donor-to-donor variability in JMP to answer important research questions: determining significant process parameters (independent of patients/donors), constructing proven acceptable and normal operating ranges (PAR and NOR), and estimating process capability. After demonstrating the implications of modeling donor as a fixed block effect, the authors then present the advantages to modeling donor as a random effect, first by presenting modeling and simulation features in JMP and then additional features in JMP Pro.

Auto-generated transcript...

Speaker	Transcript
Heath Rushing	All those running Red Sea.
	ribbons.
ROBYN GODFREY	yeah.
	I might be a little obsessed.
	hi how's it going.
Heath Rushing	Good good, for this is ever.
ROBYN GODFREY	good for you, oh.
Heath Rushing	yeah oh yeah.
ROBYN GODFREY	Oh good oh.
	check my checklist here to make sure that I'm doing this right I'm.
	So gonna record Okay, so you know you'd be recorded right.
	I'm.
	your microphone talk a little bit so going to hear you.
Heath Rushing	Hello cast 123.
ROBYN GODFREY	percent.
	return.
ROBYN GODFREY	Everything silence your phone and everything computer.
Heath Rushing	yeah Andrew Andrew is going to join to so.
	that's one thing is like so I'm not sure how it worked with I sure did it did a poster kind of on my own, so this year, what we'll do is is there's a certain point where he'll.
	Take the screen.
ROBYN GODFREY	Okay.
Heath Rushing	So we'll go for me to him, but we're not switching back and forth I'm not.
	not to confuse things you know what I mean.
ROBYN GODFREY	Alright, last time I forgot to turn myself off for I started recording sure they love that.
Heath Rushing	Florida.
	Florida Florida live right on the beach.
ROBYN GODFREY	Nice Nice.
Heath Rushing	yeah yeah.
	lanta Colorado so I'll be back up there in September to go play on the land.
ROBYN GODFREY	Oh Nice.
	yeah.
Heath Rushing	yeah.
ROBYN GODFREY	I was in the cooler nothing here right now sick of humidity and oppressive heat.
Heath Rushing	where you live near the coast of North Carolina right yeah.
ROBYN GODFREY	About four miles inland is the bird flies.
	Okay.
ROBYN GODFREY	yeah I'm really close to the River the Cape fear river and then.
	yeah.
ROBYN GODFREY	So there's the river and then just down the road is the movie called snow sky it's like a cut from the River to the waterway yeah.
	yeah now really close by guys like just a ton of.
	Water but it's not like a beach access or anything like that it's just.
	yeah I'm remember intracoastal waterway, but the oceans, a little further out.
	But I ran at the beach this morning, so that's the nice thing about living here is, I can.
	If I o'clock in the morning there's not much traffic, you can get over to the beach and is pouring rain and we saw a beautiful sunrise.
	So stop raining retina something up.
	that's Nice.
Heath Rushing	yeah for me, I live by the part of the beach where where I live, is is the end so there's 10 miles to the direction that just national seashore.
	Nothing that 10 miles.
	So there's a there's a there's a bike.
	bike lane.
	yeah and so there's a bike path for about three miles and then for seven miles is a bike lane, all the way to the bar.
	Nice.
ROBYN GODFREY	Pennsylvania, which.
Heath Rushing	Will beach beach yeah.
ROBYN GODFREY	We were in Destin.
	Christmas time.
	yeah.
ROBYN GODFREY	yeah like that area in that whole 38.
Heath Rushing	year old nice area.
	Oh yeah.
ROBYN GODFREY	Alright, so we gotta wait for Andrew.
Heath Rushing	Are you good yeah, no, no, no, no we'll definitely wait for him.
ROBYN GODFREY	Okay yeah.
Heath Rushing	Okay, let me.
	send me a message hold on.
	Mr.
	If you just sent me a message.
	I'll do is I'll do the first part with the slide a few slides just introduce someone gene therapy is.
	And then I'll say hey I'm going to play like on the scientists and Andrew I just sent you a data set and I need some help and then him and I'll have a conversation and then he'll take it from there with john so.
	I figured that'd be a good way to do this.
	And really we just figure out that interaction thing like on.
	Monday
	You know, like we had a this where we're going to do, and then I was like, but we need some way of interacting what was trying to figure this out so.
ROBYN GODFREY	need as men in the paper me and rob and my coworker Ted and.
	All the video and it's pretty comical.
Heath Rushing	We always tried to do that.
	I'm supporting some hot water but we always try to make sure to throw something in there, that is.
	That it's entertaining to because.
	Doing playing playing talk for a minute, especially if you're on a screen right.
	yeah would you guys do a talk on.
ROBYN GODFREY	We did it on we did like a scenario of.
	People like in a production environment rocking from like a manufacturing environment.
	yeah so we did like a scenario of you know, a manager who's decided to hire a consultant to come in and create a you know application.
	And then the consultant leaves and they're trying to get you know the seals to you know be properly manufactured and they're failing and the consultants gone and the code isn't working.
	And so it's like what they could have you know if they would have gone the JMP route, and you know we just talked about the four r's reproducibility robustness rapid you know responsive so.
	We just kind of concentrate on we created like three different scenarios and so we're calling it like gambling on your data, so we did this whole like gambling theme.
	walk and, at the end, though, is the best part because we've got like a Kenny Rogers singing and and, like the outtakes of us doing this video and it's it's.
	The outtakes are always the funniest part right and so rob did all these outtakes in like Kenny Rogers the same the gambler at the end, and he has like these credits that are really funny and so it's kind of started at live ish going.
	But a bs it's really it was fun, you know we would use play a different role of like you know, a project manager or an engineer, you know just like dealing with different scenarios and our production, environment and how they could have done it better.
	If they had yeah.
Heath Rushing	yeah yeah.
ROBYN GODFREY	And then rob kind of demos, you know different things that you do use the profiler and that's, all I can say.
Heath Rushing	That we don't we don't like to do a lot of slides.
	But we'll look look who joined us.
	Look at the time to join us.
	hey hey Andrew.
Andrew Karl	hey run.
Heath Rushing	Your early so.
Andrew Karl	yeah I'm four minutes early rope good.
	yeah.
ROBYN GODFREY	So he's your little like your cameras kind of hi.
Heath Rushing	hi my name.
ROBYN GODFREY	yeah like you need to bring it down a little see more of you.
	Doing bro.
Andrew Karl	I don't want that.
Heath Rushing	I don't I don't either I don't either I mean what my face.
ROBYN GODFREY	Like right under your chin and.
Heath Rushing	I always decided to go like like like it yesterday was find yearly dermatology appointments where they you know dirt off things too, so you may see.
	Red spots on that yeah yeah I mean it's it's always like I didn't even think about like hey I have the day off, you know how did they often and really.
	A big part of it was was a just in case that I need to do anything with the presentation and so, then you know you go in the garden some.
ROBYN GODFREY	Big slots for nothing.
Heath Rushing	yeah pretty pretty much.
	yeah yeah.
	yeah.
ROBYN GODFREY	Okay, so I'm going to turn off my camera.
	and seek, I have to.
	lose myself here, here we go.
Heath Rushing	hey so so Andrew at one point out just say hey I'm going to play the scientist I'm going to do, the first part of the slides I'll say.
	hey I'm going to play scientists, so you know you play Andrew the statistician I'm real play the scientists data.
	And then I'll say an Andrew and I'll say Okay, so let me get going, Andrew I just sent you a data set my two responses or somebody ability and byproduct.
	Did you get it, and then you kind of go from there okay so that'd be whenever I'm saying that kind of stuff is you can go ahead and start sharing the screen you see them saying.
Andrew Karl	got it yeah.
ROBYN GODFREY	Do you want Andrew to be on here.
	At the same time you're talking he can he can hide himself.
	Until you're ready for him to show up.
Heath Rushing	yeah we can be on the same time.
ROBYN GODFREY	Okay he's right now it's like dual screens in ruin one side, on the other.
Heath Rushing	yeah yeah I.
	mean like like what what whenever whenever we started out I'm gonna stop you can only see our ugly mugs so I'll share my screen and then so we're going to focus on really not talking like what I'll share three now I'll be doing some slides just to introduce the topic.
	And then it won't take very long introduce the topic and then what we'll do is is I don't know, do you do an introduction or anything, or do you want to introduce yourself, I heard that work.
ROBYN GODFREY	You guys.
	I would do it like I would do a little introduction you know, like you, would, if you were a discovery right.
	Okay, they don't they don't want me to do they don't want me to be in both little okay.
	So you guys got everything hidden on your computer like new email is going to pop up or anything like that.
Heath Rushing	yeah.
	that's wrong I'm not that.
	sounded them.
ROBYN GODFREY	Okay, so I turn off my camera on mute myself and then I'll be I'll let you guys.
	start, and I will interrupt you, if you need to start over just let me.
Heath Rushing	Know you're just used to tell us when you just say when and we'll get going I'll get an introduction Andrew you're doing an introduction I'll say as soon as you're done.
	Is what I'll do is I'll start sharing my screen okay sounds good.
ROBYN GODFREY	sounds good okay so.
	Give me like about five seconds after I tell you to go because.
	My dogs are watching.
	Everybody mute myself and then start the recording so I'm going to me right now and start doing worse, give me like five seconds.
	Okay.
Heath Rushing	All right, good morning. So this morning Andrew Karl and I are going to give a talk on, really, an application in in biopharma, which is really looking at cell and gene therapies. It's a very unique way of making therapeutics.
	A little bit about my background. I'll say a few words, then let Andrew say a few words on on his background. I used to be in the military ended up being a college professor, teaching applied stats and DOE. Then I went to work in biopharma, went to work for a small biotech
	company called Applied Molecular Genetics or Amgen for short, worked in their manufacturing and then worked in their their R&D,
	before taking a job as a as a manager of training that for number of years and then became a consultant for for SAS.
	And and really what we do is we focus a lot on pharma, biopharma in the medical devices and doing a lot of teaching and and a lot of consulting, and get get a lot of unique problems we get to we get to create value for for companies. Andrew, do you want to say a few words?
Andrew Karl	Yep, my name, my name is Andrew Karl and I started working for Heath back in 2012 right after leaving Arizona State with PhD in statistics.
	And I've been doing all kinds of interesting stuff with Heath and Jim Wisnowski there and lately, a lot of pharmaceutical applications,
	everything from formulations development to process characterization, and anything in between there, so I'm excited to be here with Heath today and talk about our applications.
Heath Rushing	Okay, let's get going, and I am immediately going to share my screen.
	and
	I have a few slides, and I do not like to...I do not like to
	focus on slides in a in a JMP presentation, but what I would do is, like, to kind of introduce you to the subject, just the condensed and the very unique application and really, I think, more importantly, the very unique challenge that we're going to talk about today.
	You know it's interesting, we we did this, we probably wrote this abstract back
	at the beginning of April, and then I saw a there was an article written at the innovate ??? very much like this, so
	want to make sure that no one...no one thinks that we copied any article. But really what we're doing is we're talking about you know how to model
	in, you know, we say what a patient/donor's variability is, is how to take into account donor/donor variability in a very unique application in biopharma.
	And we're going to talk about gene and cell therapy, alright. So our first thing we'll do is we'll talk about, you know, why this is a new frontier.
	Right, and we'll just to talk a little bit about what the gene and cell therapies are, and then we're really going to get into the difference between those two.
	And the focus of this is, you know, how do we model that donor-to-donor variability, right? Do we treat as a fixed effect, random effect?
	And what are some very particular application that we may be interested in, right? And first, whenever I said this is the new frontier, it's very innovative, very promising. Currently there is less than, gosh, less than 10
	such kind of therapies, gene and cell therapies, that are on the market. However there's...there's over 1,000 that are currently in the pipeline, alright.
	And what you're going to see is is is with these gene and cell therapies, is there's there's some very unique challenges, okay, it's a very innovative way to to to treat
	diseases, especially rare diseases, and there's some very unique challenges. One, of course, is is how do we...how do we make sure that it works, efficacy? And how do we make sure that it's safe?
	All right, the other is is really, the one that we're going to address, you know, like, so how is it that that we address donor-to-donor variability?
	Like just because I mean patient-to-patient variability, but we want to focus on the donor-to-donor variability, right.
	So let's talk about gene and cell therapies. So what is the...what is...what is a gene therapy? Well,
	as many of you know, you know many diseases such as cancer, you know, are really born out of either you're born with defective genes or some sort of gene has has has mutated over time, right.
	So it's a very unique way to to treat these, and one is it is gene therapy right, and you know what is it that we do in gene therapies? Now, I say that we replace one gene
	with another healthy one, but really what it is, is we're really transferring some sort of genetic material, or we may be adding...adding or modifying or really turning off some gene. This can be done in two ways, so right with patients that are...that have
	that have a certain disease. One we can do is is, we can, you know, we can inject (and I'll use the term deliver)
	those healthy genes, okay, so that that genetic material...that that healthy genetic material into a patient.
	The other thing that we could do is, we could take some sort of blood or bone marrow
	from a patient, we could...we could insert that genetic material and then deliver back into the patient, okay, so it can be done either in vivo or in vitro. Now whenever I say that we are...
	that we are injecting some sort of healthy gene into a patient or into the bone marrow, we need some way of delivery. Okay, we're going to call that the vehicle.
	We often say hey, this is the vehicle to deliver that genetic material to the patients, now we're going to call that a
	vector, okay. We're going to call that a vector and that's what really...it has very unique technology. A lot of times is the vector is is a virus.
	The virus is able to insert that genetic material into cells all right. Now, so what are some, you know, unique challenges? Well, one is, you know, does it work? And in the second is, is it safe, right?
	The very...what we often call the the billion dollar question is, if I inject this genetic material into
	into a patient, are they going to reject it, right? Are they ever going to reject it, or does the body think that something's going on
	and, you know, will really start killing off, you know, these cells, right. The other really unique challenge is is that each patient is a separate lot,
	right. So think about the the manufacturing, think about the development, think about the logistics of that, right. Gene therapies, now, how is that different than cell therapies? So
	there's a lot of overlap, but in terms of cell therapy, what you're doing is is is you are inserting cells, okay, not necessarily genetic material. Now these cells may have modified genetic
	material, but you're you're taking actual cells and you're inserting them into a patient to treat these diseases. So right now, these cells can either come from the patients themselves,
	right, or or what they can do is is is they can come from a from a donor, okay. Think bone marrow transplants, whenever they do something like that, is is is they're taking something out of a healthy donor and inserting it into
	into a patient, right. Now understand that these are, you know, all done in vitro. You're actually injecting these these healthy cells into a patient, right.
	And what are you trying to do, whenever you do that? You're trying to, you know, replace any kind of missing cells or really replace
	diseased ones, alright. So same thing is is, you have you know those same unique challenges as we talked about before.
	The other thing is, is that that another, you know, barrier to both of these is is that it's very small-scale manufacturing and can't be done on a large scale, right.
	Now now you'll see that a little bit later is is, or, you know, like the one of the challenges is how do we do this on a large scale? How is it that we...
	that we can do this, not just in labs, but but on a larger scale.
	Right, so, as I said before, is is is that one of the, really, barriers to this is the efficacy and safety, alright. That's one of the, you know, the the main challenges.
	The other big challenge is this, if you were looking at that...this needs to be done on different patients or different donors. You know, how do you account
	for that donor-to-donor variability? And I was just in a class about an hour ago, and someone was talking and said that exact same thing said,
	you know, we have to account for that donor-to-donor variability in cell therapies and, you know, that accounts for the majority of the variability that we're seeing in our process,
	right. And then so think about that. Well, you know, how do you account for that?
	Very often, what you'll do is you'll say, hey, Heath, I'm doing process characterization studies and maybe we need to do this on one donor, right. So that may be feasible....
	how...that may be feasible, but you're going to see a little bit later is, is how do I account for, whenever I go forward with the commercial manufacturing process,
	that I am going to have donor-to-donor variability and try to make my process more robust to those different donors?
	So the majority of times is, we don't want to do this. Majority of times, it's not feasible and we wouldn't want to do it, even if it was.
	So what we're gonna do is we're gonna have multiple donors. So right now...so the big question is is...that we're going to try to answer today...is is, what are the different ways you could that you could account for that donor-to-donor variability?
	And immediately, what should come to mind up to any of you that know design of experiments is let's go ahead and block on it.
	And then the next question should be what...should it be a fixed block or should it be a random block?
	And what are the effects of those things, right? So let's talk about the different applications that a...that a scientist would need to use this for, okay. And what we're going to do is is
	I'm going to play...I'm gonna play the scientist, right. Andrew is going to play the the statistician that is helping the scientist, okay. So he's going to go by the name Andrew, and I'm going to play the scientist
	that needs help, okay. So I'll go by the name Brady, OK. And then, so you know, what would...what would Brady the scientist that needs help....
	need help with? Well one would be inference and process characterization studies and, also Brady would need to estimate process capability, okay. Now this is very important because
	prior to going into process validation and commercial manufacturing is, we need to understand is how capable our process is. So we'll talk about the traditional way.
	We'll introduce you to something called normal operating ranges and really what you're doing is you're taking into account
	that you're not always going to be at set point, that it could vary around set point, right. Also I'll do something and show you...
	show you another way to assess process or evaluate process capability called proven acceptable ranges. It's if you if you ask the question, what if everything is at its set point,
	but one variable is allowed to vary, right? Is what will your process capability be? And last thing is, we talked about one of the big challenges is large-scale manufacturing. How do I do a comparison of small- and large-scale manufacturing, okay? So
	Andrew, I'm going to, as I said before, I'm going to play the scientist Brady that needs help. And what I would like to for you to do is is to open your email and see that data set that I...that I sent you.
	And can we can we talk about that data set that I...that I sent you, Andrew? Yeah that's it, that's it, yeah that's exactly right. So Andrew, this is what I did. My process is...
	I did an I optimal design and process characterization, and I vary time, temperature, and pH. And I measured cell viability and
	a by product. You can probably see, Andrew, that I have already done this in column properties in the JMP data table, because it's something that you taught me,
	but if you look at the cell viability, it has specifications around it. And the specifications for...there you go...are between 70 and 90, okay. Now
	Andrew, we would usually want to maximize cell viability but we want to show people about doing process capability that is two-sided, okay, that's why I said cell viability will do this two sided, okay, so someone in the audience may say,
	wouldn't you want to maximize cell viability? What we want to do here is is is show them something that's two sided.
	Alright, and we also have a a...yeah, there we go...well to have a ??? which is by product that we want to keep below 35. Now we put by product, but think about
	in the gene and cell therapy world is, this could be something like endooxins. We want to make sure that we minimize the the percent endotoxicity, or maybe some sort of endotoxin, okay, all right. Okay now yeah that's exactly right, that's the...that's the the data set there, Andrew, okay.
Andrew Karl	Thanks, Brady. I also saw that you sent a first...first pass at a model script here.
	And I was taking a look at that and I see you...you built a...you designed and then ran your study for an RSM in those three factors. You've got the time, temp, and pH.
	And what I noticed here is that I really just see a lot of noise. I don't see a lot of association between any of these factors and your response. And in fact, if I turn on the
	ANOVA table,
	I can see that there's nothing...there's not much signal here, so nothing much predictive and even if I tried to reduce the model with backwards selection by kicking out unimportant effects,
	I'm just not seeing anything. So one thing I wanted to ask is when you ran this, if there's anything else that varied along with these three factors that might....
Heath Rushing	No, no, no, I controlled everything in
	my process. Now I do want to tell you that that...
	that we did have to run this on different donors so okay, so I could not...I cannot run all 24 experiments on on one donor, so really what I had to do, I had to run it on different donors. Would that actually matter if I ran on different donors?
Andrew Karl	I see it now. It's a hidden column, let's hide that.
Heath Rushing	Yeah I just it...was...really what I did, Andrew, is that donor is is I just recorded that for our purposes, but I wasn't really sure really how to handle that.
Andrew Karl	Okay, let me take a quick look at this.
	And let's try to include this in the model
	as a...
	as a blocking effect. Wow.
Heath Rushing	So that looks a little different.
Andrew Karl	Yeah, and so all I did here to get this into the model dialogue, I just added donor here under construct model effects, so that adds that as an additive effect here, and now we see the donor is important to cell viability. Now let's take a look at by products.
	And donor's really important to by product as well. And not only is that important, but then, when we include it and then account for that variability,
	now we see that we have some important effects, at least one here for by products is pH. And then we've got a couple of...a few important effects here for
	cell viability. Now do you know, did you include this in your custom design when you built this? Did you include the donor factor there?
Heath Rushing	Now, you know, I did. You know, whenever we first started doing this, I didn't really, you know, understand. We didn't include it in the in the design.
	I don't know if you remember this, but it was a, you know, it's many months ago, is, I think I brought that up before.
	And you actually helped me out with that. So what you did is, yeah, there it is, there it is.
	So that's what we did a long time ago. I just really didn't understand how to do the analysis of this but it looks like
	that it makes sense that we would want to design that up front. I remember you telling me that that. You just said,
	hey, you know, what else can vary? And I said, well, of course, we're going to have donor-to-donor variability and you said, let's go ahead and design that up front. If we're going to end...
	to ensure that we've accounted for that in the design, because the design could be based upon, you know, for...for how many donors you have and how many runs you have per donor.
Andrew Karl	Okay, that's great that you did that, because that's going to prevent
	aliasing or confounding between these these factors that you've got built into the model here, this RSM and these three factors in this donor. So for example,
	you're not going to run all of your high pH with with one of your donors and all of your low pH with another. These factors will be balanced out across that. So that's that's great.
	A couple other questions here. So these donors, going forward with your process, are you only going to use these four individuals as donors or are you going to have other donors coming in here?
Heath Rushing	Well gosh, no. I, you know, it's like...so when we go forward and and we're going to bring this...these, you know, these therapeutics to patients,
	is, man, we're going to have an awful lot of healthy donors, right. I mean...and I'm going to call them healthy donors because of course is we're going to be inserting these cells into patients.
	So you know that's just, you know, that that's just four that that I selected out of our...out of our cell bank and and I think that you told me before to, you know, when
	when we first started doing this, is you said...this is a long time ago, but you said, you know, you need to make sure that if you have, you know, those multiple donors, which we do, we don't have a
	you know, a large number of donors, but you said you know, try to do something where you're randomly selecting the donors that you...that you did. So what we did is is we randomly selected some donors and took four donors.
	But you're correct. Going forward we're not just going to have four donors, and I do not want to draw conclusions on just those four. I want to be able to draw a conclusions about really any and all donors that we could possibly have, Andrew.
Andrew Karl	Okay. Well so one thing we want to consider here is when we just look this, this is what I just showed, you know, this is a...
	this is a reduced model where we...
	we kicked out the insignificant effects with our backwards selection. One other thing we could do, rather than including donor as a fixed effect, is include it as a random effect. And the reason we might want to do that
	is...let me pull up Graph Builder here and show you. So here is cell viability and byproduct, so cell viability is on the left.
	And each color is a different donor, and these are from the experimental results. Now the fixed effects model is going...only estimates
	one variance component and that's the error or the residual variance component, and so that is what you see within each of these individual distributions.
	And then it's assuming that is constant, it's going to be the same, so it's going to pool those together to come up with that estimate.
	And in order to use the fixed effects, are going to use the mean of each of these distributions to say, okay, there's a mean shift in cell viability, there's mean shift in byproduct.
	But what it's not taking into account, is what is the variability between these distributions that we see here. So for example, the green is all the way over here for donor three to the left for cell viability.
	and then this purple all the way to the right. So there's this variability with...across donors that you get. And so, if you're trying to
	build your confidence intervals or you're trying to get your standard errors for your parameter estimates,
	there's information you're missing there if you don't take that into account, that extra source of variability. So that's an advantage of use...treating these as random effects.
	And since you have four donors, that's something that's feasible here. If you only have two donors, it's not something you could do.
	Three is kind of on the boundary of what you can do, because, really, what you're doing is estimate an extra variants component for just the number of...across donors. So in this case now you've got an extra variants component
	with the sample size basically of four, so something feasible here. Let's take a look at what that would look like.
Heath Rushing	Andrew, can I ask a question. Can you bring that up one more time? That looked...I really like....
Andrew Karl	Yeah, yeah.
Heath Rushing	Can I just ask a quick question on this. Whenever you're...whenever you're doing this is, and I did not account for that donor-to-donor variability, is would it...what would it be doing? I mean, would it just be like putting all those together so it would be like one big distribution?
Andrew Karl	Well, so we'll we'll get a better view of this, let me answer this
	as we go along through
	the monitor reports are both, but so
	what it's going to do is for your predictions going forward,
	is your fixed effective model (let me turn on the crosshairs here) so your fixed effects model is going to say, well, these four donors average together to
	80. And what it's going to do is, it's going to use whatever the width of this is. Pretend you can calculate a standard
	deviation for any one of these individually and it's going to have that distribution with that amount of variants that you see only using the within donor variability centered around 80.
	What the random effects model is going to do is say, when you have...when you're predicting extra cell viability outcomes, to say, well,
	not only are we sampling within, do we have this area of variability here, but also we're going to pick out randomly a donor that looks something like the population that's...
	comes from the same population that these four came from. So the mean might not actually be at 80, the mean might be over here down here closer to 75,
	up here closer to 82 and so that extra source of variability is where it's going to build into,
	not necessarily your estimate for your mean, but your your your variants estimates for your confidence intervals and prediction intervals, so those are going to be wider to reflect that extra variability.
	Otherwise you're gonna get...you're gonna get overconfident in your prediction and how wide those intervals are.
Heath Rushing	And I don't like surprise and so that was quite helpful.
Andrew Karl	Yes, so this is a good thing to take a look at then.
	So let me bring these up.
	So here in the left we've taken a look at this already. This is where the donors included is a fixed effect, and on the right-hand side, I'm going to have donor as a
	random effect. Now the only thing I did to to fit this, just as a reminder some model dialog, all I'm doing here is changing this to random effect
	and then running this.
	Okay, so that's what I've got on the right-hand side over here.
	So a few things to highlight. So first of all, you might notice that the donor is no longer listed in the effects summary and you won't see it in the parameter estimates, that's because now is built differently into the model; it's not a fixed effect anymore.
	A couple of...another thing you'll see is that you see a big difference in the actual by predicted. So this actually looks better for the fixed effects model
	than it does for the random effects model on first glance, and then, if we scroll all the way down to the joint profiler,
	we can see that these vertical axes are the same between the two, and we see these much wider confidence intervals
	tor the random effects models, that might be a little bit off putting. But we need to dig in a little bit deeper to see why that's the case.
	And for for an example of that, let's take a look at the prediction expression down here for by products. And by products, looking to the parameter estimates of both of these,
	we see that we have...
	So pH is our only important effect, significant effect in both models, plus the donor in the fixed effect model. In our prediction expression...
	for the random...for the fixed effects model, our prediction expression is conditional on which donor you have. So to come up with a prediction expression, you include the pH information and then
	we also have the information about which donors. We're controlling for the donors here. And you'll see this else=> zero. That's kind of a
	heuristic solution. It's not really built into the fixed effect model, but we just say if it's a new donor we're just going to assume it's zero. There is no effect from donor.
	And that general...generalization to new donors is again not really built into the fixed effect model; it's built in a little bit more to the random effects model, which you can see, in this case is kind of a...
	we got lucky to see that the parameter estimates are the same here between the...for the intercept and slope between the fixed and random effects models. That's not always going to happen.
	Mainly here that's a function of these being close to an orthogonal design from your your custom design.
	But we don't have anything about the plus donor over here.
	Because this on the right from the...from the random effects model is a mixed model is called a marginal model, which means, this gives you your average response over donors,
	whereas in the left-hand side with this fixed effects model, we're getting a conditional model, which is conditioning on each of these
	donors. So we can actually see that, here in the prediction profiler for the mixed model, if we selected conditional predictions,
	now we can see that we've got...
	these are relatively similar now to what we see on the left-hand side, and also what that helps with is we can see the dependency
	on donor now. So this is the...this is the response surface within each donor.
	That's hidden, maybe confusingly a little bit, over here in the prediction profiler for the fixed effects model.
	And the reason it's hidden is because that was a blocking factor and those don't automatically show up in the prediction profiler, but we can turn that on with reset factor grid.
	We can show the donors, and now we see how this response depends on this donors. So now, if we want our inference to generalize just beyond these donors,
	this is where the benefit of the mixed model comes in, because now we're averaging over the donors, instead of having this this dependency on the individual donors. And the last thing I want to show you before I
	turn it back to you, Brady, is the unexplained why does this look so much worse for the
	actual I predicted for the fixed effects model? And that's because this has the benefit of controlling for those donors
	using these prediction expressions here. So it's taken out that noise from that, whereas this gives us the actual I predicted without controlling for the donors. If you happen to have JMP Pro, you can take a look at
	the same model.
	Instead of standard least squares, if you switch over to mixed model, it's going to give you both of these plots
	the standard least squares. But behind the scenes, if you control for those
	random effects...random effects predictions, then we get basically the same thing as the the fit...the fixed effects model. So it's not that the fixed effects model is better, it's just that what it's showing is the conditional model versus the marginal model between those two.
Heath Rushing	Okay.
	So let's see, two things. Number one, thanks for showing me that reset factor grid on the on the fixed block. Don't judge or anything but I'm just...I'm just
	just just a scientist. I'm not a JMP...JMP expert so I didn't even know it was there, so that that's awesome.
	The second thing is is is I really like this approach. I really like using this as a random ??? effect and I think that makes sense, especially for what I really need to do next...is really what I need to do next is, I need to assess process capability.
	And, really, what I want to know is...
	is is with the both of these two, right, we have specifications around them.
	Is is, as I move forward, I need to provide, you know, my my manufacturing facilities and and my my facilities is is what we would expect in terms of process capability. What percent out of spec, CPK, things like that. Can you help me out with that?
Andrew Karl	Yes, and that's that's nice that you put in the specs for the spec limits as column properties here wo we've already got that loaded in. So now we've got that...
	that information, so now going forward, we're going to see as we...as you repeat this process, you've got more donors and, as you were at different factor settings, what percentage of those new observations are going to be outside of those spec limits?
	So we're going to go down to the prediction profiler, and we're going to use the simulator here to help us out. And
	something I wanted to mention...I should have mentioned to you earlier, so these these extra...the extra width in these confidence intervals, again, that comes from the fact that it's smaller here because it's controlling for the donor.
	But where's this extra variability coming from?
	And it comes from up here at this REML table, so now we get two variance components, I mentioned that earlier, so the this...we get the error variance, which is called the residual variance here.
	And we also get the variance, the donor-to-donor variability. And in this case for by products, you've got 93% of your variability is donor-to-donor. Does that sound?
Heath Rushing	Yeah, yeah. And really, you know, as I was saying before...before that, you know, I had this conversation,
	that I just said, whenever you asked me, you know, are you going to be using anything different that we need to block and I said, I think that we'll...you'll see a lot of donor-to-donor variability, so that doesn't surprise me at all.
Andrew Karl	Okay, so one thing I'm going to point out here is that it's a little bit hidden back in the fixed effects model, but if you want to see your error variability here, you can look at the ANOVA table.
	And if you look at the mean squared error, so that's your error variability, your residual variability, so .8477 for the by product.
	And if we look over here for by products in the mixed model, we see this .847, okay. And they're not that...so the reason that those are close because that's the residual variability in both models
	and estimating basically the same quantity in both models. But the...this model, the mixed model, gives us the extra variance component here for donor and that's going to help us for this process character...for this
	capability analysis that you're asking about. So let's go down and take a look at that. So how could...what would we do in the
	fixed effects model?
	Well, again, we know that the...you're going to be getting different donors, so your values can be shifting. So if we turn on the simulator, really, your only option here
	and it's not a terrible approximation if you have more than one...more than a few donors, but let's take a look at this. So we'll turn on the add random noise and here we're going to see this
	These values are from the square root of your MSE, so this is your variance, and so the square root of this .8477 is going to be your standard deviation for your error variability.
	And so we can also see that over in the REML table, it's not on by default, but if you right click and turn on column square root variance component, now we can see that here.
Heath Rushing	Another trick. Thank you.
Andrew Karl	Yeah.
	So .92, and so that's the same...that's what this .92 is coming from, so including the error variability that it detected in your model already.
	And if we just simulate now though,
	now we get this narrow range, but this is a simulated for Donor 1, Donor 2, and you can see how the means shifted up there so the mean shifts up
	for Donor 4. If I shift to Donor 3, which is smaller than over here, I'm going to shift down.
	So one thing you can do to approximate there, is you can set the donor to random and that's kind of a bootstrap over the available donors you have. And you can see now our standard deviation is getting up to this 2 to 3,
	which looking over here to the mixed model, where we have donors random effects, so we have total variability which is not only the residual variability, but the
	variability including the donors as well. That's about 3.5. So if you do that, though, you get this artificially bumpy simulation distribution, and that's an artifact of only having four donors here. So
	not the best of approaches, but that's the...kind of the best you do with the fixed effects model.
	Also one other thing I'll point out without going into too much detail, is your donor estimates here
	in the fixed effects model are pretty similar to these random effect predictions in the mixed model. The difference is that these over in the mixed model are going to be shrunk appropriately, based on this variance ratio. It's called a
	shrinking property of these estimates and it gives you a little bit better predictions, a little bit better estimates of your donor variability...
	of your donor effects. So, I won't go into that too much more, but I just want to point out that they're pretty similar, but there is a small difference there, in case you need to look at that yourself.
	But going over here now, how can we do this in the model with the random donor effects, is I can turn on the simulator.
Heath Rushing	Okay.
Andrew Karl	And one note of caution before I show you the right way to do it, is I've noticed, if you turn on the conditional predictions here, and you try to add the noise,
	And the noise now is set to those...it's pretty similar to over here what we saw in the fixed model, but that's only right now, including the residual variability right now.
	But if I try to do this with this conditional profiler....
	So I find that...change to Donor 3, notice if I changed the donors, this is not shifting up and down so don't...don't try that same effect there. There's something going on here with the conditional predictions and the simulator here. So
	don't simulate for conditional predictions. We want to simulate from the...
	with that option turned off, as it is by default.
	But now you can see that there's this kind of narrow distribution here, relative to this wider confidence interval.
	And the reason is this default standard deviation is not the correct one we want to use. We want to use...go up here to the REML table...and we don't want just the variability within donors, we want the total variability.
	And so the total variability standard deviation here is 3.53.
Heath Rushing	Okay.
Andrew Karl	For by product, so we want to change this using that REML table.
	And I didn't hold control and broadcast earlier so I'm going to add this
	square root variance component again; it's up here, 3.16.
	And now, when I do that and simulate, you can see
	I get this wider distribution,
	yeah, which is going to
	better reflect what you're going to see going forward with additional donors from the same population and give you a better estimate of your...of your defect rate.
	And you can compare this now to what you see over here, this bumpy distribution, this artificially bumpy distribution
	in the fixed model. Now if you have 1,000 donors, which obviously you're not going to have the budget for, but if you did, these distributions would approximate each other, and you can see the standard deviations are not too terribly far apart. They're both getting closer to that 3.1 and 3.5.
Heath Rushing	Yeah, I think...I think the random effects model is what we'd actually expect.
	Yeah that's about what we expect, especially if we had a large number of donors. Okay, good.
Andrew Karl	And I've noticed now you've got these...these are all fixed. Is that how you'd like to do it for your capability analysis, because you
	see, you've only got the variability from donors and then error variability, and then you have these fixed?
Heath Rushing	You know, that that's that's traditionally how that we do...that we've done it, however, that there's there's really two things that I want to address.
	One is called normal operating ranges, and let's see if you have an example for that. But really what the normal operating ranges is this...
	is that if you look at something like time, okay? Time is something that we could fix. Say we wanted to fix it at 5.5. If we put in a 5.5 and when it goes, it goes from 5.5 hours, and then it stops.
	But, however, temperature and pH are not like that, like if we set temperature at, you know,
	some 27.45 or at some set point, but what it's going to do is it's going to have, you know, some natural variability around that from like a ??? capability,
	or just, you know, just it's biopharma. So so, you know, like, temperature is not going to be fixed at that, so can we account for that in terms of our process capability?
	Normal operating ranges
	???
Andrew Karl	Okay, do you have an idea of how much they vary historically? What what might be a standard? Because what we can do in JMP is we can sit this for fixed random.
	And we can say okay it's got a mean of
	27.45.
	And if you have any idea, and we could do that for pH too, and if you know what those...what reasonable, kind of, standard deviations are, how much these vary.
Heath Rushing	Yeah yeah I do, you know. I've done some data analysis on some some things that we've done in the past, and, you know, I would say, like if you looked at something like that, it looks like a distribution, I'd say, 95% of the time.
	Let's see, it would have to, gosh, be in between, you know, plus or minus one, so I'd say maybe the standard deviation would be .5.
	for temp. Okay, and for pH...for pH, it would be between if I set that 6.54, if I send a 6.5 is maybe similar, I don't know like say say...
	you know,
	the standard deviation there would be .1, okay.
	And I think our set points...
	I think our set points for, let's see, what was it for temperature? Was that was that was for temperature 27.45 and 6.54?
	6.5, I think it's 6.5.
Heath Rusing	Let's see...
Heath Rushing	20...yes 6.5. It'd be 6.5 and for temperature, it'd be 27.5. That's what it would be. There we go. Yeah, yeah okay yeah. That's that's exactly what I'm talking about, Andrew. It's called normal operating ranges and I want to account for that, in terms of estimating process capability.
Andrew Karl	OK, so now, now that we've enabled that, those settings, and we have this total variability for donor, plus the error variability,
	now, when you...when we simulate, we get the extra variability now
	that's introduced by the variability in its factors, and you can see that reflected in the defect rate.
	And also,
Heath Rushing	I just want to point out here too,
Andrew Karl	you can also, since this isn't taking that long to run, you can increase your number of runs
	to get a better estimate. You can also see how you get a smoother distribution is from simulation here.
	And then also you can simulate these values to a table, where now you get, for each simulation, you get a different row, and you can see the predicted cell viability and by products at that simulator setting, and you can do additional analysis with this,
Heath Rushing	Yeah in that case you would get something like a
	CPK, right? I mean if you did it that way, you get some sort of estimate CPK that way?
Andrew Karl	Yes, yeah you can get that together.
Heath Rushing	Okay, awesome. Okay, so that that was the first thing is, is how to how to account for normal operating ranges, but there's one more thing ??? process capability that I want to address. You know, it's not necessary
	however, is is, we like to look at what's called proven acceptable range. And this is what proven acceptable range is.
	It says, if I said temperature and pH at our set points and did not allow them to vary,
	but I allow time to vary between it's low and high settings, and it would be like equally possible that it would be this low versus this
	high setting, what would I expect, what would my process capability be? And really what we're doing is we're trying to prove that if two of these are set point and one of them varies, then it's within you know acceptable process capability. So how would we address that?
Andrew Karl	Okay, so you said, time and...temp and pH are at their set point.
Heath Rushing	That's correct.
Andrew Karl	And then time, though, so it can vary across it.
	So what we'll do, instead of saying normal, what we'll do here is set this to uniform.
	And these are the low and high settings from the experiment now, 1.9 to 9.1.
	And you can change that as needed, but is that...
	Did I hear you right? Is what...
Heath Rushing	That was that...that that's exactly what it...
	that's that's exactly what it is. So if I...if I did something like that is...and really, really...
	in this case, like, can we even change those? Can we even say that's we said in the experiment, but I want to see if our proven acceptable ranges, saying between two and nine, I mean, could I...could I change that in any way?
	Yeah you can. Okay I don't even know that.
	Yes, so you can change those and even simulate what it would be and that's called proven acceptable ranges, okay,
	where we're fixing those two. So what we would do, I appreciate you showing me this, but, boy, and I'm going to try this on my own, what I would do is I would do that for each one of these, right, I would do that for time and I'd do that for temperature and I would do
	that for pH. And just to ensure that we have acceptable process capability.
Andrew Karl	Okay, nice thing is, that's really easy to do here so.
Heath Rushing	Can you show me just one more thing, okay? Just, yep, go ahead and do that one right there, just between 25 and 30, the uniform between 25 and 30. 25 and 30. Okay, can you simulate that to a table?
	Make table and just look at...I don't...just look at the distribution of cell viability and by product.
	Oh that's right. It does give you that process capability. I love that.
Andrew Karl	Because you get you put in the work you save
	that as a column property for those spec limits.
Heath Rushing	That's...that's so easy. Okay okay well, I appreciate that. I think the only thing that I was...that I was wanting to ask you about is...I also sent you another data set
	where we're looking at,
	you know, doing a comparison of small and large scale in comparability, right. And then really what we're trying to do with comparability
	is, I have some runs on small scale, I have some, you know, some runs on large scale, and for my CQAs(?), I want to show that they're not exactly the same, but they're within, you know, some sort of...
	some sort of limits, alright, some sort of practical significance, and it's called equivalent to comparability. I know that you know what that is, but how would I, you know, how can I account for, you know, donor-to-donor variability there?
Andrew Karl	Okay yeah and I saw that. And I saw that also so...
	First of all, just like we looked at the previous data set, let's see what happens if you don't include that donor information.
	Okay, so and you mentioned that the for the cell viability, you're using an EAC equivalent, a criterion of 2.
	And what we see here
	is that the the max P-value is...we do not reject the null hypothesis, which, in this case, means we do not detect equivalence, which
	doesn't doesn't match up with our expectation here that you think these are actually equivalent processes.
	So what's happening is this...that information is getting lost in the noise of the donors.
	And like....
Heath Rushing	Whenever I see that...just just scroll up just a
	second, just a second...I see that. I see that. I see that point that shows me on the graph that shows me the difference of
	negative .49
	on the graph and I go, well, that that's that's that's not too big, but it just looks really, really wide.
	Okay.
Andrew Karl	That's because the extra variability going into that estimate for the the width of the confidence intervals.
	because remember blocking takes out...takes variability away from your estimate.
	account...
Heath Rushing	Okay.
Andrew Karl	That's that's exactly what's going on here, so in both these cases, you're not detecting equivalance between your large and small scale,
	between your...for either cell viability or byproduct. And you can see there's a lot to this report here. It's kind of a lot of work to set this up and I'll show that after I show you one more thing here. All I want to show now is that now I included the donor effects as a as a random effects,
	and now you can see
	with a small P value we reject, which means we actually are now concluding that...concluding equivalence, given the EAC of 2 for cell viability and also for now for the max P values, one we're going to look at where
Heath Rushing	The three.
Andrew Karl	For the EAC ofthree for
	for by product. And the way we set this up is, let me show you this because I know you gotta go you gotta jump to another meeting another minute or so.
	We've got scale and then I'll just do cell viability for now. That's how we are going to set up the test without the donor effect, but if we add donors, now we can save them as a random effect.
	Run the model.
	And we go to estimates, multiple comparisons.
	All pairwise. Students T. OK, and then from here,
	equivalence test.
	Yep, use our EAC of 2.
Heath Rushing	Yeah.
	And then even shows from those confidence interval to such as those two P value.
	also shows from the confidence interval.
	Good buddy of mine, Chris ? in his PhD studied intervals so him and I really love it...
	love intervals. He always talks about it when we're skiing, but that's what it says that lower and upper 95% confidence interval, that'd be comparing it to that too.
	Yeah that's all.
Andrew Karl	Mmmhmm and.
	What my last my last point for us, you might say, okay well earlier, we talked about the benefit of random effects versus fixed effects for the donor in the not...in the
	not comparability not equivalence testing, just the testing for the parameters. I did put together a power analysis using JMP Pro, because in JMP Pro if you build a mixed model,
	under the mixed personality, there's this nice feature called save, columns save simulation formula. Now this...
	and if you go into here, it makes it very easy to simulate mixed models. You can go through and change the settings.
	And so I did that, and so I ran a power analysis to see which is better after simulated under a mixed model to either use...treat the donors as fixed or random effects and, at least in the case that I ran, I was getting higher power which you can see in the boxes here. I had a power of
	.62 for my random donor effects under the same test versus .52 for the...
	for the fixed effects. And both of them were maintaining the nominal error rates for that, so just something I started to take a look into, but it does look like there might be a benefit to also treating the donor effects as random within the equivalence testing as well.
Heath Rushing	Yeah, I think, Andrew, I think you've convinced me most definitely that as we move forward,
	you know, we're always, being that we're in cell therapies, that we're always going to have that donor-to-donor effect, I think. Most definitely this convinced me that it makes sense for us to look at it as a random block effect.
	So everyone we we hope that you enjoyed the the application of looking at both process capability and and and comparability, two things that are very important
	for for our scientists and engineers that are in the pharma and biopharma. I think that the very specific application we talked about is very
	innovative way that we're developing therapeutics now called a gene and cell therapies, and I hope that that we
	that we provided some value by showing you, like, how that can be handled both inference process capability and comparibility. And we look forward to seeing you at the conference, for us to present this again and answer any questions that you may have.
Andrew Karl	Thanks, Brady, appreciate it. I didn't get your last name, Brady. What was it was that.
	Brady Brady.
	Brady.
Heath Rushing	Okay, my first name is Brady, my last name is Brady okay it's.
Andrew Karl	Okay.
Heath Rushing	Great yeah yeah yeah to you yeah yeah yeah.
Andrew Karl	Thanks, Andrew. Appreciate it. Yep, thanks, everyone, for joining us.

Presenter

Andrew Karl

Biopharmaceuticals' New Frontier: Modeling Patient/Donor Variability for Gene/Cell Therapy in JMP (2021-US-30MP-887)

Presenter

Files