Although Andrew Karl, and Jeff Hofer , and Rick Burdick, some teammates of mine ,
did the majority of technical work here, I'm going to be the one presenting today .
I'm going to talk about how JMP and JMP scripts can be used
in a very particular specific application in cell therapies.
I'm going to talk a little bit about what gene and cell therapies are
and the very specific instance that I want to talk about
is comparability.
I'm going to focus on process changes.
Interestingly enough, last year,
I gave a talk , and it focused on cell and gene therapies.
They're very novel therapeutics.
The first one was approved in the United States in 2017.
A little bit different than most of ,
what I call the small molecule and the large molecule therapeutics
that you may have heard of in the past.
Let me just touch base on what is a cell and gene therapy.
First thing I'm going to do is touch base on what a gene therapy is.
What you're essentially doing is you're replacing
a gene with a healthy one,
or turning off bad genes.
A lot of cancers are caused by defective genes.
What you're doing is you're inserting these healthy genes back into a patient
of either in vivo or in vitro.
An in vitro would be more of a bone marrow transplant.
Last year, I talked about ,
the challenge with gene therapies is that patient -to -patient variability.
I focused on process development.
Then I talked about cell therapies.
In the cell therapies , what you're doing is you're replacing disease cells.
You're either transferring some sort of healthy cell into a patient,
or we're replacing missing cells into a patient.
Where do these cells come from?
They either come from the patient themselves,
so you would have to deal with that patient -to -patient variability,
or in most cases, they come from a healthy donor.
Now you're not dealing with this patient-to-patient variability,
but you're dealing with donor -to -donor variability.
Whenever I say donor, I'm talking about a healthy donor.
I could be a healthy donor.
Then someone else could be a healthy donor also.
In both of those cases
is you have to deal with that patient-to-patient
or donor -to -donor variability.
What's interesting is last year , I gave the example in process development,
and it looks something like this.
It was the exact same data set that I used last year that I said,
say that you were developing a process where you look time, temperature , and pH,
and you're measuring their effect on cell viability and by product.
In that case is, I cannot use one donor material,
I had to split that up into four different donors.
I said, "If you ran these experiments for process development,
and you did not consider that there was donor -to -donor variability,
this is what you would see.
What you would say is we're looking for P-values that are below 0.05,
you would say nothing affects cell viability
and nothing affects by product.
You were not able to detect that you had any significant
or critical process parameters
for the very reason that you do not consider
that there could be a difference in donor.
Right now, if you do consider those as what it's called a fixed donor effect,
the only thing that I did is I brought in donor.
Then you see that NAND.
This really sticks out what significantly affects cell viability
and what significantly affects by product.
The whole talk was on how does that donor-to-donor variability
affect statistical inference and also process capability.
I'm going to focus on that statistical inference.
What are you trying to do in process development
is you're trying to determine if things like pH, and temperature, and time
significantly affect your critical equality attributes.
Say that I was a drug manufacturer ,
and I have set up a process development study .
I send this process development study .
I want to determine if temperature affects,
and I'm going to call it cell viability.
I say, "Hey whenever I'm looking at that , is I want to make sure that if something
significantly affects my quality attributes,
I control that in my process.
But if it doesn't, I am not spending money and time and resources controlling it."
What I'm concerned with as a drug manufacturer
is the Type I error rate.
I do not want to inflate a Type I error rate.
A Type I error rate would say,
"Hey , this is significant when, in fact, it's not."
What do you think that regulatory agencies would be more concerned with?
You controlling more things?
Are you not controlling things that should be controlled?
That is exactly right is they'd be more concerned
about that patient risk, that Type II error.
In process development is drug manufacturers
do not want to inflate the Type I error.
They also want significant power. Why?
Because that controls that patient risk.
The whole point of me showing that last year
was to show the effective donor-to-donor variability
on trying to determine your critical process parameters.
I call it statistical inference.
Right now, what happens if I change my process?
I had a colleague just last week , I was working with her.
Whenever we're talking about cell and gene therapies, she said,
and this is her quote, "Heath , at cell and gene therapies,
things are constantly changing.
You could have things like analytical methods change.
You could have things like process change."
Today, I'm going to focu s on this process right here.
Mainly, I'm going to focus on that process change.
I do want to point out that regulatory agencies understand
that you have a need for improving your process.
Even if you improve your process, are you changing your process ?
They recognize the need for that, but they also recognize the need
that the therapeutics that you're making from that process
should be similar in terms of product quality.
You're using these in clinical trials.
What does it mean to be similar?
That doesn't say that they have to be exactly the same,
that they have to be similar or comparable.
In terms of me saying that something is similar,
what I want to do is I want to make sure that I have some similarity condition.
That's the whole point of comparability.
For very low risk attributes,
what I can do is I can show that process A and process B
is similar in side -by -side plots .
For more higher risk attributes,
what I want to do is maybe something like a quality range.
In terms for quality ranges,
I just take that reference group, the old process,
and I built some range around it
and ensure that all of th e measured quality attributes
from the new process fall within that range.
For very high risk attributes,
what I want to do is I want to do equivalen ce testing.
This is what I'm going to focus on today,
tell you about what equivalence testing is,
and how that acceptable difference or that similarity condition is set.
It's called equivalence testing, Two One Sided t-test.
To reiterate what we talked about before,
whenever I'm using design of experiments in process development,
what I do is I'm changing some variable -like temperature
from low to high ,
and I'm measuring the effect on my critical equality attributes.
I am assuming in the null hypothesis that they are the same.
What I do is I set up a design to see if they're different.
A Type I error in that case would be me saying,
"Wow, they're different " when , in fact, they're not.
That would mean that I would control that.
I would spend resources controlling that in the process.
If I'm a drug manufacturer,
I do not want to control things that I don't need to.
I'm concerned about that Type I error rate .
If I was a regulatory agency,
I would be even more concerned with the Type II.
There's no difference when, in fact, there is .
You should be controlling something and you're not.
If I was a regulatory agency,
I'd be more concerned with the Type II error.
Now, we're going to flip it.
We're going to talk about equivalence testing.
Equivalence testing is I'm not saying that they are the same.
I am assuming that there is a difference.
I just want to make sure that the difference isn't too big.
That too big, I'm going to call delta.
T here's a lot of different ways to calculate that delta.
I'm going to call it d or that delta right there,
often called the equivalence acceptance criteria.
I would like to come from subject matter expertise,
but the majority of times, it comes from me taking
some k-value times that historical value.
That's split into two different tests .
On one, I'm determining if it's less than positive d.
In the other one , I want to show in the alternate hypothesis
that that difference is greater than a negative d.
I'm going two different sides.
That's what's called the left -hand side of the bottom,
or the top.
In terms of, if I was a drug manufacturer, what would I want to do?
I would want to be able to reject both of those hypotheses.
I would want high power, low Type II error and high power.
T his is equivalent to taking a 90% confidence number
around the difference in means in ensuring that 90% confidence interval
whenever I'm looking at the low and high or within the balance of those lower delta
in the upper delta.
If you're looking at this, you should think to yourself is,
"I want the width of that confidence interval to be very small."
What are the different ways that I could make the width
of that confidence interval
for the difference between those two means very small?
I could decrease my standard deviation.
That's a good thing.
I could increase my sample size.
That's a good thing.
I could also increase my alpha level.
Maybe that wouldn't be so good because what you're doing
is you're inflating your Type I error rate.
In inflating your Type I error rate, what you're saying is,
I am stating that they're equivalent when indeed they're not.
The different ways to control the width of that confidence interval
is to lower s, increase in, or increase alpha.
We talked about two of those being good and one of those not being good.
It makes sense that if I'm a drug manufacturer,
I want to maximize the power of the design.
That's the flip. I want to minimize my Type II error.
Regulatory agencies want to make sure that you do not inflate
that Type I error rate.
That Type I error rate would be saying your assume equivalent
or you're stating equivalence when indeed they're not.
In JMP, you can do these equivalence tests,
and I want to show you an example of that.
For my journal, first thing I want to do
is I want to show you that in terms of determining
your Type I error rates and your Type II error rates
is JMP provides power curves
under Sample Size Explorer , Power , Two Sample Independent Equivalence.
Caleb King did an awful great job with this.
I say awful great job, but he did a great job with this.
Let's just say that my margin, my equivalence acceptance criteria,
is plus or minus 2 standard deviations.
I'm just going to put a 2 here,
and that's just 2 times the standard deviations that I'm talking about.
That's all that I'm doing.
Let's just say that in my historical process
is I have 10 lots,
and I'm going to compare it to a new process that has 5 lots.
I want to see what the power is if they are exactly the same,
but there's no difference between these.
A few things that I want to point out here is JMP gives those power calculations.
The other thing that it does is it allows you change those.
What's going to happen if I do things like increase my new process,
the number of samples in my new process day report ,
my power is going to go up.
What would happen if I do things like,
"Hey, Heath, I want to decrease that margin of error
to, instead 2 standard deviations , to say maybe 1.5 standard deviations,
essentially , as I'm taking those boundaries
and I'm tightening them up."
What I see is my power is going to go down.
I'm able to ask myself all those typical questions
that you would in equivalence testing.
This is something else that I want to show you that's going to come up
is JMP has the ability to say,
do I know the true standard deviation or not?
If I know the true standard deviation, that is going to be better.
You're going to see that your power goes up.
Indeed, what happens is my power goes up.
That's usually not the case.
I always call that the utopia,
which uses the cases if I do not know what that true standard deviation is.
I always call this the optimum, a car of the utopia.
I always call the no, the realism.
I would be remiss
if I did not show you the tools that JMP does have
for showing that equivalence
like if I had an historical process where I had 10 lots and I made 5 new ones.
First thing I want to do
is I want to look at this through Graph Builder,
and I see that there is no effect between those two.
I can see both of those , and they both look like
they came from the new process, the blue versus the red.
How about if there is an effect ? What I do is I see a shift.
Just like I showed you before is that is Two One Sided t-test.
JMP has tools for that.
Jin Feng did a great job with this. My goodness. I love the scores plot.
Here's the difference in means. Here's the lower , and here's the upper,
and that's within the boundaries.
In that case, what you've done is you rejected both the null hypothesis
in favor of the alternate,
which is the same as what you see in the picture.
What you also see here is that if there is an effect,
is I am not going to reject both the nulls.
One of those is I am going to fail to reject and indeed I did.
What you'll see is my confidence intervals outside that boundary.
I would like to talk about a very specific case.
A very specific case in cell therapy is called split apheresis design.
In a split apheresis design , this is a situation where
in cell therapies is you're changing the process.
What you do is you're using donor materials split
between the two different processes.
We kept getting questions over and over and over again
from our customers about,
"Can I look at the sample size and power calculations
for these pair of designs ?"
Cannot overlay them .
You cannot see if they're dependent upon that donor -to-donor variability?
Let's talk about a split apheresis design.
In a split apheresis design, first thing I want to do
is I want to tell you about the regulatory expectation.
This is even a recent draft guidance document from the FDA
in July of 2023, just last month.
In that , they said that you need to select a suitable statistical test
for analysis in difference between paired data
where those donors are paired up.
That's where the split apheresis design comes from.
For every single donor material that you have,
you split it in between process A and process B.
This is not two independent t-tests.
What this is, is a paired design.
That's the first thing that I wanted to talk about.
The second thing is,
I wanted to talk about that you are very often in early stage,
so you do not have a line of donor materials,
so you have very low sample sizes.
It's hard to get power out of low sample sizes.
The third thing that I'm going to tell you is ,
how do you come up with your EAC?
How do you come up with your similarity condition ,
that difference, that acceptable difference?
What you do is you use historical data that is made off of multiple donors.
You take the standard deviation used off of historical data.
I'm going to call that n 1 or historical.
You take some k number of standard deviations of historical data .
You do a test , and you're using the split apheresis design
to judge off of that historical data.
These are two examples that I want to show you.
The first example here
is where you're looking at process A and process B.
What you see is you do see six different donors here.
What you see in the one on the left is the majority variation is coming
from donor-to-donor variability,
not the difference between process A and process B.
You have high donor -to -donor variability.
I'm going to call that, prho.
In the case on the right, what you do is,
is the majority variation is coming from the difference
between process A and process B, not the donor-to-donor variability.
The majority variation is coming from the analytical or the process.
What that tells you is you have very low rho.
You'd have low donor-to-donor variability.
I'm going to show you a series of scripts that we worked on.
These are typical questions that came from our customers.
In our cases, we do not know what the standard deviation.
How does that compare to the known?
How about those Type I and Type II error rates?
Remember, if I'm a drug manufacturer, I want to increase the power.
If I'm a regulatory agency,
I want to make sure that you do not inflate that Type I error rate.
How are we going to do this?
This is from the European Medicines Agency, 2001.
The best way to do that is with things
called expected operating characteristic curves.
That gives you power on the y-axis and a shift in the main.
I 'm going to go through a series of scripts,
and these series of scripts ...
It's really one script that have right here,
that it's going to allow me to change things like that rho,
that proportion of donor-to-donor variability.
That k -value, remember, how do I set the acceptance criteria?
It is k times that standard deviation.
The typical way of doing this is that k times those historical lots.
This is the number of historical lots that you use n1.
n2 is the number of lots that I'm going to use for that paired design.
Whenever you run the script, what happens is you come out ,
and it does a series of simulations.
In this case, it did 5,000 simulations ,
and it calculates the power for you.
In those 5,000 runs, what percentage of those passed?
It looks something like this. It gives you a lot of different options.
My goodness . I can look at different k -values.
I can look at a different number of n1, which are called historical lots.
I can also look at the different number of n 2 or paired lots.
Right now, I want to talk about...
Whenever I do this, what I can do
is I can select which of these different cases that I want to look at
to be able to answer typical questions.
Let me open up my typical comparisons here.
The first one I want to talk about is,
"Heath, what if I have a known standard deviation?"
Look s something like this.
That's what the known standard deviation looks like.
A few things that I want to point out
is this is the percentage of time that you're going to claim equivalence.
If they're exactly the same that you said you're going to claim equivalence
a high percentage of time.
If there's a huge difference between them like a two standard deviation shift
or a three standard deviation shift , is you're not going to claim equivalence.
That's a good thing.
The other thing that I want to show you here
is if you're looking at this alpha of 0.05,
being that I set my k -value at 2 ,
k number of standard deviations versus 10 historical lots,
the standard deviation of 10 historical lots,
you would expect that alpha level would be 0.05,
the exact alpha level that I set in my equivalence test.
Right now, the thing that I want to show you
is this is for a proportion of donor -to -donor variability of 90%.
What happens if I change that?
What happens if I change that to 60%.
What happens if I change that to 30% ?
There's no donor-to-donor variability.
What you see is that paired test, the power curve looks really good
whenever I have high donor -to -donor variability.
The other thing that you notice with the known standard deviation
is the alpha level regardless of operating characteristic curve
is always at 0.05.
Let's talk about some other typical questions.
One typical question is ,
how does it compare for the different levels of rho?
How does my typical way of doing this ?
I do not know what the standard deviation is.
My typical way of doing this is in the blue.
The known standard deviation is in the red.
One thing that I want to point out
is I want to point out this one right here.
What you see is the preferred approach,
the approach that even regulatory documents have said that you should do,
the paired approach,
using the standard deviation
that is calculated off of my historical lots,
is I have an inflated Type I error rate.
This should be 0.05 just like it is here.
That was really strange to us, and we looked into this.
When we looked into it, what we found is ,
it has everything to do with this right here.
The reason why it has everything to do with this right here,
as I said, how do I decrease the width of that confidence interval?
The way that I decrease the width of that confidence interval
was either to decrease s , or increase n , or increase my alpha level.
Understand this.
This is why you have an inflated Type I error rate
with this paired test
is those deltas , which you're using to judge this off of, those deltas
are using the standard deviation off of historical data
that contains donor-to-donor variability.
That confidence interval right there
does not contain donor-to-donor variability.
Why? Because you did a pair test .
That contains only analytical and process variability.
That's where that inflated Type I error rate comes from.
Using this paired approac h is understand
you have an inflated Type I error rate.
We see that, and it's even more prevalent
when you have high donor -to -donor variability.
Why? Because if you have low donor-to-donor variability,
th at process variability
is the largest part of the variance component that you have.
Let's look at a few more questions that you have.
A s I said, this one script answers these different questions.
This is answering the question ,
"Hey, Heath, if I use that paired approach that's recommended,
can I look at what happens as I increase sample size
from 3 to 4 to 5 to 6?"
Two things that I want to point out here
is number one , what you see as I increase sample size,
is I'm going to have higher power.
I still do not have adequate power if there's no donor-to-donor variability.
That means that I have 0 donor-to-donor variability.
I would need at least a sample size of 8 or 8 different donors.
If I do have high donor-to-donor variability,
like 0.9, 90 % of that variability, which you see is I do have high power
for no difference between the means.
What I can do is I can make sure to answer those questions
with overlaid operating character s occurs for different sample size.
I can also answer that question if I was looking at , and I say, "Hey ,
I've stated my different sample sizes,
but what if we look at the different k -values?"
Understand that your acceptance criteria is k number of standard deviations.
What's going to happen is that acceptance criteria
are those what I call go post are going to widen as you increase k.
Therefore, you're going to have a much higher ability
to pass equivalence ,
and you're going to have much higher power.
Another typical question is this.
What if I want to change both of those together?
I'm a big fan of Graph Builder .
What Graph Builder is what you're looking at here
is not only are you looking at,
"Hey , Heath, I am increasing sample size in blue, that would be 3,
in red, that would be 4,
in green, that would be a 5, and in purple, that would be 6 ,
but I also looked at it for different k -values.
What would your operating characteristic curves look like?"
Good?
I want to revisit this.
Just like I said before , I said, "Hey , I want to revisit this
and show you that for..."
Whenever I have a large proportion of donor -to -donor variability ,
I said, "What you see for 2 right here, I would expect my alpha level
that my proportion of time that I pass this test would be 0.05."
But what you see is you have inflated Type I error rate.
How does this look?
Whenever I'm looking at a rho
or a proportion of donor-to-donor variability
that is very small,
I do not have much power.
The question was , what if we did this instead?
If we had low donor-to-donor variability,
if what we did is we used information from those historical lots.
If I have no donor-to-donor variability or very low donor-to-donor variability,
why couldn't I just do a independent t -test,
where I compare from process A or my historical process,
not just the paired lots,
but I also consider those 10 historical lots
and not comparing to the mean of the new process?
We wanted to see how that compared .
Doing it that way is the independent test is in the red.
The paired way is in the blue.
What you see is, if I have little to no donor-to-donor variability
in my cell therapy split apheresis process ,
you said that the independent t-test
has much better profile than the paired approach.
However, if I have high donor-to-donor variability,
that paired approach in the blue
has a much better operating characteristic than the red.
Right now, the question is instead of just automatically
doing that split apheresis pair design,
maybe it would be better
to make a decision based upon that donor-to-donor variability.
How does this compare whenever I'm looking at different k -values?
I see the exact same thing, the exact same phenomena that
with a low donor-to-donor variability,
it makes sense to do the independent t -test.
With high donor-to-donor variability is I have a much better
operating characteristic curve
are higher power associated with the paired approach.
It doesn't matter if I looked at a k of 1.5 , or 2, or even 3.0.
Regardless of the k -value ,
I have a much better operating characteristic curve
if I consider that donor -to -donor variability.
What if I looked at different values of those historical lots?
I looked at 3.
We looked at 4.
W e looked at 5 paired lots. We looked at 6 paired lots.
Regardless, you see the same phenomena.
We're currently writing a paper on this
to try to propose
that if you have low donor-to-donor variability,
maybe it does not make sense for you to use a split apheresis
or a paired analysis approach.
Maybe the approach is only good
whenever you have high donor -to -donor variability.
T hese are typical questions that are asked in the split apheresis designs.
What I want to do is I just want to cover
t wo or three more of these
j ust to show you a few o ther things that you could do.
These are different things that we were looking at .
We looked at, "Hey , how does the operating characteristic curve,
how does that compare if we looked at in the blue
that's using nothing but the historical lots
to estimate the standard deviation
versus if you use the paired and the historical lots,
which is in the red?"
What you see is there's not much difference between these two,
especially if I'm using higher sample sizes like the n 2.
W e also looked at,
"Hey, if I estimated that standard deviation
using a few different ways,
what if I looked at estimating
that standard deviation using the historical lots,
which is in the blue, versus in the red
is using the historical lots and the paired lots?
I compare the independent case versus the paired case.
What do I see? "
As I said before, you see that exact same phenomena
with a low donor-to-donor variability.
The much better way of doing this would be an independent t -test
on the lower right -hand corner.
That is where you high donor-to-donor variability.
It makes sense that we would use the paired approach.
Last one that I want to show you
is this is something that we've been working on.
We looked at the paired approach versus the independent.
The paired approach is in the blue.
The independent is in the red.
I've said this over and over and over again.
That it makes sense that if I have low donor-to-donor variability,
the independent case in the blue looks much better.
If I have high donor-to-donor variability, the paired approach looks better.
But one thing that we did i s we took a look and just said,
"What if I took a look at the approach that gave me the shortest
with that confidence interval?"
That's in the green .
What you see is that usually gives you the best approach
regardless of what your rho is
or what your proportion of donor -to -donor variability is.
In closing,
I would like to just point out a few things.
This script that we have answers, along with the typical questions
that our customers have on operating characteristic curves,
associated with these split apheresis designs,
what I do want to pull away from here, though,
is if you do have a low proportion of donor-to-donor variability
is you'll see that these designs are very underpowered
for fewer than 8 lots, fewer than 8 different donor material.
We live in a world in cell therapies
where you do not have a lot of donor materials,
so you have very low sizes.
It would be much more efficient if you had low donor-to-donor variability
to use the independent case.
We do have the other revisions that we made on this
where if you were able to make multiple lots
for those paired approaches with the same donor,
or if you're able to take multiple measurements
to be able to look at those operating characteristics curves.
Thank you.
