Thank you for giving us this opportunity to talk about how we use JMP in pharma process validation and continuous process verification. My colleague, Louis and I will give you a guide to the background of it, how we do it, and what is the outcome.
The agenda is that first we will describe the background and the concepts. Since its validation, we must justify the assumptions we have behind our models, so sanity check is important. Then I will go on to talk about the particular methods we are using and which formulas we have to make for things that are not there in JMP upfront.
Then Louis will take over and show how the JMP script is doing it to automate the calculations and implement what is missing in JMP. Finally, we will make some conclusions and hopefully there will also be time for questions.
Now to the background and the concept we are working with. Louis and I come from a company called NNE, and we do consultancy within the pharmaceutical industry, and we are JMP partners, so we help our clients getting value out of the data, of course, using JMP.
A big issue within the pharmaceutical industry is process validation when they're going to launch a new product. T raditionally for many, many years, it has been done by making three batches. I f they were good, the validation was passed.
But this is not really predicting the future, this is predicting the past. So, o f course, you can make three batches that were good, and then the problem can appear later on. To compensate for that, that, the industry has traditionally had an extensive and thereby costly QC inspection on every batch to ensure it was okay.
However, if we could prove that all future batches would be good, then we could reduce the QC cost heavily. But how do we predict that the future batches will be good with confidence? Yeah, we could use prediction intervals or individual confidence levels, as I call in JMP, at least in some menus.
If we put batch as a random factor, then the validation batches we are looking at will be seen as a random sampling of all future batches. T hereby, we can predict how future batches are going to be. If this prediction is inside specification, we have actually proven that all future batches will be good given that we can maintain the validator state force.
However, this might take more than three batches, but do they all have to be in state two, as we show in this graph? Maybe not. We actually should just still make three batches in your first process of validation. If it's not enough, then make the rest in which we call straight three A, which is after commercialization, because then you can just sell them one by one. It's not a big problem that you have to make more.
However, this requires two things to be fulfilled. The first is that we still have this extensive QC test that we can find out if your batch is bad because we are not yet tooled, it will not happen. A lso, of course, it requires that the estimated performance is okay. How can we see if the estimated performance is okay? We can do that by seeing if control limits are inside the specification.
Then we can go on the market and later on, when also prediction limits are in such specification, then we can actually remove or at least reduce the QC inspection. We might be going from monitoring the product until monitoring the process.
What is it all about? Validation is about predicting the forms of future batches with confidence, not just predicting the past. How can we do that? We can do that by using prediction or 12 intervals in JMP. How many batches will we suggest to make in stage two? We still suggest three, but you might not be done.
How can you then pass stage two with few batches so we can go on to the market? That's, as I said before, look at control intervals because they are without confidence. You're not punished by only having three batches. T hen later on, look at prediction or 12 intervals.
If control intervals are inside specification, prediction intervals are not. The most problem of the reason for that is lack of data. This shouldn't be limiting you to go onto the market.
Then how many batches to make in stage 3A? That's fairly simple. Simply until your prediction limits are inside specification, then you have passed your stage 3 A. These limits you can also use as prediction limit after stage 3 A when you go into stage 3B.
We can also see it on this flow chart where you in upper left corner, start with analyzing your validation ones, typically tree. Then you calculate your prediction limits in JMP as a random factor. If they are inside spec limits, everything is fine. You have passed both stage 2 and 3 A and everything is fine.
If it turns out that prediction limits are too wide compared to the spec limit, I would, as a next thing l ook at my control limits which are without confidence. If they are inside, then we just need more data and we will collect more data and we calculate and a point of time we will go out here and have passed also strict TA.
Of course, if it happens that even control limits are outside specification, then actually the estimated value is bad and you are not really ready for validation and you have failed. Hopefully, this is not going to happen.
Let's go on to the sanity check. Now I will go into into JMP. I will try to demonstrate how it works. Here you see a JMP table which we have made, or not we have made, it actually comes from ISPE, International Society of Pharmacoepidemiology, who put us this data set as a good example on how a validation set should look like. Many companies or consultancy has tried to calculate on that and see how would they conclude.
It's basically a data set from tablet manufacturing, and classically it contains these three batches A, B and C. Then we are taking out powders in 15 locations in the blender, and then we are measuring the API content, that's the strength of the actual [inaudible 00:06:52] ingredient, which would be between 85 and 115 to be inside specification.
If you look at the variability chart of the data, you can see here we have the three batches A, B and C, and we also have the 15 locations in each batch. Then within each batch location combination, we have four measurements.
The first you can see is that the within location variation on batch B is somewhat bigger than on batch A and C. We do not have ba tch heterogeneity. To make that more objective, you can put S control limit on your standard deviation chart and you can see that you are out of control in Batch B.
Another way of seeing it would be to make a heterogeneity of variance test as you're seeing down here. Then it's also very clear that there's more within or in location variation on batch B than on the other ones.
Strictly, you cannot pool these variances however you can, but we need then to do weighted regression where we weigh with inverse variance. That's the next step we will do, go into log variance modeling, which you can do in JMP.
We have simply here just made a log variance model where you can see that the location, the batch, the interaction all have a significant influence on the level, but only the batch has a significant influence on the variance. As you can see that the variance for batch B is somewhat bigger than within variance on batch A and C.
When you have built this model, you can from up here just save the columns and you can save your variance formula. I've already done that. It's already in my data set. If I unhide here, here you can see my variance column from my log variance model. You have a formula here saying, what is it in group A and C for batch A and C, and what is it for batch B? Based on that, I can now make weighted regression where I simply just make a weight column where I weight my regression classically with inverse variance.
Then I'm ready for making a sanity check on my data because now I can pool the variances after weighted regression, so I can start doing sanity check on my data. The first thing I would do is to look at a systematic model where I have my location, batches, and the interaction. T hen I can see on my [inaudible 00:09:28] type residual plot, I have no outliers and outlaws , but it is inside.
I can also see on my box plot transformation, there's no need to transform my data. Here it's reasonable to assume that I have these normal distributed residuals, which is the assumption behind a fixed least squares model.
Then next step is to put batches as a random factor, not just a systematic factor, because I would like to predict future batches. This is what I have done here. T hen suddenly you get these variance components. What is the between batch variation? What is the batch location variation? W hat is the residual mean within batch within location variation?
Since, of course, I have weighted it with inverse variance, this number, of course, gets close to one. However, in this model, there are also some additional assumptions, namely that the batch effect and the batch location effect is randomly distributed, meaning that these random effect predictions up here should follow a normal distribution.
To test that, you can just put them into a table, which I've done here, and we can test the loops just making an empty model where you put them as a response. You can see both the batch signals or the batch variances and the interaction variances all are inside the residual limits. Also here we can justify the assumption that these are just random factors. This is the sanity check of the model.
Now to make models for batch A and batch C, we need to scale them differently. That's what I have done in these columns over here. If I unhide my columns here, you can see I made an AC scale weight where I simply just multiplied my weight factor with the scaling. So I get the variation I have seen for batches A and C, and the same thing for my batch B. I can scale the weight with the variance I would like to have as a residual variance, which is just coming from my block variance modeling. That's also pretty straightforward to do.
N ow I can actually have my two models from where I can predict. One where I scale with my AC scale weight and one where I scale with my B scale weight. Thereby I can see, if I just run one of them, one scale with the AC scale weight, then you can see that you have this model, you have these residuals, you have these balance components.
What is very easy in JMP is just to go up here and you can go and you can save your columns. F rom up here, you can actually save the prediction formula that would be your center line. But you can also save your prediction interval that here are called individual confidence interval.
I have done that both for this model and also for the other model. I also want the other model with the different scaling for batch B. T hen I save these columns as individual confidence intervals. Then they are just saved down here as these prediction formulas. Then I can make a combined one by just combining these depending on which batch group things are coming from. Now I have limits for both batch A, B and C, which I can plot on the same graph.
This is straight over to do. Until now, I just used what's i nside JMP. But to get the control limits and to get the tolerance limit, I need to calculate them myself, because in JMP, you don't have tolerance limits, you don't have control limits in Fit Model.
Of course, you have control limits when you do control chart, but that's also only for one mean and one standard deviation. Of course, you also have tolerance limits in analyzed distribution. But again, that's only for one mean and one standard deviation.
Here we have to calculate them ourselves. The first thing we then do is just get the variance components. This is what we have here. W e just saved the variance components from the two models, one where we scaled to batch A and C, and one where we scale to batch B.
Because when we have the variance components, it's pretty straightforward to calculate the control limit because it's basically just the prediction formula minus a normal quantile, and then the square root of the total variance, and then either variance A and C or B, depending on what we're looking at. I t's actually pretty straightforward to calculate the control limits that way.
The same we can do for the tolerance limits. If you prefer tolerance limits instead of prediction limits, many companies do. Again, unfortunately, they are not in JMP model, but we can calculate them. There we simply just put in the classical formula for a tolerance limit. Where one sided, we put in the T quantile, we put in the degrees of freedom, we put in the normal quantile, and as before, of the proportion, and then we multiply by the total balances from the variance component.
The only little bit tricky thing here is that we need the degrees of freedom under total variance. JMP is not really giving you that. It's only giving you degrees of freedom in systematic models. But we can see based on the width of the prediction interval, that's what we have here. Then we can actually calculate based on the width of the prediction intervals, how many degrees of freedom do we have there, and then we just apply the same degrees of freedom on the tolerance limits and then we are ready to go and we can save all of these limits down here. If we want, we can plot them. This is what we have done here.
Here you can see actually the prediction limits and the tolerance limits and the control limits. The control limits are the dotted line, the prediction limits are the full line, and the rough dotted line are the tolerance limits. You see that for the 15 different locations and for the three batches A, B and C and of course, limits are wider for batch B because it has more within location variances.
You can actually see that even the control limits at the dotted line are outside for location 1 and 2 and actually also four. Even though this process, this is a good example of a validation, I would say, well, if batch B is representative, you have actually failed the validation.
If you look at A and C, then all the dotted lines are inside, meaning that we have passed stage two because the estimate value is fine. But on a few of them, you can see that the prediction limits are just outside. But when you have this big gap between control limits and prediction limit, just one batch more would probably make them inside. This is how it works.
Some companies they do not think it's enough to look at if the limits are inside specification. They would actually like to get a Ppk capability index on them. We also put that one into our method here. If you look at the formulas here, you can see where we have the prediction limits, that's also put in a Ppk formula.
Basically what we do, we just take the prediction formulas, this into the spec limit, and and divided by the half width of the prediction limits. This is this Ppk that corresponds to the prediction limits. Because they are with confidence, this is actually Ppk with confidence.
You can do exactly the same with your control limits. Just take the classical Ppk formula and just put in control limits instead of prediction limits. B ecause control limits are just estimated values, this is Ppk without confidence. Of course, you can do exactly the same thing also for the tolerance limits, which we have down here, a Ppk based on the tolerance limits. You can also calculate all of these. If you want, of course, you can plot them on a graph as you see here.
Up here, we have estimated Ppk without confidence, that's the red one. T hen we have the blue one that is based on tolerance limits and the green one that is based on prediction limits. Up here, it's for the good group A and C with indication variation. Down here how it is with batch B, is a bigger one.
You can actually see, as we saw with the limits for batch B variation, even the estimated value has a Ppk below one for one, two, and four positions. While all the estimated value is nicely about one for A and C. A lso the Ppk based on prediction, the green one, most of them are about one, and they're very close to one on location one and two. This would just require one additional batch probably, and then they would be inside.
However, as you'll probably see it's a bit tedious to do all these calculations by hand. This is why we have made a script for it, which Louis is now going to demonstrate. I should just Louis take over now and shows how the script works. I'm sure he will also make a few more examples.
Yes, Pierre, thank you very much. I will just try to share my screen here then. Yes. First thing first, as Pierre has shown us, we face a lot of issues during the special validation. Luckily for us, JMP has a lot of straight up box functionalities to help us deal with these issues. We really like to use the functionalities coming out of the box with JMP because this really lessens our validation burden when going forward.
Some of the things JMP has solved for us is, for example, the issue that we are working with complex processes and these cannot be described by just one mean and one standard deviation. Here to deal with this, we're leveraging the Fit Model platform in JMP where we can have multiple systematic factors to handle the many means, and we can also have multiple random factors to deal with many variance components.
Then there's the issue with data often needing some data transformation. Again, JMP has the Box-Cox transformation window we use to deal with this directly in the Fit Model platform. Then we usually see a lot of outliers we have to handle in bigger data sets. H ere we again leverage the Fit Model platform, more specifically the student sized residuals plot where we exclude outliers based on the Bonferroni limits.
Here it's just important to note that at least our concept is that we like to exclude the actual model, but we always report them separately. It has to be visual that these are not included in the analysis.
Then there is the lack of homogeneity of variance, which Pierre also showed how to deal with using the log variance modeling to get out the variation and then use this variance to compute a weight formula.
Then again, there is the prediction intervals which we use quite a lot here. Here as well JMP is ready to go with individual confidence intervals directly in the Fit Model platform.
But then as Pierre also mentioned, we felt like there were some parts where JMP could not bring us all the way home. This includes, for example, controlled intervals in JMP, which can only handle the one mean and one variance component.
How we overcome this is that we calculate controlled intervals from the Fit Model using the prediction formula and standard error on prediction formula and residuals to calculate the total variance, and from here, the control intervals.
Then there is the fact that tolerance intervals is preferred by many of our customers because this allows you to separately set the confidence and coverage. However, we only found that JMP can do tolerance intervals in the distribution platform, and it can only handle one mean and one variance component again.
Of course, in the script built- ins that it can calculate the tolerance intervals similar to the control intervals. W hen we do the calculation of the tolerance intervals, we use the same degrees of freedom as used when JMP calculates its prediction intervals.
Then maybe the primary reason for doing this script, or the initial reason at least, is that calculation and visualization of all these intervals is very time consuming and very prone to human errors. Actually, we decided to make this script to automate calculations and visualizations.
Then lastly, many of our customers require capability and dialysis such as Ppk. We have, of course, also included this in our script such that it calculates Ppk corresponding to both prediction limits, what we call Ppk with confidence, and also control limits, what we call Ppk without confidence.
To talk about the script a little bit, the script we have designed here consists of two parts. We primarily have the JMP script you see here in the middle, and then we also have a template document, as you see up here in the upper left corner.
When making this script, there were a few things we really wanted to achieve. We wanted to make it really transparent, meaning that our customers could go in and look under the hood, so to say, and follow the calculations of these different limits and potentially even edit them themselves, which takes us to the next thing.
We wanted this to be, at least to some extent, customizable by our customers. If they don't like the way we calculate Ppk, we would like to give them the option to go in and edit the equations.
The template document in itself is actually, so to say, an empty data table. It has a lot of defined columns with column formulas, but it has no rows in there yet. What our script does, essentially, is that it works from this template document as an input. It also takes the original data file on which you base your analysis as an input, and then it takes the model window.
Here it is to be said that the model window is assumed to be a complete model. So you have done all the all your data manipulations, ensured that this is a proper response you get from the model. A lso you have made these sanity checks and justified the assumptions behind the model ensuring it's valid.
What the script does itself is that it takes the template document, it takes the original data file, and then it copies the data from the original data file to the template document. We've done this such that we don't interfere with our customer's original data either.
It then also takes all the relevant parameters from the model window and insert it into this template document. F rom there on, more or less, this template document will fill out itself through the already defined formulas in the columns.
This template document will then in the end be converted to an output table where the first many columns would look very much like the original data file, and then you will have the remaining calculation and input values as additional columns further on.
I think we should go to an example and see how this looks in real life. I will just try to stop sharing this screen and share this one. What we have here is in fact, the same ESBIC case as [inaudible 00:25:20] here. What you would do is you would have your data, you will run a model. In this case, I run the model where peer scale it to correspond to the variation in batch A and C, the batches with the lower variation.
What you simply do is that you have your model, ensure it's done in sense of sanity checks and so forth. Then we run this script on this model, which is here. We have the model and we run the script.
We'll see a few things pop up here. First, we have the template document, output document which has now been populated with a lot of different rows, some table variables, and so forth. We have our Ppk graph here visualizing the Ppk based on different limits. The blue being based on control limits, the red being based on prediction limits, and the green one being based on tolerance limits. Then lastly, we have our graph here plotting all the data, specification lines, and as well our prediction limit, control limits, and tolerance limits here.
We get the image that Pierre also showed earlier. What we see here is, in fact, that if we look at our data and assume all batches varies as much as batch A and C, we are actually in an okay position here. At least we would have passed our stage two because control limits are for all batches, all locations within specification.
We do, however, have a small problem out here on the location 1 and 2, where we see at least the prediction of tolerance limits extending a bit over the upper specification limit. However, we believe this could be fixed by just adding a few more batches to know the variation better.
But what we also see here is, of course, we have an observation outside our limits, and this is because what we're looking at here is in fact batch B, which as we remember, had a higher variance. I would just try to run this model here, scale according to batch B, so you can see the difference.
Now what we see here is that all our limits has moved out from the mean. Now they're actually going beyond specification, not only on the first two locations, but on multiple locations here, which would, in essence, for us, especially because you have the control them as outside specification, mean that you have failed your validation here. Maybe not the best example for this data set to showcase a good validation.
Now I thought I would just try to go through how we envisioned the entire process relating more to the flowchart, Pierre, showed you in the beginning of this presentation. I have brought a customer example from an actual PPQ validation run. They ended up producing a total of six batches here.
What we have done here is that we have made a model. I will just deploy local data filter to simulate where it started, having only these three batches. Then what we have here is just, so to say, our finished model. What we do now, after having produced the first batches, analyzed the data, we just run the script. We get a result looking like this.
The scenario we see here is that both our control limits and prediction limits, the prediction limits up here, tolerance limits here, are both very much outside the specification. However, lucky enough for us, our control limits are actually inside specification.
This means in the sense of the flowchart that we now have passed stage 2 and we can actually start from now on to when we do our continued validation, getting our control prediction limit and tolerance limits inside specification, that we actually can start sending batches to market simultaneously.
But bear in mind here, we still have a pretty high QC effort in the end. The reason we believe that, because many customers also ask, how can we do this if we are so uncertain about the batches we produce? But if you want to know the quality of your already produced batches, we actually have to look at this in a different way.
Here in this model, we have batch as a random effect because as we consider it as a part of a larger population. But if we want to look at the batch individually, we actually have to put it as a systematic effect in here. If I run the script on this one, we will see the limits narrowing significantly in here. I forgot to deploy data filter, but I think this makes sense anyway.
What we see here is that the limits now are much closer to the center lines of each batch here, and they are way within our specification because we are now looking at the batches we have already produced and not trying to predict the performance of future batches. Which is also why we believe that this is an acceptable point to go to market while still having the high QC effort at the end.
If I just continue with our example, what we saw in this example was we have passed stage two because control limits are inside specification, but prediction limits are outside. The next step here would be to simply produce another batch, create your model, it's done here, and then run the script again.
What we see now here compared to the first view is that our prediction and tolerance limits are moving towards spec and also toward our control limits, which is the behavior we will assume as we start to get the between batch v ariants or s tarting to know the between batch variants better and better.
However, still, prediction limits are not inside specification, so the routine would simply be to do everything again. Now because I know how this will end, I will simply include the two last batch and we will on the model again.
Finally, after including the sixth batch, the results we get here are that all our limits are now within our specification, which means we have passed both stage 2 after 3 batches, and now we have also passed stage 3 A. Which means we now go into stage 3B, where we will start to put our attention on reducing the high amount of QC effort in the end and replacing this with continued process verification and CTV.
Just to sum up, I will go back to the presentation. The purpose of a validation is, at least in our opinion, to predict the future of batches will be okay. We firmly believe that if you can describe your validation set with a model, you can also predict the future with confidence. This can be done from either prediction intervals or tolerance intervals.
We believe that JMP has unique possibilities to model your data and also to justify assumptions behind your model. I t offers the opportunity to look at variance heterogeneity. If we do not have this, we have the option to do a log variance model to find a weight factor as Pierre showed you, and include that in our normal regression models.
Then we can look at whether or not the residuals are normally distributed. If they are not, we can do a Box-Cox transformation, also directly from the Fit Model platform. Then we have the problem of outliers, which we can handle through the student task residual plot, excluding observations outside the 95 % Bonferroni limits. Again, a window directly accessible through the Fit Model platform.
Then lastly, there is also that we request our random factors to be normally distributed, at least effect fronted. This we did by the blue test, and here we can also exclude outlier level.
Then to make the process easier for us, our customers, and also adding a bit on top of this to actually achieve the functionalities we desire, we have decided to make a script that can automate visualization of prediction intervals. It calculates and visualizes tolerance intervals using the same number of degrees of freedom as when JMP calculates prediction interval. It also calculates control limits for more complex processes where you have many means and many variance components. Lastly, it calculates also capability values for where these might be needed.