When the Question is 'Is It Comparable?', JMP Has the Answer (2026-US-PO-2964)

I'm Heath Rushing. I am a principal at Adsurgo. We develop statistical training and consulting. We've been doing that since about 2010. Today I'm going to talk to you about comparability and really the tools that JMP has not only in the software, but also through the use of the flexibility of JSL.

Whenever you're thinking about the more modern process and analytical development, even manufacturing, is there's always a need for change. That change could be improvement of a system, that could be some a pre-change process versus a post-change process, or if you're transferring an analytical method, creating efficiencies where you were trying to decrease costs or decrease time to market.

It reminds me of a story. A few years back, I was at a consumer goods company, and I just taught a two-sample t-test. As I was walking out of the class, there was an engineer in the class that said, \!"Hey, I just got something from our statisticians, and they said this change that I made in the process was significant.\!" I said, \!"Yes, indeed, those are statistically significant.\!" He said, \!"Heath, it's not important.\!" I said, \!"You're really not asking the question, is it significant? You're asking, is it comparable?\!"

That industry and most industries there's a need for comparability, for saying, \!"I'm not trying to show that it's statistically significant, but I'm just trying to show that it's similar enough,\!" or what I'm going to call comparable.

I'm going to focus on a few examples from the pharmaceutical industry. It's an industry that we see a lot of need for comparability. It really comes down to the regulatory agencies want to ensure that whenever you change the process, or you change an analytical method, that they don't have to be the same, but they just have to be similar enough.

This really started in 1996. There was a guidance document put out by the FDA April 1996 about pharmaceutical manufacturers improving a process or even analytical methods. In this document, they talked about the need of that you no longer had to perform additional clinical studies in order to demonstrate safety and efficacy, but you needed to show that the pre versus the post-change process was comparable.

This guidance was extended in 2004 with Q5E. Q5E really had additional guidance on that pre- and post-change manufacturing process. I think this document right here is really what accelerated the use of comparability in the pharmaceutical industry.

Back when there was a need for biosimilars, they really looked back on generics, and they came up with taking your critical quality attributes and putting them into two tiers. Tier 1 was considered your most critical attribute. Actually, it's three tiers. Tier 1 was your most critical attributes, and those required the most what they call statistical rigor, but it was really the most scrutiny.

There was a request that equivalence testing be done for your Tier 1 attributes, pre versus post-change process or analytical method. Tier 3 was more process consistency, and you could do graphical techniques with that. While Tier 2 was somewhere in between. That somewhere in between is it was requested that you do a quality range. Very simply, just take the mean plus or minus some number of standard deviations of the reference process or analytical method and ensure that there is a large percentage of the post-change that fall within that range.

Today I'm going to talk about four examples. I'm really going to split it up into the first two and the second two. The first two are going to be very common examples in the pharmaceutical industry where equivalence testing is used. The first one is going to be for a one-sample t-test. The second one is going to be for a two-sample t-test.

The first example is about a change in reference standard. In the biopharmaceutical industry, the very first lot that's made, you have a reference standard that the potency of every other lot after that is judged against that first lot. It's often called relative potency because it's related to that first lot. At some certain point, that reference material is going to run out, so you have to show that the next batch of reference material that you're using to determine potency is at least as comparable to the first lot.

Not only does JMP have the tools to analyze this, it also has the tools to be able to determine power and sample size for the design of this. On the right-hand side, I really want to point out that it has some really great tools for determining if something is equivalent and even graphical tools for being able to communicate these things.

The second example that I want to talk about is used whenever you're changing a manufacturing process or transferring an analytical method. I'm going to focus on the analytical method transfer part of this. They're very similar tools because with the analytical method transfer, there's really two things that you need to consider. Say that I have a sending lab, and I am going to transfer this analytical method, this measurement system, into a receiving lab.

Whenever I do that, not only do I want to determine if, on average, both of them are comparable, but I also need to make sure that whenever I transfer that analytical method, that the standard deviation/variance/CV in the receiving lab isn't that much greater than what it was in the sending lab. I want to make sure that it has comparable precision.

Two different things are done: a two-sample equivalence test in JMP, and right after that or at the same time, what you want to do is you want to do an equivalence test for standard deviations.

I have a very simple example here where I have reference material and post-change is where it has the same mean and standard deviation. The post-change 2 has the same mean, but it has twice the standard deviation. Post-change 3 is where you see a shift of two standard deviations from the reference material.

In JMP, you can both do an equivalence test of means. As you notice, the first one, post-change 1, passes the equivalence test as it should. Post-change 3, being that you have a 2-standard deviation shift away from the reference, it does not pass. If you look at right in the middle, it fails for the reason that you have unequal variance. The difference is really no difference in means, but you see in that case it is failing, or it is alarming you that there is a difference between these two labs because there is an increase in the variance.

The test of non-inferiority, which can be done at the same time, would highlight you to the need for ensuring that the precision in the receiving lab is comparable to what it is in the sending lab. All these tools, including tools for design, power and sample size, and also analysis, are available in JMP.

However, occasionally something comes up that is more nuanced. It's a question that comes up from a customer that is not necessarily just a very common approach that is done in either equivalence testing or with quality ranges.

What I'm going to bring up is called scale-down model qualification. Scale-down model qualification is you are ensuring that your small-scale experiments are comparable to your large-scale results. Your large-scale results are really your reference. The scientists and engineers have a guidance document that tells them that they can either use an equivalence test or a quality range.

An example came up one time where someone said, \!"Heath, how does it compare if we did an equivalence test where we used an EAC, an acceptance criteria of three standard deviation shifts versus doing a quality range approach?\!" They want to make sure that they have an adequate power associated with either the quality range or the TOST.

We very simply developed a JMP script that asked them for the number of observations in the reference, the number of observations in the new product, what their EAC would be for their equivalence test, and what their C value would be for the quality range. What you can do is you can compare the power curves associated with each and ask yourself those questions about if I expect to see a small shift, which one would have higher power, if I expect to see a bigger shift, and at what point would it not matter if I use a quality range or an equivalence test.

The last example is really a very special example to do with accelerated stability testing when a pharmaceutical manufacturer is in the middle of a clinical trial, and they improve their process. When do they need to show that it is comparable in terms of means and maybe even the standard deviation, but they also need to show that it's at least as stable as it was before, that the expiration date that they have provided for this drug is the same pre versus post. In this situation, what you're doing, you're either doing an equivalence test for larger sample sizes or a quality range, and you're doing these on the slopes.

JMP doesn't do this necessarily, but you can write a very simple script. Indeed, what you see is the very simple script will do both of those tests for you. But right here, what I have is it will have the quality range. It will plot the quality range and use the variance associated with either the reference data or if you have some measure of the analytical variability that you want to use on your own.

What we did is this is very interesting because if you're comparing the slopes of the pre in blue to the post in red, one thing that you would probably want to do is you would probably want to adjust the graphs so that they all have the same intercepts so that you can compare the different slopes. Once again, just very simple, use the flexibility of JMP scripting language to be able to develop a script to answer very unique questions in comparability studies. In this case, it's accelerated comparability studies.

I wanted to show you multiple examples from the pharmaceutical industry, but I really believe that any of these tools that JMP has available or even in JSL can be used to answer a lot of the design and analysis questions that you have whenever you're not trying to determine if something is significant, but more importantly, you're trying to determine if it is comparable.

The two examples that I used with JMP's existing capabilities was a change in a reference standard for a one-sample equivalence test, and a change in a manufacturing process or analytical method transfer. I used the analytical method transfer to show you that you can do a two-sample equivalence test in JMP, and even a test of non-inferiority.

If you have a little bit more nuanced or more challenging questions in your comparability studies, like in scale-down model qualification, how does a quality range compare to an equivalence test or an accelerated comparability study where what you're trying to do is determine if the post-change slopes are comparable to the pre-change slopes, you can very easily use the flexibility of JSL to write a script to be able to answer those nuanced questions for you.

I did want to reference the 1996 FDA comparability guidance and also the 2004 Q5E, as well as our customers who always come up with these challenging comparability questions, and the team, Jim Wisnowski and Andrew Karl and Rick Burdick and Jeff Hofer and JoAnn Coleman, that always answer the challenge to be able to answer these challenging comparability questions.