Hello everyone.

This presentation will be about Measurement System Analysis,

and optimization of a UHPLC M easurement System.

Presenters are myself, Frank Deruyck

from HoGent University College of Applied Sciences,

and also Volker Kraft from JMP Academic Program

will take care of the demos

demonstrating the JMP tools for data analysis.

Okay. The problem statement and description.

Well, in a chemical company,

just [inaudible] chemical company,

the presentation was inspired by an internship of a student

and a chemical company, and of course the material had to be kept confidential.

So I just will talk about the chemical company,

and also the figures are a little bit modified, but no problem.

What is the statement?

Well, SPC revealed significant batch to batch variation in raw material

and has caused problems in product quality.

So that, okay,

there was an issue with the supplier that was necessary that all supplied batches,

it was necessary to analyze them all.

But of course one problem is that

there was a too slow analysis procedure GC,

and a fast UHPLC analytic method is under development,

was in development, but not ready

for validation because of too strong measurement variation.

And the goal of this study is to specify robust and optimal settings

of the UHPLC method so that validation of the new method will be possible.

Thanks, Frank.

Working with the JMP academic team for more than ten years now,

we helped many university professors worldwide to get access to JMP licenses

for teaching, but also to teaching resources like the case study library

at the link, jmp.com/cases,

professors get free access to more than 50 cases,

each telling a story about a real world problem

and a step by step solution, including the data sets and exercises.

What we present today is available as a series of three case studies,

focusing on statistical process control,

measurement systems analysis, and design of experiments.

While Frank will talk about the problem

and the solution they developed for a Pharma company in Belgium,

I will demo some of the analysis steps using JMP Pro.

Let me say thank you to Frank for sharing these cases with the academic community,

who really welcome such real-world examples

coming from practitioners in the industry.

I also want to thank Murali, from our academic team in India,

who plays a key role in enhancing our case study library

like the development of these cases together with Frank.

Okay, here in this plot you see very clearly the illustration the problem.

So what here is shown is a plot of the measurements of the new UHPLC method,

but non- optimized and also as a function of the measurement

of the GC standard method, which was very accurate and precise.

And you can clearly see that there are some problems.

So different operators made some measurements on different batches

and you can see that the prediction intervals are quite large.

You see a range sometimes over 100 milligrams per liter.

And you can also see that sometimes it's not clear whether a measurement

is within specification, like on the left graph you can see it

and also on the right graph you can see there's also ratios with accuracy

meaning that there's a serious problem,

and first of all we will explore the variation root causes

using measurement system analysis,

the causes of variation, and also the DOE will use optimized

according to the statistical thinking concept

which will be illustrated in the next slides.

You see the statistical problem solving process flow.

So for the cost of the problem,

the problem with our U HPLC measurement error,

we will tackle this by measurement system analysis,

and to export the variation root causes, I may use of course DOE,

also to optimize the process settings of the U HPLC system.

The method we will use for quantifying the variation sources is

the measurement system analysis, and some theory I will show.

So it's about the quantifying of the variance components of the total variance.

Total variance means the variance by all measurement,

by different operators, different products.

So we have two different components,

the product variation,

Sigma square product, and the measurement variation.

The measurement variation very important is also decomposed

in two important components, the repeatability the variance

due to lack of precision by repeated

measurements, and also the values between operators.

So the Sigma square large R which is the reproducibility.

And very important criterion in order t hat, sorry,

important criterion

stating that the measurement system is only suitable for detecting variation

in the process, process variation, is that the percent G auge R&R, which is the

measurement error divided by the total error, should be less than 10 percent.

So the fraction of measurement error was

below 10 percent total of the total variation.

Then we can use it for process follow up.

If it's not the case, it is higher,

then we run the risk that we will control our process

on measurement variation, and of course not a very healthy issue.

So it must be lower than 10 percent, that's the main criterion.

Okay, let me go to next slide.

And for this we will use a Gauge R&R study, a Gauge R&R study

is mainly an experimental design,

so that we will select three random operators,

John, Laura, and Sarah,

who will do repeated measurements two times on four different batches each.

So we have done,

we have been able to quantify the within operator variation

repeatability, the reproducibility, the between operator variation,

and also the product variation, the variation between batches.

So, Volker, I'll leave the floor to you now.

Okay. Thank you, Frank.

And before I will come to MSA,

I would like to briefly cover what's included in the first part of the series,

namely control charts and process capability.

So this is one of the data sets here

measuring the two compounds, our continuous responses,

compound one and compound two, using the good but slow GC method.

So data has been collected over eight days,

with two batches per day and for two different vendors, A and B.

So when the team started,

the first activity was to check and confirm normality of the data

for this, they looked at normal quantile plots.

They also fitted a normal distribution.

Okay, sorry for this.

So they looked at a normal quantile plot,

and they also fitted a normal distribution followed by a goodness of fit test.

And there was nothing critical from that analysis.

Exploring distributions, the problem became clear.

So looking at data from different days, we can see a huge

batch by batch variation, even for batches coming from the same day.

And this means that e-process monitoring is not possible

because

the variability of this GC method was high

and the method was too slow.

So it needed a lot of time

to monitor the batches.

And therefore one team activity was

to work together with the vendors and others to reduce that variation.

And another activity,

which is described in the other parts of the study, investigated a faster method,

as mentioned by Frank, using the new UHPLC measurement method.

So just looking a bit more into the old method, the GC method.

So the team looked at one dimensional

control charts and also process capability for both compounds, of course,

and they also looked at multivariate and model driven

control charts.

And here we see that there were some extreme points in a multi dimensional

analysis, and they could also see the contributions of the different

compounds or responses.

This is the conclusion of the first investigation using the old method.

So here the outcome was that

both processes looking at the process performance here, that both processes,

so compound A and compound B, were incapable but stable.

For vendor A

and for vendor B,

we see that compound one was even unstable following these colors here.

So all of this motivated the team to improve the measurement process.

And this brings us to part two of this series

and this is about analyzing the measurement system.

So this data here were collected for all the combinations, repeated twice,

between four batches and three operators,

using the new UHPLC measurement method.

So the goal here was to measure all

batches of raw material using this faster method and maybe to allow some

inline monitoring of the raw material in the future.

So to get started with the new data,

the team looked at a two way ANOVA, and this may fool you.

This output.

So on both compounds,

compound one and two, the batch effect is highly significant.

So that's good news.

And also the operator effect and the interaction between batch and operator,

these effects are non- significant, but the RMSE is quite high.

So that means that we are maybe looking at data

and the effects we are also interested in are just hidden by noise.

So before you look at such an analysis,

the first question should be where is the variation in our data coming from?

And second, are we measuring the signal?

Are we really looking at the signal or are we just measuring noise?

And to get an idea about this,

a perfect visualization of the patterns of variation is a variability chart.

And for these two sources of variation, batch and operator,

here we see all the data points for all operators and all batches.

So we have two measurements per batch per operator.

And for these two, we see the mean.

We also see the group mean for one operator and we also see the overall mean.

That's the dotted line here for all our measurements.

And we can look at this for both compounds, of course.

So here, for instance, for compound two, we see that Laura has

quite high variation, at least compared to the other two operators.

So that's a visual analysis.

The analysis method and procedure to use

to get a better insight into the measurement system's performance

is an MSA or measurement systems analysis.

And this was also done for the non- optimized U HPLC method.

So here, for instance,

for the first compound, we see this average chart

and this chart shows the data together with the control limits

or with the noise band.

And what we see here is not good news at all,

because our measurements, our data is within this noise band,

so it will be really hard to detect any signal with that noise level.

Another output is this parallelism plot.

So here we can check interactions between batches and operators and this would

indicate an interaction if some of these lines are not parallel.

And this is the EMP method.

So this stands for evaluating the measurement process

and you probably know this as Gauge R&R output.

And that's what also Frank mentioned.

So here we see the signal, that's the product variation,

but we also see the measurement variation

split into repeatability and reproducibility.

And here for the first compound,

we see that we seem to have an issue with repeatability.

So the same operator doing the same measurement again.

And for the second component,

we see that there's also a slight issue with reproducibility as well.

So these are measurements between different operators.

So the conclusion here is that the measurement system

or the measurement process is unable to detect any quality shift caused

by a significant systematic variation between our batches.

And with that, I hand back to Frank.

Okay, thank you, Volker.

I think we can shift to next slides because that's what we discussed.

Okay, it's what you showed, Volker, that's also in the slides.

Okay, yes, here we can start.

And we start with first of all, in the design of experiments,

the optimization study, the process improvement study.

And first o f all,

in order to better specify our experimental goals, we should first of all

go to the root cause analysis for the high measurement error.

The lab team,

after a brainstorm with the lab team,

it came out that were two main

root causes, one link to the equipment and one link to the method.

And the one link to the equipment was

the main source for poor repeatability because it was a very strong issue

with unstable column temperature and eluent flowrate,

the UHPLC uses column, it's a chromatographic technique,

and also an eluent to make separation,

to lead the compounds to the column and make separation possible.

As a matter of fact it was drift between

different experiments and even within one experiment.

This results of course in a poor repeatability.

And the first task, of course, of the lab team

was to stabilize of course the column temperature and eluent.

Because if you go to any experimental design,

of course, we need to have fixed settings of the temperature

and eluent flowrates, which are of course two important factors.

For the second issue,

the method because, besides the stability problem,

which of course was fixed, there was also an issue with

low resolution because of non- optimized analysis process settings which were quite

low and also unstable, meaning that when we make small shifts to flowrate

and temperature, there were sometimes huge shifts

also in resolution, indicating that there was also not only

optimal problem but also a problem with robustness.

So the goal was specify not only optimal settings but also robust settings.

So the variation of the resolution should

be minimal as a function of some variation around the settings

of the process, practice, of the analysis.

Let's just now go to the DOE.

And the goal, of course, of the DOE is that

we specify the response variable is Y, the compound concentration

in the standard samples, and we have to make models of this Y,

the compound concentration is a function of the UHPLC control factors.

And once we have the equations,

we can then of course go to optimization, and what is the optimization criterion?

We will use the quality P over T ratio criterion.

That means that,

that our,

fraction of the error

in the tolerance range should be lower than 10 percent.

If you have the tolerance ranges of our compounds which are specified,

compound one

in the standard sample was a 300 milligrams per liter

plus or minus 200 milligrams per liter,

then we have to make sure that and also the, sorry, the standard sample, two,

two standard samples, the target of compound two and the standard

sample two is 450 plus or minus 150 ppm spec limits.

And of course if you want to reach 10 percent

of this specification ranges, that means that okay, we have to

given our desirability function for optimization, is that okay?

Why should match target compound concentration lower than 10 percent?

Meaning that for standard sample one there

should be 300 plus or minus 20 ppm, 20 milligrams per liter .

And for standard sample two the compound concentration should be

450 plus or minus 15 milligrams per liter

that's the criterion for optimization.

So as for our model getting together with lab experts.

The factors are the main effects, the main effects and all quadratic effects.

And the main effects are the temperature of the column,

with the range 25 to 35 degrees Celsius, eluent flow rate,

five to 15 milligrams per milliliter, and also a gradient.

What is a gradient?

Well that means that there is an additive, Acetronitrile, in the eluent and this

concentration of this Acetonitrile increases as a function of the volume,

added as a function of flow, the volume, through the column.

That means

from volume zero and one milliliter it's a range between five and 20 percent.

And once the volume is five to six milliliters,

we have a range of 35 to 70 percent of Acetonitrile in the eluent.

Also an important factor is the UV wavelength .

The detector is by UV and it should be controlled between 192 and 270.

And brainstorming with the lab experts, who did already quite some preliminary

experiments and had some experience with the UHPLC,

as a matter of fact, only two interaction effects were selected.

That's the temperature and eluent flowrate and also

the eluent flowrate interacting with all gradient factors specified above.

So this means that the design chosen

to meet the goals and to model, and to specify the model parameters.

It was the custom design, and Volker will illustrate what design was about.

Okay.

Thank you Frank.

So talking about the third part of this case study series which is about designing

experiment and what we learned so far is that we have to reduce the measurement

variation which is caused by this non-optimized UHPLC method.

And this method can be described, as Frank pointed out, by these control settings

or process settings, like temperature and so on.

So we also want our responses remain within their limits.

So this follows the 10 percent rule as given by Frank,

and these limits are also added to our data.

So to design such an experiment the team looked at a definitive screening design.

So this one here, and also added custom design and both with 25 rounds.

So comparing both designs, they use the compare design platform,

and you can see several reasons which are in favor of the custom design.

So here, for instance, for main effects we see slightly better power for the DSD.

But for the other higher order effects we see

really a high benefit for the custom design.

Same here, looking at the fraction of the design

space we see that the custom design is doing better.

You could also look at the correlation maps,

and finally, the efficiency is also in favor of the custom design.

And for that reason the team used the custom design,

a custom design for those studies.

So here we have the completed data

for the custom design, completed with both response measurements.

And for this we also have

the corresponding linear model.

So here for the first response,

compound one, and for compound two, both with their profilers of course.

And here's a combined profiler

with both responses at the initial, just the mid settings.

And by maximizing the desirability,

so I would get to the optimal settings and we see that we are matching

both targets here, 300 and 450 respective, we are matching them perfectly.

However, we also see quite large sensitivity indicators.

These are the purple triangles, and they are telling us that at the optimal point,

our response surface is quite steep in some dimensions

and this reduces the robustness of our process

in case of some random variation of our process settings.

And this can be further analyzed by adding

the simulator to this profiler and that is done here.

So here the simulator

defines the random variation which was defined by our process experts.

And just keeping the mid settings, plus this random variation, simulating

10,000 of response values we see that all of our response values are out of spec.

These are all defects. Of course, we are just at the mid settings ,

so nothing better to expect here.

Switching to our optimal settings and simulating again.

So now we see that the defect rate to be expected is above 12 percent.

So from here the robustness can be further improved,

either manually, using the profiler and the simulator

and the sensitivity indicators, or automatically by running

a simulation experiment which is also built into those profiles.

So the team used a manual approach

and these are the robust settings.

They came up with these red settings here.

And if I simulate again, §we see that the defect rate

now drops below one percent.

So this is a Monte Carlo simulation, it's all random so the defect rate changes

slightly with each new simulation, and you can also see how the histograms

of our simulated response data behave quite well.

So they are within our range, within our limits

which support this 10 percent rule.

And, at these

robust settings.

So going back to the other profiler.

So here I also have

some contour plots using the contour profiler, and they can also be used

to better understand the best regions for the processing and for configuring

the process and these are typically the white regions which provide

the in spec regions for a combination of two control factors.

And I hope you like this journey,

and with this I hand over back to Frank to discuss this outcome.

Okay, thank you, Volker.

So,

now we can go to the validation experiment,

of course, once we optimize the settings

now we can make an experiment to check whether really the measurements of the new

UHPLC system and the GC analysis are equivalent.

So for this, again, we set up a Gauge R&R study,

measuring also one

which is quite similar to the one

discussed before but also now we make measurements with the GC in order

to compare the two measurements methods. You see here also included as one extra

factor in the Gauge R&R is the instrument factor.

Okay.

So here are the results and you can see

really that we make quite an improvement that you see in the Gauge R&R results.

That the main variation now is product variation.

You see that now the batch variation is no more obscured by noise measurement error.

The gray area is quite narrow compared to the ones

we had before, and that's a very good news.

You see it's mainly product variation.

So the Gauge R&R and ratio is quite, matches our target.

Meaning that precision tolerance ratio here is about eight percent.

And the precision/tolerance ratio is six times the Gauge R&R figure

divided by the tolerance of the compound , 400, which is eight percent.

So the precision is okay.

So that this measurement system is suitable

to be used in quality control for compound one.

Nice.

Compound one. Okay.

You see on the parallel plot just a little crossing,

you see that for Sarah, indicating a little

maybe interaction between the batches and operators.

Okay, for compound two, we have to see the same thing even better.

So only 5 percent of Gauge R&R,

the precision tolerance ratio, 5 percent,

same thing, very narrow noise range and no major crossing of the lines.

Quite parallel, indicating no operator bias, no interactions.

Modelling the compound one analysis, we see that it's mainly

influenced by batch, and also little batch to operator interaction effect.

Compound one and for compound two.

Okay, now we can see this very small

interaction fact because we have reduced the measurement noise so much

now, that very small facts, of course now become visible.

In the first time we could not see it, we could not detect it.

But that is because of a very poor experimental power.

But now we increase

the experimental power

seriously by just reducing the experimental noise,

of course, now this interaction in fact becomes visible.

But the small one, you can see that also

for the link to Sarah, the green line of Sarah,

but here also was a little problem in the GC

analysis as well, not only for the UHPLC, but also for the GC analysis.

Okay, that's an issue to be tackled for wrong.

But now it's here.

Those two graphs illustrate fairly clearly

that both measurement systems are nearly equivalent, the UHPLC results

versus the GC results.

We see that there's nearly a perfect,

very good correlation with all points on the mid line.

So the slope is nearly one.

It's not significantly different from one.

And also there is an intercept with the y axis,

is not significantly different from zero.

So the method was ready for validation, UHPLC is accurate and the

non-significant difference, which you see, standard analysis , so very nice results.

And we could say that the problem with tackling the problem with MSA and DOE

was very powerful, leading to a good, very nice solution to our problem

which could be implemented, of course, now in production.

Thanks for your attention and if there are any questions,

please let us know.