Managing Reference Drift in QC Assays

[#1]

In our lab, we have a QC assay for the release of new batches of a product. The assay works by comparing the new batch to a reference batch, approving it if it performs comparably. Over time, we have had to change this reference since we do not have unlimited quantities of it and it expires. Therefore, we often need to change the reference. Due to differences in the references, we have experienced drift over time, where the reference either becomes worse or better than previous references. This leads to situations where we reject a batch that performs worse than the current reference but would have performed just as well as an earlier reference. My question is whether anyone has ideas on how to manage reference drift in both directions.

[#2]

Hi @DI1 ,

Interesting question - I think a few questions would help clarify this:

1) What is the 'reference' material? Is it a batch that has passed in the past? What makes it a suitable reference material (i.e. what criteria does it pass to be called a reference).

2) What criteria defines a batch as 'worsening' or vice versa? Is this something that can be measured (i.e. % active material?) - if so you could look at the level of active material between each reference batch and account for how much this differs from the overall mean response of the active material (and apply this in comparisons of the new batches).

[#3]

Hi Ben,

Thank you very much for your thoughts.

(1) The reference is a batch that has previously passed our QC. Typically, we try to choose a batch that resembles the previous reference as closely as possible, but it will never be identical.

(2) Yes, it can be measured. We measure the fluorescence intensity of the reference batch and compare with the fluorescence intensity of the test batch.

[#4]

This is a horrible but very common problem. The bigger solution is to find a way to stabilize the reference (lower storage temps, adding antioxidants or other preservatives.) If your reference material has a consistent degradation rate with straight forward kinetics, it might be possible to add a time dependent offset. 

[#5]

When you say 'drift' three phenomena come to mind that we had to constantly monitor and deal with. Is it the mean that's 'drifting' or the variance or both? As my former JMP colleague (I'm now retired) @Byron_JMP mentioned...none of these are a good thing. Basically we had often had an 'accommodation' built into the measurement system SOP that called for periodic study of the references from time to time in a kind of control chart, augmented by DOE approach whenever a new reference was introduced. We used a Phase II control chart approach to monitor stability of the 'in use' reference...then when using a 'new' reference was imminent we'd run a designed experiment with the two reference materials as blocks to try and quantify the delta involved in the mean and variance. Then we'd just 'adjust' the value of measured quantity to go back to the baseline reference, recognizing that whatever number we come up with is just that a number...with inherent variation. Never really happy with this solution path...but it was the best we could come up with...but again...I heartily endorse @Byron_JMP 's idea over mine...but sometimes you gotta do what you gotta do.

[#6]

One other thought for you regarding the reference drift issue. Sometimes when we had batch to batch variation where there were different mean values of quality characteristics of interest, we'd blend the batches over time so as to create a less variable what I'll call 'current' batch. Not exactly a Lean way of doing things...certainly an accommodation...but something we did do. So for example for any one current 'batch' there might be three or more batches blended together in varying proportions. Eventually the 'oldest' batch is replaced by subsequent batch inclusion, but hopefully you get the idea. This tends to keep the mean of the current batch closer to the long term average of batch to batch production.

[#7]

Hi P_Bartell,

Thank you so much for sharing your thoughts and experience from similar cases!

[#8]

Given your reliance on this reference material, have you performed an FMEA or Root Cause Analysis to determine the cause of the drift or other variation? Have you performed a designed experiment with the factors identified in such activities to isolate the noise factors (affect variance)?

[#9]

Hi Mark,

No, we haven't done that, but it could definitely be something we can try to understand the drift.

[#10]

The key issue is making sure you properly characterize your reference material and making sure it is comparable to the previous reference material.  All too often what can happen is that a new reference material is prepared, and tested against the previous reference material but with an inappropriate statistical design.  Ideally you want a statistical design for reference standard evaluation/calibration that will encompass all of the typical and expected variability in the measurement process, and with sufficient replication so that you will be able to average over all of that variability and get a good estimate of the new reference standard's defined content.  

If you are simply doing a relative potency (test vs reference), the above still applies, but you may have to determine an appropriate adjustment factor (using the above approach) for the new reference batch to use when determining the relative potency.