I'm an Instructor in the JMP Education group.
My co-authors for this presentation are Jordan Hiller and Byron Wingerd,
who are both JMP Systems Engineers.
We're here today to warn you
of the dangers in using JMP's Levey- Jennings control charts.
We wanted to present at Discovery because we believe Levey-Jennings charts
are being misused, resulting in missed signals.
Today, I'll start with a quick control chart review
followed by a quick history,
and that will lead into how we've seen Levey-Jennings charts being misused.
We'll conclude with some recommendations
for use of these charts and some final thoughts.
Now, every quality system has a bit of process control in it.
These are the DMAIC steps from Six Sigma,
but you might be using QbD or another quality system.
Control charts are used at the end of an improvement process
when you've put the process on target and minimize the variability.
The process is now stable,
which means the probability distribution is not changing over time.
Now it's time to monitor that process variability
to verify that the distribution stays the same
and is not changing in the future.
What is a control chart?
A control chart is simply a run chart of the data
with the addition of control limits.
The center line is based on the historical mean of the process,
and the width of the control limits
is based on the historical variation of the process.
If the process distribution does change,
observations will eventually fall outside of the control limits,
signaling to the process owner to take action to assign a cause
to the out-of-control point and restore the process to a state of control.
Many types of control charts have been developed for different situations.
XM-R charts or individual and moving range charts
for data collected one point at a time.
Xbar-R or Xbar-S charts for summarized data
collected multiple points at a time.
CUSUM or EWMA control charts to detect small shifts,
and even m odel-driven multivariate control charts
to control multiple correlated variables on one chart.
To find those all important control limits,
you have to answer this question,
how do you estimate the historical process distribution?
First, you have to determine if the process is stable enough
for control charting.
Or does it need active process improvement?
This is called Phase I.
It's an active phase of data collection and sampling plan adjustment,
limit calculation, maybe even some statistical modeling.
This phase ends when you have collected enough data
to determine that the process is stable.
Then the SPC system shifts into Phase II.
The sampling plan and the control limits are fixed
and new data are judged by the fixed limits.
There's many considerations for Phase I data collection.
The sampling plan, including rational sampling
and rational subgrouping, the sample size,
how much data do you need to collect to fix those limits,
the type of data that you're collecting,
the type of control chart that you want to make.
Rather than talking about all these in detail here,
I will refer you to our excellent, free, online,
self-paced, e-learning course called Statistical Process Control.
It's available in the JMP Community.
You'll often hear that control chart limits are three- sigma limits.
This means that we need to estimate
the standard deviation of the process from our data, multiply by three,
and then add and subtract from the estimate of the mean of the process.
When Walter Shewhart developed the first control charts in the 1920s,
he realized that, like other statistical procedures,
he needed to test for signals using the noise.
He needed to compare the between subgroup variation
using the within subgroup variation.
This led him directly to X bar- R charts,
where the average of the within subgroup range
is used as an estimate of sigma.
Specifically, it led him away from the naive thinking
that we should use X bar plus or minus three times the sample standard deviation.
Don't estimate sigma using s for control charts.
Let me say that again.
Don't estimate sigma using the sample standard deviation.
Oh, and here it is in red.
Don't use the sample standard deviation to estimate sigma.
Why not?
The purpose of control charts is to detect a signal from your process,
detect a change in the process distribution.
Using the sample standard deviation
aggregates all the data, including a potential signal,
and this will inflate the estimate of sigma and you will miss signals.
That takes us back to the beginning.
If the purpose is to detect signals,
why would you make a chart that makes you miss signals?
Just don't do it.
But the default method
for estimating sigma in JMP's Levey-Jennings control chart
is to use the sample standard deviation.
The Levey-Jennings chart was developed for a very specific situation
in clinical chemistry,
one where an analytical method is validated using a gage study
with factors that are relevant to that process.
The data are collected according to a specified experimental design.
Here are some possible random and fixed factors in the experiment.
The variance components for the random factors
are calculated using a statistical model.
Then the total variability is calculated from the variance components
and used as the estimate of sigma for the Levey-Jennings control chart.
This chart has a specific purpose in clinical chemistry.
It can be used whenever you have an external estimate of sigma.
How did we get here where these charts are being used for situations
in industries other than the one I just described?
The history of SPC started about 100 years ago
when Walter Shewhart developed the first control chart.
He figured out to use the within or the short-term variation
using subgroups with multiple items.
But he couldn't figure out how to estimate
short-term variation for subgroups of size 1.
There is no within.
In 1950, Levey and Jennings published their paper,
and they did not use s to estimate sigma.
Their paper introduced control charts to clinical chemistry
and used Shewhart's Xbar- R chart with two replicates, so subgroup size 2.
Now, just after that, in the early 1950s,
Tippett discovered that you can use
the average moving range of individual values
as a short-term estimate of sigma.
It's not perfectly a within estimator, but it's short-term enough
that you're not aggregating over all the data.
Very quickly, ASTM and Western Electric published books
and standards using Tippett's X- MR chart.
But clinical chemistry didn't hear about these charts.
In 1952, Henry and Segalove published a paper using s
to estimate sigma using subgroups of size 1.
Don Wheeler indicates that
XM-R charts were not used very often until about 1980.
He introduced them to Deming,
and Wheeler and Deming taught thousands of clients and popularized the XM-R chart.
It's really interesting to me that they weren't popular back then
because in my experience, which started in the early '90s,
charts for subgroups of size 1 are much more popular than Xbar charts are.
All right, just as the XM-R charts were gaining popularity in manufacturing,
James Westgard published his Westgard rules,
and wrote his popular book, Basic QC Practices for clinical chemistry.
He calls the control charts, Levey-Jennings.
They were added to JMP in version 5 in 2003 by customer request.
Now, ever since Jordan and Byron and I have been working at JMP,
we've seen examples from all industries
of the misuse of the default Levey-Jennings chart.
But now you know better, and you will never use
the sample standard deviation
to estimate sigma for control charts ever again, right?
All right, I'm going to turn it over to Jordan,
and he'll show you in JMP the problems with misusing these charts.
Thanks, Di.
We created a simulation in JMP to show how
the Levey-Jennings chart, when misused, will definitely lead to missed signals.
You can download this.
It's a JMP add-in that's saved in the JMP C ommunity
with the other presentation materials.
Here we show two control charts on the same data set.
Here's the individual chart on top,
and the Levey-Jennings chart on the bottom.
Just a brief note, the individual's chart
is usually displayed with the moving range chart underneath it.
We're omitting that for visual simplicity here.
I'll also mention that the Levey-Jennings chart
is drawn in the way we're warning against.
That is, these are not historical limits,
these are limits calculated from the data in the chart itself.
The 30 data points that we are graphing here
are drawn randomly from a standard normal distribution.
That is, the mean is zero, standard deviation is one.
Whenever I click this new data button over here,
we'll get a new random sample.
Because we know the true population parameters,
we can tell how accurately these charts perform.
The true population sigma is one, and we're going to hope
that the sigma estimates from the two charts are close to that.
The data is stable, it's from a normal distribution.
Whenever we see a point outside the control limits,
it's a false positive here.
In a simulation of 5,000 data sets,
the individual's chart has a false positive about 7% of the time,
while the Levey-Jennings has one about 3.5 % of the times.
This small difference,
well, it's double, but this difference in the false positive rates,
we think is one of the reasons that folks have tended to prefer
the Levey-Jennings to the individual's chart sometimes.
But here's the problem.
What happens when we have some signals in this data?
We're going to talk with a couple of different kinds of signals.
Let's start with a shift.
A shift is an abrupt change in the mean of the process.
Let's start by looking at a shift of three standard deviations,
and I'm going to hit the new data a couple of times.
Remember, the purpose of a control chart
is to distinguish the signal from the noise and to detect signals.
As the signal gets larger, we should be more likely to detect it,
and the sigma estimates should remain close to the true value of one.
The individual's chart does exactly what we expect.
It usually detects the shift,
and the sigma estimate for the individual chart stays close to one.
The Levey-Jennings, not so much.
It conflates the signal with the noise.
The sigma estimate is inflated, and signals generally can't be detected.
How big does this shift have to be
before the Levey-Jennings chart is going to detect it?
Well, I'm just kidding.
It's a trick question.
The Levey-Jennings chart is never going to detect a shift in this scenario.
I can jack it up to 50 standard deviations,
and the larger the shift that we induce,
the more that sigma estimate is inflated and will never detect a shift.
Let's see, we did some simulation work
to show the differences between the performance
of the individual's chart that's in red and the Levey-Jennings chart in blue.
We're looking at a shift that occurs
after the 20th data point in the series of 30 data points.
That's what we're looking at in the demo I just showed.
In that situation, a shift after the 20th data point,
you can see the performance of both of these charts
as the shift gets larger.
Individual's chart does exactly what we hope it would do.
As the shift becomes larger,
it's easier to detect probability of having an alarm is greater.
The Levey-Jennings performs terribly as you saw,
and paradoxically, as the shift gets larger,
the probability of detecting it approaches zero.
Let's take a look at that here in the demo.
We can change the shift location, right?
The shift was after the 20th data point.
But you'll see that as the shift approaches the end of the series,
and we'll put it at 29, the last data point here.
When the shift occurs at the last data point,
both of the charts perform similarly.
In other words, if you are running your Levey-Jennings chart
after every additional data point, you'll probably detect most signals.
And this is to be expected.
If the shift occurs later, there's less opportunity for that signal
to contaminate the noise estimate,
and the sigma estimate is going to be accurate.
Here's what that looks like in the simulations.
As that shift gets later and later in the series,
the performance of the Levey-Jennings chart in blue,
approaches the performance of the individual's chart.
That's the story with shifts.
It's even more alarming with drifts.
I'm going to reduce the shift size to zero.
As we introduce drift into the data,
drift is a gradual linear trend
as the mean of the process moves up in a consistent way.
That's drift.
No matter how large the drift that I induce here,
it will never be detected by the Levey-Jennings chart.
Just can't be detected.
Okay.
N ext let's consider a batched process.
Here, we're going to simulate
batches with size six, five batches here in the data.
Some folks use the Levey-Jennings chart
as a way to avoid having an alarm that's due to expected batch variation.
Well, it's easy to see the problem with this approach.
When we overlay a shift or a drift on top of that batch effect,
the Levey-Jennings chart is still never going to detect it.
I'll add a shift too.
Yeah, the Levey-Jennings chart is insensitive, too insensitive,
and the individual's chart is too sensitive.
Neither of the charts is really great for this situation.
Di is going to talk about this in a few moments.
Before I turn it back to Di,
let me show you what
a Levey-Jennings chart done right looks like.
This is a demo that was generated by my colleague, Byron,
using data that come from the Westga rd website,
and let's just talk about maybe this top chart here.
We're showing a chart based on a data series.
We have 28 data points.
The mean is 198.75 in this sample.
The standard deviation is 5.9 in this sample.
The point is, we are in Phase II,
and as such, we have set the control limits
using a historical estimate of sigma.
This protects us against all of the problems that we were just discussing.
It's a stable estimate of sigma,
and it will not be contaminated by any potential signals in the data set.
Okay, that is all I have.
Di, back to you.
All right.
Thank you, Jordan, for that alarming demo.
When Jordan and I and Byron
thought that it was time that someone gave this talk,
we asked our JMP friends if they were seeing what we were seeing
from talking with customers,
and that is, misapplication of Levey-Jennings charts.
What we found was that these charts
were commonly used for specific situations.
The first one is what Jordan was talking about, that batch processing.
It's called short- run SPC, where you might have batches
that change frequently, and each batch has a new mean,
maybe even a new standard deviation.
Using the sample standard deviation
to estimate sigma will contain batch shifts
as well as within- batch noise.
Like you saw, you won't be able to detect shifts within a batch very easily.
Instead, you can use a chart that plots
the difference or the standardized difference
from the batch target or the batch mean.
The second situation is autocorrelated processes,
and this is where observations taken close together
are more similar than observations taken further apart in time.
This behavior is often seen when measurements are taken very frequently
in a continuous process.
Using the sample standard deviation,
again, includes both noise and known process drift
due to the auto correlation in the estimate of sigma.
There are a few ways to deal with autocorrelated data,
including reducing the sampling frequency if you can,
or using a different type of chart if you can't.
Charts like CUSUM and EWMA can be adapted for autocorrelation,
or you could try to model the autocorrelation
and then use control charts on the residuals from the model.
The residuals also contain information
about process shifts, and they should be uncorrelated.
F inally, we've heard a lot of people say that using the sample standard deviation
is useful because it gives wide limits that are able to detect huge shifts,
but not too wide to detect small and moderate shifts.
In that case, we recommend that you just make up some limits
and don't advertise that you're using control charts
because control charts don't ever use
the sample standard deviation to estimate sigma.
Remember that West gard's wonderful book
is called Basic QC Practices , not Basic SPC Practices.
Feynman's wonderful quote is applicable here.
We've also heard some arguments about Levey-Jennings charts.
They are more forgiving than Shewhart's charts.
Of course, they are.
I like this one.
The range charts were optimized for hand calculation,
and we've got computers.
Why aren't we calculating standard deviation?
As we've seen, it's not range versus standard deviation,
it's which standard deviation?
You should always choose within variability
when you calculate three-sigma limits.
Aggregate over noise, not signal or potential signal.
Why use an estimate of sigma when we can just calculate sigma?
Oh, this one hurts me. This comes from a terminology issue.
We may say three- sigma limits,
but we don't know sigma and we have to estimate sigma from data.
This is the way.
I inherited this system, and my boss says I have to do it this way.
Well, can you find a new boss?
Really, this is my Oprah moment.
Maya Angelou said,
"In the past, you did the best with what you had,
and now that you know better, you will do better."
I hope that you can educate others to do better as well.
Thank you for listening.
I've got some references for you here at the end of the presentation.
We'd like to leave you with two thoughts.
First, don't use the Levey-Jennings charts
as they have been defined in the modern world.
The purpose of a control chart is to detect process changes,
and those changes are found by comparing signal to noise.
Use of the sample standard deviation
to estimate sigma inflates that noise and it will obscure any signals.
The second thought is that when you are in Phase I, use XM-R charts,
individual moving range charts instead of Levey-Jennings.
When you move to Phase II
for ongoing process control, fix those limits.
A sign of a stable process
is that the sigma estimate from using the average moving range
is similar to the sigma estimate from Levey-Jennings.
