Sequential sampling is like a relationship. After a while, we can either accept it, reject it – or keep trying. Compared to its stodgy cousin, fixed size sampling, we are nimbly making a decision every step of the way. This method has a very big advantage over the fixed size approach: it often requires far fewer parts, yet provides the same statistical power. Fewer parts means it saves time, money, and most importantly, everyone's afternoon. 

The method presented here was originally developed by Wald (1945) and is called the Sequential Probability Ratio Test (SPRT). We start out with an hypothesis of what we believe is the underlying defect rate of our population, p. Parts are sampled and tested one at a time and are determined to either pass or fail. After each measurement, we compare the observed %defect in our sample to decision thresholds calculated from SPRT theory and make a decision: accept our assumption about p, reject it, or keep on testing.It's a little like leaving a presentation after a few slides and saying, "OK, got it." 

The operating characteristic (OC) curve for fixed sample sizes is compared to that from SPRT for various scenarios. A sequential sampling add-in available on the JMP Marketplace is used to facilitate comparisons between the two techniques. In this contest, SPRT can regularly best its cousin by offering improved efficiency and adaptability.

 

 

Hello, thank you very much. My name is Charlie Whitman. I'm a JMP systems engineer. Today, we're going to be learning how we can save time and money using sequential sampling. To start off, I just want to show where we're headed. Here is a roadmap. The first thing is, I'm going to be talking about the OC curve or operating characteristic curve and how it works. From this, we're going to see the benefit of sequential sampling. We're also going to need to delve into some terms like producer's risk and consumer's risk, AQL and RQL; I will define those. Lastly, I'm going to give an example using the sequential sampling add-in. These two graphs really summarize the advantage of using sequential sampling versus the fixed sample approach.

Before I go into too much detail, I want to point out a few things. First, I'm only going to be talking about attribute testing—that is, the data is pass/fail. I'm not using any continuous responses here. The two different types of approaches I'm going to be using are a fixed sample size approach and the sequential approach. For the fixed approach, I take a predetermined number of parts and I test all the parts in that sample.

Let's suppose I have a lot of material. That lot has 10,000 parts. I'm going to pull a sample of 100. For a fixed approach, I would end up testing all 100 of those parts. Then, out of that sample, if there are c or fewer defects, I will ship the entire lot—all 10,000 parts—onto the customer. If I get more than c defects, I will reject the entire lot. For the sequential test, I'm only testing one part at a time. In this case, I'm only going to test one part at a time, and I have a stopping rule. I check my results after each trial, and I check it versus the stopping rule, and I make a decision. Either I can accept the entire lot, I need to reject the entire lot, or I need to keep on testing. Also, just to make you aware, there's a term of art called lot quality. Lot quality refers to the underlying defect rate that's sometimes called p.

Let's talk about this top graph here. This is the OC curve. Let's see how this works. On the y-axis, I have something called the probability of acceptance. You can think of this as the probability of shipping your lot to the customer, and the x-axis has the lot quality or the defect rate. If the defect rate happens to be high—say 6%—I can see where I'm going to intersect the OC curve; I go over, and I see that the probability of acceptance is pretty low, say, under 10%, and that's good. If the defect rate is high, I don't want to ship that lot to the customer. I want the probability of shipping that lot to the customer to be low, so this is good. If, on the other hand, say the lot quality is pretty low—like under a percent—well, that's good; then I want to ship that lot, especially the producer wants to ship all the lots that they can. Here my probability of acceptance or shipping is going to be pretty high, like say 95%.

Notice also that there are two curves here. I have a blue line and a purple line. The blue line is for the fixed approach, and the purple line is for the sequential approach. You can see that these lie pretty much right on top of one another, so there's no practical difference between the OC curves for these two approaches.

Now, the bottom graph gives me some sample size information. For a fixed plan, I'm testing all the parts. In this case, my sample size is 150; so for a fixed plan, I'm going to test all 150 parts. That's going to look like this. It doesn't matter what the underlying lot quality is; I'm always testing 150 parts. But for a sequential plan, it's going to be less than that. Now, since it's sequential, that means that we don't exactly know what the sample size is going to be. It's a random walk, but we can get an idea. We can see what the expectation is on the average of what the sample size is going to be, and that's this purple line here. If my underlying defect rate is 8%, then from a sequential plan, I would expect on the average, it might take me, say, 30 parts to learn that I need to reject this lot. On the other hand, if the defect rate is very low, then it might only take me, say, 75 parts. Again, in both instances, you can see I'm well below this 150-part mark. This is really where the money is: I'm using a sequential plan, and I can greatly reduce the sample size necessary to test my lot.

Also, the sequential plan and the fixed plan have the same OC curve. They are both equally good at distinguishing between a good and a bad lot. That means I haven't given up any power. I'm able to reduce my sample size without giving up any power or capability in distinguishing between good and bad lot. That's really where the money is. Let's talk about fixed sampling. Again, I'm going to take a fixed sample from a batch of parts. If there are a few defects, then I'm going to accept the entire lot and ship it onto the customer. If there are too many defects in my sample, I'm going to reject the lot and probably scrap it out.

Of course, no test is ever perfect. It's possible to make a mistake. It's possible from sampling that I could either accept a bad lot or reject a good lot. If I mistakenly reject a good lot, that's something that the producer does not want. The producer wants to ship all the lots that they can. They don't want to have a perfectly good lot and then not ship that to the customer. The risk of that happening is called alpha.

On the other hand, the consumer does not want to be shipped a bad lot. If we make a mistake and ship them a bad lot, then that means they're taking a risk, and that's the consumer's risk—that's the term beta. Usually, we want alpha and beta to be low. We want alpha to be maybe 5 or 10%; you want beta to be maybe 10 or 20%. I'm going to show graphically what this might look like. Let me suppose I have a production process, and the underlying defect rate is 10%, and that's represented here by these red X's.

What we're going to do is we're going to take a sample. This is not really a random sample, but it's good enough for our purposes. I take my random sample, and I count the number of defects. Based on that defect count, I'm then going to make a decision. Of course, now, let's suppose I have a sampling plan, and my sampling plan is I pick 100 parts. If I take 10 or fewer defects out of that sample of 100, then I can go ahead and ship the entire lot to the customer.

Here I'm taking my sample, I count the number of defects, and I count 16 defects. 16 is more than 10, so I would reject this entire lot. But this was rotten luck because if I had taken a different sample, I might have gotten—or very likely, I would have gotten—less than 10 defects, so I would have ended up shipping, so this is a mistake. This is the producer's risk: that they would incorrectly reject a perfectly good batch. On the other hand, suppose I have a different sampling plan, where again, my sample size is 100, but I'm going to use c equals 4. If I get four or fewer defects, I will accept the entire lot. If I get more than four, I reject it. I pull my sample, I count the number of defects, I only have three defects, so I say, "Great, I can go ahead and ship this to the customer." Three is less than four. But if I had taken a different sample, very likely I would have gotten more than four defects and I would not have shipped. Again, I'm in rotten luck. I got lucky that I had very few defects in my sample, and I mistakenly ship a bad lot.

Let's talk a little bit more about the OC curve and how it works. Again, I have my probability of acceptance here and my lot quality here. The first term we need to be aware of is called the AQL; this is the acceptable quality limit. This is the highest average defect rate of the process that is acceptable most of the time. The value of the AQL is a defect rate. This is a probability of getting a defect. This is something that the producer is probably going to have a pretty strong opinion on. They want to choose an AQL where, according to the sampling plan, most of the time, they're going to be able to ship a good lot. The AQL is probably going to be roughly where their process is now running—the defect rate of their process. There's also something called the rejectable quality limit, or RQL. This is the highest defect rate that the customer will tolerate. At the RQL, this is where the customer starts to scream; the defect rate is too high. At the RQL, we can pick this off and see that the chance of shipping a lot with the defect rate at the RQL, in this case, is 10%. These values for the AQL and RQL are probably going to be subject to negotiation between the producer and the consumer.

We can also talk about the distance delta between the AQL and the RQL. In fact, that's just the difference. If I tell you what the AQL is, and I tell you what delta is, then you know what the RQL is; it's just the sum. Delta is also called the margin of indifference, because in here, it's a statistical no man's land. Neither party is getting full protection from the sampling plan. But also we're indifferent: if the lot quality happens to be in this region, then neither party is particularly upset. As an aside here, it turns out that the OC curve can be derived from a series of hypothesis tests. Maybe you took a statistics class in college, remember things like the null hypothesis and the alternative hypothesis. This is a practical application of that idea. The null hypothesis is just the status quo, and we're going to assume here that the defect rate is at the AQL—that's the status quo. This is also called p0, and the zero refers back to the zero from H0. If I have too many defects, I'm going to reject H0 and reject the lot. If I have a few defects, then I'm going to accept H0, accept the lot, and ship it onto the customer.

The alternative hypothesis is the counterclaim that what do we do if the defect rate is actually at the RQL? That's called sometimes p sub A, again, for alternative. I've also included a link here; if you're interested, there's a paper which describes how you can get the OC curve from a series of hypothesis tests. Let's talk about the types of trade-offs then we have to make and how we can use OC curves to make trade-offs between the consumer and the producer. Here are three OC curves, and these all have the same sample size with different acceptance numbers. You can see that as my value of c decreases, then these curves start to hug the axes. Let's suppose we're in the middle of negotiation between the consumer and the producer, and some numbers are thrown out. Let's say, "Okay, the producer says they'd like an AQL of 1%, and the consumer says they want an RQL of 2%." What does this imply? Let's look at the sampling plan for c equals 1 here first. I'm going to pick off the value for the probability of acceptance at the RQL—so 2%—I go over. That's probably right around 10% or so.

The consumer might look at that and say, "That sounds good to me. I'm fine with that." But what about the producer? At the AQL of 1%, I see where that intersection is, it's at 0.4. This is my alpha. My alpha is like 60%, and very likely at this point, the producer is going to say, "That's way too high risk; I can't tolerate that. We have to choose something different." What happens if we use, say, c equals 3 instead of c equals 1? What happens then? At the AQL, I go up, I see where I intersect the green curve, I move over, and my alpha is considerably reduced. Maybe not perfect, but it's much better than it was before. The producer likes this situation much better. How about the consumer? If the RQL is 2%, we go over, now my risk of shipping a bad lot is, say, 40% or more. That's really bad. Probably the consumer is going to say, "No, I don't sign up for a 40% risk." Now, what can we do? One thing we can do is just change the AQL and the RQL: let me move them farther apart. I'm going to use the same sampling plan for c equals 3.

At this new AQL, my alpha—this distance up here—is pretty small, like say 5%, that's good. If we choose an RQL of 3%, now my beta is less than 20%, and maybe the consumer would be satisfied for that. This is one possible manifestation of our parameters—unless, of course, maybe both parties are really married to the previous values for AQL and RQL. If that's the case, what can I do? I'm going to have to change my sampling plan. I'm going to have to change the number of parts I test. I'm probably going to have to change my acceptance number. At 1% and 2%, we can see the beta here is like 20% and the AQL is maybe 10%. Here, this would probably be acceptable to both parties. Anyway, you're getting the idea of the trade-offs that you have to make between alpha and beta, and AQL and RQL, and sample size. That's the big takeaway here. This table just summarizes these terms for us: our defect rate or lot quality, p; the values of AQL and RQL are defect rates; the values of alpha and beta are probabilities. The probability of acceptance at the AQL is 1 minus alpha.

The probability of acceptance at the RQL is beta, and then delta is just the difference between the two. Now, let's talk about sequential sampling. The method I'm going to describe is called the SPRT, or sequential probability ratio test. This was developed by Abraham Wald during World War II. Again, what we do is we adopt a stopping rule. In this case, I'm only going to be testing one part at a time. Once the stopping criterion is met, I stop testing, and I'm either going to accept or reject the lot. If the criterion is not met, that means I have to keep on testing. Again, since this is something of a random walk, the sample size will be pretty random, but we can estimate what we expect the sample size to be. Here, again, this is where the money is: sequential sampling will often result in a much smaller sample size than a fixed sample size approach for the same consumer's and producer's risks. You're not giving up any power in distinguishing between a good and a bad lot, but you are able to reduce your sample size. Let's see how this might work; let's do this graphically.

I have a defect count on my y-axis, and I have a trial number on the x-axis. See, these regions are colored orange and blue. In the orange region, if I count too many defects, I'm going to reject the lot. If I get too few defects, I'm going to accept the lot. These regions are defined by the math; this is where the math lives. In this particular case, on trial number one, I got one defect. On trial number two, I got another defect, so my defect count goes up to two. On the third trial, I didn't get a defect, so it's flat. You can see, again, I'm here in the white region. In the white region, that means I have to keep on testing, and this curve is going to wander up until, finally, I cross into the orange region. In here, I'm going to reject the lot, and that took me nine trials to figure that out. Instead of a defect count here, I'm going to use a defect rate. This is just the same data, but divided by the sample size. Since I'm dividing through by N, those straight lines become curves, so it's a curved region now, but it's the same result.

Again, it took me nine trials to figure this out. Of course, it's also possible that I'm going to have very few defects in my sample. In this case, it took—on my seven trials, or actually on my eighth trial, I got a defect, and after that, I got no more defects. I cross into the blue region after, say, 22 trials. I can also do the same thing by using a defect rate by dividing through by the current sample size. Again, it takes 22 trials to figure out that I need to—in this case—accept the lot. Let me go here. All right. When would I want to use a sequential versus a fixed test? It sounds like I'd probably want to use a sequential test all the time. I have a couple of figures here. This is just symbolic, showing how things might work. If I'm using a sequential test, I might take my part and put it in a test fixture and then see if it passes or fails. Whatever happens, I put that part out, I put another part in, I test again, and I keep on testing until my criterion are met.

For a fixed approach, for my sample size, very likely I'm going to want to use a test rack. I'm going to put a whole bunch of parts in a test rack and note how many pass and fail—the advantage here being that I'm able to test all these parts simultaneously. When would I want to use a sequential approach? If each part is really expensive, or setting up and running a test is really expensive, I probably want to minimize my sample size. In that case, I would want to use a sequential test. What if the test is destructive, and I want to minimize the number of parts I destroy? Every part that I don't destroy in my test, I can ship to the customer. In that case, again, I might want to use a sequential test. If the quality tends to be either very good or very poor—if you remember here from the previous plot, if I have a high defect rate or a very low defect rate—this is where the sequential approach really shines. I can figure out things very, very quickly using a sequential approach. But there are still instances where I might want to use a fixed approach.

What if I have a really large setup time, but after I've set up one part, the marginal cost of just adding a bunch of more parts is actually pretty low? In that case, it might be reasonable just to go ahead and test a whole bunch of parts, especially if each part is inexpensive. What if I have a very long test time? What if this takes 3 days to test, and then the next part takes 4 days to test? That's a week. It might make sense, then, to just put a whole bunch of parts on test all the same time and run them in parallel. That way I can save myself a lot of time. Also, it might be good just for logistic simplicity: it might be easier just to load up a test rack, put all these parts on test and then walk away and go do something else. If I have a test that I have to test one at a time and I have to be there, that can be very time-consuming; that means there's other stuff I'm not getting done. Maybe I don't want to do that. Maybe I want to test parts in parallel.

There are reasons why you might want to use a fixed approach. Let's look at a couple of examples here, but before I do, I want to show you the JMP Marketplace. This is the JMP Marketplace, and this is where you can find the sequential sampling add-in. It's free to download, so I hope you do; I hope you find it helpful. This is what the sequential sampling plan looks like. I've also opened up a data table. Excuse me. I have two tabs here: I have a sequential sampling tab—this is for data analysis; this is where I'm going to input my data or maybe run a simulation. The other tab is for generating an OC curve for both a fixed approach and the sequential sampling approach. Let's say we want to input some data. I'm going to input this data table—it says DT_accept—so that lets you know that I'm going to end up accepting this batch. How do we work the add-in? First thing we have to do is input the data. I'm going to use this, click on the radio button here, and click okay.

Now, I have a column list of data tables that are available. I'm going to pick the one, DT_accept, and I choose this data table. Now I have a column list, and I have to assign the column lists roles—the column's roles. Here, the first column is called IDX. This is basically a trial number: 1, 2, 3, et cetera. The second column, it says fail equals 1. This means that if I get a—this is an indicator column, so I have 1 if I get a failure, 0 otherwise. I want to tell JMP this: I want the trial number to be on the x-axis, so I choose IDX, and I go to click on X. Fail equals 1 is my response; that goes on the y-axis. Now I have to tell the add-in what values I want to input for alpha and beta, et cetera. I've already worked that out. Let's see what we got.

Alpha and beta is 0.1 and 0.2. These have already been set to those values, so great. We can set p0—p0, again, is the same as the AQL. That's going to be, in this case, 0.1. I want an RQL of 0.15; that means I want a delta of 0.05. My margin of indifference is 0.05. Remember, RQL is equal to AQL plus delta. Here I'm going to put in a delta of 0.05. I'm going to click OK. I get a little graph. We could ignore this output. On the Y-axis here, it says proportion success. You can think of it this way: I have successfully found a defect. This is basically a defect rate. On the x-axis is my trial. I'm just seeing what my response is; I'm doing the calculation until, finally, I've crossed the blue line. I did that after trial number 43, and my underlying defect rate there was about, say, 5% or so. It took me 43 trials to figure out, in this case, I should accept H0. That means I accept the lot; the lot is good enough to ship onto the customer.

I can also do one where the defect rate is higher. I'm going to choose that data table; I click here. Everything else stays the same. I'm just going to click OK, and I can get rid of this. Now, here we can see that my defect rate is wandering up until, finally, I cross the orange. Orange line says I need to reject H naught; I reject the lot. I was able to do that in 32 trials, and my underlying defect rate was a little under 30%, 40%.

That's how that works in terms of inputting the data. But we can also do a simulation: what would that look like? Instead of inputting the data, I'm going to simulate the data. I choose this radio button and I click OK. Now all this user input goes away, and I have to input values again. I have my alpha of 0.1, my beta of 0.2—p0 is the same as my AQL; I'm going to make that 0.1. Delta is going to be 0.05, and I have to choose a value for the simulation: what is the true underlying defect rate? I'm going to choose 0.1. You can see that my true underlying defect rate agrees with my AQL. If that's the case, I'm going to expect that I'm going to end up rejecting the lot, or rejecting H0 about 10% of the time—that's equal to alpha.

Now, I can do this just, say, I'm going to run one simulation; I'm going to click OK. In this particular case, I have accepted the lot after 35 trials. Really, what I ought to do is, if I want to run a simulation and see how it behaves, I should do a whole lot of iterations, but that takes a little bit of time. I've already done that, so I'm just going to show you the result.

Here's the result of my simulation. By the way, I did 500 iterations here, and it took about 100 trials or so for each one of those iterations to either accept or reject the lot. That implies about 50,000 rows. Here, you can see I have 55,000 rows. Let's look at the distribution of the results. Here I have H0, or I have accepted the lot 92% of the time. I have rejected it about 8% of the time. That's close to our alpha; our alpha is 10%, and we're right around that mark. That's pretty good. How long did it take for me to figure this out? We can look over here at the distribution of the number of trials. On the average, this took 110, but this is a skewed distribution, so maybe I should use the median—here, is 83. Roughly speaking, I'd say it took about 100 trials, in this case to—most of the time, I'm going to be accepting the lot and shipping it on to the customer. Remember that number: it took about 100 there when the underlying defect rate I chose was 10%.

Now, let's look at a different example where I run a simulation. For this simulation, I'm going to assume that the true underlying defect rate—my RQL—is 15% rather than 10%, and that's going to agree with my RQL. My RQL is 15% and the underlying defect rate is also 15%. I'm right on the border of where the customer would start to scream. Here, in that particular case, I would expect to accept the lot at a rate of beta, which in this case is 0.2. From the simulation, I would expect about 20% of the time I end up accepting the lot, and that's what happened. I accept the lot here, in this case, about 21% of the time. That worked pretty well. How many trials did it take for me to figure that out? On the average, it took 125; the median was 99, so, again, ballpark-y, around 100 trials, and I would usually end up rejecting the lot.

Okay, so let me see how we can use this information for the OC curve. I'm going to go to OC curve here, and I'm going to look here; I've already worked this out ahead of time. I'm going to put in values for alpha and beta: I'm going to put in an alpha value of 0.1, a beta value of 0.2, and I have to tell it what my acceptance number is and what my sample size is. Again, I worked this out: I'm going to put in 21 for my acceptance number, and 170 for N. I click OK, and it returns for me the AQL and the RQL. The AQL here is right around 10% and the RQL is right around 15%. That's what I had here before. Remember, the AQL here was 10% and the RQL was 15%.

Here is the OC curve for that situation. Let's look at—remember, the lot quality of 10%—what was the sample size required from the simulation? Right around 100. The purple curve is telling me that the sequential plan is able to figure this out: if the defect rate is 10%, it took me—on the average, I would expect me to take 100 trials to figure that out, and that is better than 170. Also, if the defect rate is 15%, again, it took right around 100 trials to figure that out. Again, I'm doing better than I would if I used a fixed approach.

Again, the purple and the blue lines here are overlapping, and we can see that we're getting every bit as good a response—or every bit as good a performance—from the sequential plan as the fixed plan. The OC curves are the same. Just to let you know, down here, I have a swizzle on this approach, except here you're able to solve for either N or c if you want to. Let's suppose that I know what N is: I know my test rack can handle 170; I'll put in 170. I'm going to use the same alpha: 0.1. I calculated AQL here as 9.7%, 0.097. My beta is 0.2; I'm going to put in 0.2 here. It's going to calculate the RQL for me. I click OK. It calculates the RQL, and this RQL agrees—I've added the RQL here. What was my acceptance number? My acceptance number was 21, and that value also agrees with the 21 up here. That's just another way of doing the same problem, solving the same problem.

Okay, to summarize: I went over and showed how the operating characteristic curve works, so you can have an idea how to use them. Basically, the idea behind them is that you want a way to ensure that good lots are shipped, and bad lots are caught most of the time. We went into some detail about the trade-offs to be had between alpha and beta, AQL, RQL, and the sample size and the acceptance number. Again, the important thing here is that there is a large expected savings using sequential sampling, and you're not sacrificing either the producer's or the consumer's risk. The sequential approach is able to provide every bit as good an OC curve—is every bit as good at distinguishing between good and bad lots for a much smaller sample size. That's my talk. Thank you very much.

0 Comments
Presented At Discovery Summit 2026

Presenter

Skill level

Intermediate
  • Beginner
  • Intermediate
  • Advanced

Files

Published on ‎07-16-2026 11:09 AM by Community Manager Community Manager | Updated on ‎10-30-2026 03:34 PM

Sequential sampling is like a relationship. After a while, we can either accept it, reject it – or keep trying. Compared to its stodgy cousin, fixed size sampling, we are nimbly making a decision every step of the way. This method has a very big advantage over the fixed size approach: it often requires far fewer parts, yet provides the same statistical power. Fewer parts means it saves time, money, and most importantly, everyone's afternoon. 

The method presented here was originally developed by Wald (1945) and is called the Sequential Probability Ratio Test (SPRT). We start out with an hypothesis of what we believe is the underlying defect rate of our population, p. Parts are sampled and tested one at a time and are determined to either pass or fail. After each measurement, we compare the observed %defect in our sample to decision thresholds calculated from SPRT theory and make a decision: accept our assumption about p, reject it, or keep on testing.It's a little like leaving a presentation after a few slides and saying, "OK, got it." 

The operating characteristic (OC) curve for fixed sample sizes is compared to that from SPRT for various scenarios. A sequential sampling add-in available on the JMP Marketplace is used to facilitate comparisons between the two techniques. In this contest, SPRT can regularly best its cousin by offering improved efficiency and adaptability.

 

 

Hello, thank you very much. My name is Charlie Whitman. I'm a JMP systems engineer. Today, we're going to be learning how we can save time and money using sequential sampling. To start off, I just want to show where we're headed. Here is a roadmap. The first thing is, I'm going to be talking about the OC curve or operating characteristic curve and how it works. From this, we're going to see the benefit of sequential sampling. We're also going to need to delve into some terms like producer's risk and consumer's risk, AQL and RQL; I will define those. Lastly, I'm going to give an example using the sequential sampling add-in. These two graphs really summarize the advantage of using sequential sampling versus the fixed sample approach.

Before I go into too much detail, I want to point out a few things. First, I'm only going to be talking about attribute testing—that is, the data is pass/fail. I'm not using any continuous responses here. The two different types of approaches I'm going to be using are a fixed sample size approach and the sequential approach. For the fixed approach, I take a predetermined number of parts and I test all the parts in that sample.

Let's suppose I have a lot of material. That lot has 10,000 parts. I'm going to pull a sample of 100. For a fixed approach, I would end up testing all 100 of those parts. Then, out of that sample, if there are c or fewer defects, I will ship the entire lot—all 10,000 parts—onto the customer. If I get more than c defects, I will reject the entire lot. For the sequential test, I'm only testing one part at a time. In this case, I'm only going to test one part at a time, and I have a stopping rule. I check my results after each trial, and I check it versus the stopping rule, and I make a decision. Either I can accept the entire lot, I need to reject the entire lot, or I need to keep on testing. Also, just to make you aware, there's a term of art called lot quality. Lot quality refers to the underlying defect rate that's sometimes called p.

Let's talk about this top graph here. This is the OC curve. Let's see how this works. On the y-axis, I have something called the probability of acceptance. You can think of this as the probability of shipping your lot to the customer, and the x-axis has the lot quality or the defect rate. If the defect rate happens to be high—say 6%—I can see where I'm going to intersect the OC curve; I go over, and I see that the probability of acceptance is pretty low, say, under 10%, and that's good. If the defect rate is high, I don't want to ship that lot to the customer. I want the probability of shipping that lot to the customer to be low, so this is good. If, on the other hand, say the lot quality is pretty low—like under a percent—well, that's good; then I want to ship that lot, especially the producer wants to ship all the lots that they can. Here my probability of acceptance or shipping is going to be pretty high, like say 95%.

Notice also that there are two curves here. I have a blue line and a purple line. The blue line is for the fixed approach, and the purple line is for the sequential approach. You can see that these lie pretty much right on top of one another, so there's no practical difference between the OC curves for these two approaches.

Now, the bottom graph gives me some sample size information. For a fixed plan, I'm testing all the parts. In this case, my sample size is 150; so for a fixed plan, I'm going to test all 150 parts. That's going to look like this. It doesn't matter what the underlying lot quality is; I'm always testing 150 parts. But for a sequential plan, it's going to be less than that. Now, since it's sequential, that means that we don't exactly know what the sample size is going to be. It's a random walk, but we can get an idea. We can see what the expectation is on the average of what the sample size is going to be, and that's this purple line here. If my underlying defect rate is 8%, then from a sequential plan, I would expect on the average, it might take me, say, 30 parts to learn that I need to reject this lot. On the other hand, if the defect rate is very low, then it might only take me, say, 75 parts. Again, in both instances, you can see I'm well below this 150-part mark. This is really where the money is: I'm using a sequential plan, and I can greatly reduce the sample size necessary to test my lot.

Also, the sequential plan and the fixed plan have the same OC curve. They are both equally good at distinguishing between a good and a bad lot. That means I haven't given up any power. I'm able to reduce my sample size without giving up any power or capability in distinguishing between good and bad lot. That's really where the money is. Let's talk about fixed sampling. Again, I'm going to take a fixed sample from a batch of parts. If there are a few defects, then I'm going to accept the entire lot and ship it onto the customer. If there are too many defects in my sample, I'm going to reject the lot and probably scrap it out.

Of course, no test is ever perfect. It's possible to make a mistake. It's possible from sampling that I could either accept a bad lot or reject a good lot. If I mistakenly reject a good lot, that's something that the producer does not want. The producer wants to ship all the lots that they can. They don't want to have a perfectly good lot and then not ship that to the customer. The risk of that happening is called alpha.

On the other hand, the consumer does not want to be shipped a bad lot. If we make a mistake and ship them a bad lot, then that means they're taking a risk, and that's the consumer's risk—that's the term beta. Usually, we want alpha and beta to be low. We want alpha to be maybe 5 or 10%; you want beta to be maybe 10 or 20%. I'm going to show graphically what this might look like. Let me suppose I have a production process, and the underlying defect rate is 10%, and that's represented here by these red X's.

What we're going to do is we're going to take a sample. This is not really a random sample, but it's good enough for our purposes. I take my random sample, and I count the number of defects. Based on that defect count, I'm then going to make a decision. Of course, now, let's suppose I have a sampling plan, and my sampling plan is I pick 100 parts. If I take 10 or fewer defects out of that sample of 100, then I can go ahead and ship the entire lot to the customer.

Here I'm taking my sample, I count the number of defects, and I count 16 defects. 16 is more than 10, so I would reject this entire lot. But this was rotten luck because if I had taken a different sample, I might have gotten—or very likely, I would have gotten—less than 10 defects, so I would have ended up shipping, so this is a mistake. This is the producer's risk: that they would incorrectly reject a perfectly good batch. On the other hand, suppose I have a different sampling plan, where again, my sample size is 100, but I'm going to use c equals 4. If I get four or fewer defects, I will accept the entire lot. If I get more than four, I reject it. I pull my sample, I count the number of defects, I only have three defects, so I say, "Great, I can go ahead and ship this to the customer." Three is less than four. But if I had taken a different sample, very likely I would have gotten more than four defects and I would not have shipped. Again, I'm in rotten luck. I got lucky that I had very few defects in my sample, and I mistakenly ship a bad lot.

Let's talk a little bit more about the OC curve and how it works. Again, I have my probability of acceptance here and my lot quality here. The first term we need to be aware of is called the AQL; this is the acceptable quality limit. This is the highest average defect rate of the process that is acceptable most of the time. The value of the AQL is a defect rate. This is a probability of getting a defect. This is something that the producer is probably going to have a pretty strong opinion on. They want to choose an AQL where, according to the sampling plan, most of the time, they're going to be able to ship a good lot. The AQL is probably going to be roughly where their process is now running—the defect rate of their process. There's also something called the rejectable quality limit, or RQL. This is the highest defect rate that the customer will tolerate. At the RQL, this is where the customer starts to scream; the defect rate is too high. At the RQL, we can pick this off and see that the chance of shipping a lot with the defect rate at the RQL, in this case, is 10%. These values for the AQL and RQL are probably going to be subject to negotiation between the producer and the consumer.

We can also talk about the distance delta between the AQL and the RQL. In fact, that's just the difference. If I tell you what the AQL is, and I tell you what delta is, then you know what the RQL is; it's just the sum. Delta is also called the margin of indifference, because in here, it's a statistical no man's land. Neither party is getting full protection from the sampling plan. But also we're indifferent: if the lot quality happens to be in this region, then neither party is particularly upset. As an aside here, it turns out that the OC curve can be derived from a series of hypothesis tests. Maybe you took a statistics class in college, remember things like the null hypothesis and the alternative hypothesis. This is a practical application of that idea. The null hypothesis is just the status quo, and we're going to assume here that the defect rate is at the AQL—that's the status quo. This is also called p0, and the zero refers back to the zero from H0. If I have too many defects, I'm going to reject H0 and reject the lot. If I have a few defects, then I'm going to accept H0, accept the lot, and ship it onto the customer.

The alternative hypothesis is the counterclaim that what do we do if the defect rate is actually at the RQL? That's called sometimes p sub A, again, for alternative. I've also included a link here; if you're interested, there's a paper which describes how you can get the OC curve from a series of hypothesis tests. Let's talk about the types of trade-offs then we have to make and how we can use OC curves to make trade-offs between the consumer and the producer. Here are three OC curves, and these all have the same sample size with different acceptance numbers. You can see that as my value of c decreases, then these curves start to hug the axes. Let's suppose we're in the middle of negotiation between the consumer and the producer, and some numbers are thrown out. Let's say, "Okay, the producer says they'd like an AQL of 1%, and the consumer says they want an RQL of 2%." What does this imply? Let's look at the sampling plan for c equals 1 here first. I'm going to pick off the value for the probability of acceptance at the RQL—so 2%—I go over. That's probably right around 10% or so.

The consumer might look at that and say, "That sounds good to me. I'm fine with that." But what about the producer? At the AQL of 1%, I see where that intersection is, it's at 0.4. This is my alpha. My alpha is like 60%, and very likely at this point, the producer is going to say, "That's way too high risk; I can't tolerate that. We have to choose something different." What happens if we use, say, c equals 3 instead of c equals 1? What happens then? At the AQL, I go up, I see where I intersect the green curve, I move over, and my alpha is considerably reduced. Maybe not perfect, but it's much better than it was before. The producer likes this situation much better. How about the consumer? If the RQL is 2%, we go over, now my risk of shipping a bad lot is, say, 40% or more. That's really bad. Probably the consumer is going to say, "No, I don't sign up for a 40% risk." Now, what can we do? One thing we can do is just change the AQL and the RQL: let me move them farther apart. I'm going to use the same sampling plan for c equals 3.

At this new AQL, my alpha—this distance up here—is pretty small, like say 5%, that's good. If we choose an RQL of 3%, now my beta is less than 20%, and maybe the consumer would be satisfied for that. This is one possible manifestation of our parameters—unless, of course, maybe both parties are really married to the previous values for AQL and RQL. If that's the case, what can I do? I'm going to have to change my sampling plan. I'm going to have to change the number of parts I test. I'm probably going to have to change my acceptance number. At 1% and 2%, we can see the beta here is like 20% and the AQL is maybe 10%. Here, this would probably be acceptable to both parties. Anyway, you're getting the idea of the trade-offs that you have to make between alpha and beta, and AQL and RQL, and sample size. That's the big takeaway here. This table just summarizes these terms for us: our defect rate or lot quality, p; the values of AQL and RQL are defect rates; the values of alpha and beta are probabilities. The probability of acceptance at the AQL is 1 minus alpha.

The probability of acceptance at the RQL is beta, and then delta is just the difference between the two. Now, let's talk about sequential sampling. The method I'm going to describe is called the SPRT, or sequential probability ratio test. This was developed by Abraham Wald during World War II. Again, what we do is we adopt a stopping rule. In this case, I'm only going to be testing one part at a time. Once the stopping criterion is met, I stop testing, and I'm either going to accept or reject the lot. If the criterion is not met, that means I have to keep on testing. Again, since this is something of a random walk, the sample size will be pretty random, but we can estimate what we expect the sample size to be. Here, again, this is where the money is: sequential sampling will often result in a much smaller sample size than a fixed sample size approach for the same consumer's and producer's risks. You're not giving up any power in distinguishing between a good and a bad lot, but you are able to reduce your sample size. Let's see how this might work; let's do this graphically.

I have a defect count on my y-axis, and I have a trial number on the x-axis. See, these regions are colored orange and blue. In the orange region, if I count too many defects, I'm going to reject the lot. If I get too few defects, I'm going to accept the lot. These regions are defined by the math; this is where the math lives. In this particular case, on trial number one, I got one defect. On trial number two, I got another defect, so my defect count goes up to two. On the third trial, I didn't get a defect, so it's flat. You can see, again, I'm here in the white region. In the white region, that means I have to keep on testing, and this curve is going to wander up until, finally, I cross into the orange region. In here, I'm going to reject the lot, and that took me nine trials to figure that out. Instead of a defect count here, I'm going to use a defect rate. This is just the same data, but divided by the sample size. Since I'm dividing through by N, those straight lines become curves, so it's a curved region now, but it's the same result.

Again, it took me nine trials to figure this out. Of course, it's also possible that I'm going to have very few defects in my sample. In this case, it took—on my seven trials, or actually on my eighth trial, I got a defect, and after that, I got no more defects. I cross into the blue region after, say, 22 trials. I can also do the same thing by using a defect rate by dividing through by the current sample size. Again, it takes 22 trials to figure out that I need to—in this case—accept the lot. Let me go here. All right. When would I want to use a sequential versus a fixed test? It sounds like I'd probably want to use a sequential test all the time. I have a couple of figures here. This is just symbolic, showing how things might work. If I'm using a sequential test, I might take my part and put it in a test fixture and then see if it passes or fails. Whatever happens, I put that part out, I put another part in, I test again, and I keep on testing until my criterion are met.

For a fixed approach, for my sample size, very likely I'm going to want to use a test rack. I'm going to put a whole bunch of parts in a test rack and note how many pass and fail—the advantage here being that I'm able to test all these parts simultaneously. When would I want to use a sequential approach? If each part is really expensive, or setting up and running a test is really expensive, I probably want to minimize my sample size. In that case, I would want to use a sequential test. What if the test is destructive, and I want to minimize the number of parts I destroy? Every part that I don't destroy in my test, I can ship to the customer. In that case, again, I might want to use a sequential test. If the quality tends to be either very good or very poor—if you remember here from the previous plot, if I have a high defect rate or a very low defect rate—this is where the sequential approach really shines. I can figure out things very, very quickly using a sequential approach. But there are still instances where I might want to use a fixed approach.

What if I have a really large setup time, but after I've set up one part, the marginal cost of just adding a bunch of more parts is actually pretty low? In that case, it might be reasonable just to go ahead and test a whole bunch of parts, especially if each part is inexpensive. What if I have a very long test time? What if this takes 3 days to test, and then the next part takes 4 days to test? That's a week. It might make sense, then, to just put a whole bunch of parts on test all the same time and run them in parallel. That way I can save myself a lot of time. Also, it might be good just for logistic simplicity: it might be easier just to load up a test rack, put all these parts on test and then walk away and go do something else. If I have a test that I have to test one at a time and I have to be there, that can be very time-consuming; that means there's other stuff I'm not getting done. Maybe I don't want to do that. Maybe I want to test parts in parallel.

There are reasons why you might want to use a fixed approach. Let's look at a couple of examples here, but before I do, I want to show you the JMP Marketplace. This is the JMP Marketplace, and this is where you can find the sequential sampling add-in. It's free to download, so I hope you do; I hope you find it helpful. This is what the sequential sampling plan looks like. I've also opened up a data table. Excuse me. I have two tabs here: I have a sequential sampling tab—this is for data analysis; this is where I'm going to input my data or maybe run a simulation. The other tab is for generating an OC curve for both a fixed approach and the sequential sampling approach. Let's say we want to input some data. I'm going to input this data table—it says DT_accept—so that lets you know that I'm going to end up accepting this batch. How do we work the add-in? First thing we have to do is input the data. I'm going to use this, click on the radio button here, and click okay.

Now, I have a column list of data tables that are available. I'm going to pick the one, DT_accept, and I choose this data table. Now I have a column list, and I have to assign the column lists roles—the column's roles. Here, the first column is called IDX. This is basically a trial number: 1, 2, 3, et cetera. The second column, it says fail equals 1. This means that if I get a—this is an indicator column, so I have 1 if I get a failure, 0 otherwise. I want to tell JMP this: I want the trial number to be on the x-axis, so I choose IDX, and I go to click on X. Fail equals 1 is my response; that goes on the y-axis. Now I have to tell the add-in what values I want to input for alpha and beta, et cetera. I've already worked that out. Let's see what we got.

Alpha and beta is 0.1 and 0.2. These have already been set to those values, so great. We can set p0—p0, again, is the same as the AQL. That's going to be, in this case, 0.1. I want an RQL of 0.15; that means I want a delta of 0.05. My margin of indifference is 0.05. Remember, RQL is equal to AQL plus delta. Here I'm going to put in a delta of 0.05. I'm going to click OK. I get a little graph. We could ignore this output. On the Y-axis here, it says proportion success. You can think of it this way: I have successfully found a defect. This is basically a defect rate. On the x-axis is my trial. I'm just seeing what my response is; I'm doing the calculation until, finally, I've crossed the blue line. I did that after trial number 43, and my underlying defect rate there was about, say, 5% or so. It took me 43 trials to figure out, in this case, I should accept H0. That means I accept the lot; the lot is good enough to ship onto the customer.

I can also do one where the defect rate is higher. I'm going to choose that data table; I click here. Everything else stays the same. I'm just going to click OK, and I can get rid of this. Now, here we can see that my defect rate is wandering up until, finally, I cross the orange. Orange line says I need to reject H naught; I reject the lot. I was able to do that in 32 trials, and my underlying defect rate was a little under 30%, 40%.

That's how that works in terms of inputting the data. But we can also do a simulation: what would that look like? Instead of inputting the data, I'm going to simulate the data. I choose this radio button and I click OK. Now all this user input goes away, and I have to input values again. I have my alpha of 0.1, my beta of 0.2—p0 is the same as my AQL; I'm going to make that 0.1. Delta is going to be 0.05, and I have to choose a value for the simulation: what is the true underlying defect rate? I'm going to choose 0.1. You can see that my true underlying defect rate agrees with my AQL. If that's the case, I'm going to expect that I'm going to end up rejecting the lot, or rejecting H0 about 10% of the time—that's equal to alpha.

Now, I can do this just, say, I'm going to run one simulation; I'm going to click OK. In this particular case, I have accepted the lot after 35 trials. Really, what I ought to do is, if I want to run a simulation and see how it behaves, I should do a whole lot of iterations, but that takes a little bit of time. I've already done that, so I'm just going to show you the result.

Here's the result of my simulation. By the way, I did 500 iterations here, and it took about 100 trials or so for each one of those iterations to either accept or reject the lot. That implies about 50,000 rows. Here, you can see I have 55,000 rows. Let's look at the distribution of the results. Here I have H0, or I have accepted the lot 92% of the time. I have rejected it about 8% of the time. That's close to our alpha; our alpha is 10%, and we're right around that mark. That's pretty good. How long did it take for me to figure this out? We can look over here at the distribution of the number of trials. On the average, this took 110, but this is a skewed distribution, so maybe I should use the median—here, is 83. Roughly speaking, I'd say it took about 100 trials, in this case to—most of the time, I'm going to be accepting the lot and shipping it on to the customer. Remember that number: it took about 100 there when the underlying defect rate I chose was 10%.

Now, let's look at a different example where I run a simulation. For this simulation, I'm going to assume that the true underlying defect rate—my RQL—is 15% rather than 10%, and that's going to agree with my RQL. My RQL is 15% and the underlying defect rate is also 15%. I'm right on the border of where the customer would start to scream. Here, in that particular case, I would expect to accept the lot at a rate of beta, which in this case is 0.2. From the simulation, I would expect about 20% of the time I end up accepting the lot, and that's what happened. I accept the lot here, in this case, about 21% of the time. That worked pretty well. How many trials did it take for me to figure that out? On the average, it took 125; the median was 99, so, again, ballpark-y, around 100 trials, and I would usually end up rejecting the lot.

Okay, so let me see how we can use this information for the OC curve. I'm going to go to OC curve here, and I'm going to look here; I've already worked this out ahead of time. I'm going to put in values for alpha and beta: I'm going to put in an alpha value of 0.1, a beta value of 0.2, and I have to tell it what my acceptance number is and what my sample size is. Again, I worked this out: I'm going to put in 21 for my acceptance number, and 170 for N. I click OK, and it returns for me the AQL and the RQL. The AQL here is right around 10% and the RQL is right around 15%. That's what I had here before. Remember, the AQL here was 10% and the RQL was 15%.

Here is the OC curve for that situation. Let's look at—remember, the lot quality of 10%—what was the sample size required from the simulation? Right around 100. The purple curve is telling me that the sequential plan is able to figure this out: if the defect rate is 10%, it took me—on the average, I would expect me to take 100 trials to figure that out, and that is better than 170. Also, if the defect rate is 15%, again, it took right around 100 trials to figure that out. Again, I'm doing better than I would if I used a fixed approach.

Again, the purple and the blue lines here are overlapping, and we can see that we're getting every bit as good a response—or every bit as good a performance—from the sequential plan as the fixed plan. The OC curves are the same. Just to let you know, down here, I have a swizzle on this approach, except here you're able to solve for either N or c if you want to. Let's suppose that I know what N is: I know my test rack can handle 170; I'll put in 170. I'm going to use the same alpha: 0.1. I calculated AQL here as 9.7%, 0.097. My beta is 0.2; I'm going to put in 0.2 here. It's going to calculate the RQL for me. I click OK. It calculates the RQL, and this RQL agrees—I've added the RQL here. What was my acceptance number? My acceptance number was 21, and that value also agrees with the 21 up here. That's just another way of doing the same problem, solving the same problem.

Okay, to summarize: I went over and showed how the operating characteristic curve works, so you can have an idea how to use them. Basically, the idea behind them is that you want a way to ensure that good lots are shipped, and bad lots are caught most of the time. We went into some detail about the trade-offs to be had between alpha and beta, AQL, RQL, and the sample size and the acceptance number. Again, the important thing here is that there is a large expected savings using sequential sampling, and you're not sacrificing either the producer's or the consumer's risk. The sequential approach is able to provide every bit as good an OC curve—is every bit as good at distinguishing between good and bad lots for a much smaller sample size. That's my talk. Thank you very much.



Start:
Mon, Jun 1, 2026 09:00 AM EDT
End:
Mon, Jun 1, 2026 10:00 AM EDT
N/A
Attachments
0 Kudos