topic Re: Sample Size Question in Discussions

Sample Size Question

chrsmth — Wed, 21 Feb 2024 17:15:28 GMT

If I know my parts have a 30% failure rate, how many parts do I need to inspect to be 95% confident I will find at least one failure? Assuming I made a process improvement and want to see if my failure rate decreased from 30% to 10% w/ 95% confidence, then how many parts do I need to inspect? I think JMP can give me these answers through the DOE>Sample Size Explorer>Power wizard, but I'm not exactly sure how to use this script to get the answers to these pretty basic sample size questions.

Re: Sample Size Question

Mark_Bailey — Wed, 21 Feb 2024 18:11:22 GMT

Have you seen the documentation for the Power for Two Independent Sample Proportions?

Re: Sample Size Question

chrsmth — Wed, 21 Feb 2024 18:24:56 GMT

Hi Mark, Thanks for the quick response as this does show me how to use the nice sample proportion wizard for this type of thing. Taking a slight step back from my above question, let's say it takes 4hrs to inspect my part to see if it will fail. I need to find a failure so that I can do some failure analysis work, but management wants me to give them a estimate of time for how long it will take to inspect parts before I actually find a failure. Let's assume a historical failure rate of 30% rate and I want to be 95% confident I am giving management a good estimate of time. How many parts should I anticipate inspecting before I find a failure?

Re: Sample Size Question

dlehman1 — Wed, 21 Feb 2024 19:06:30 GMT

Sometimes I think it can be more illuminating to simulate the process. If there is a constant 30% chance of a failure, the geometric distribution will give the time to the first failure. You can use a random geometric formula in a column, add a lot of rows, and look at the distribution of times to first failures. If you also create a column using a different failure probability, comparing these will tell you a lot. Not as elegant as some other solutions, but simulation is often more intuitive for me.

Re: Sample Size Question

Mark_Bailey — Wed, 21 Feb 2024 19:27:41 GMT

I like @dlehman1's approach. This explorer for reliability might also be helpful.

Re: Sample Size Question

chrsmth — Wed, 21 Feb 2024 19:48:36 GMT

Hi dlehman1, thanks for your feedback and I like your approach also. Do you have a couple of screenshots of how to set this up as far as getting the rows with the ones and zeros in it based off a 30% failure rate? Thanks!

Interestingly, I thought my question above was pretty basic and I was missing something simple in JMP as I can tend to get lost in all the statistics terms(i.e. alpha, null hypothesis testing, proportions, beta, power, etc. etc), but it seems to not be as straight forward as I thought? There are many different ways to couch this question, but I think the way I stated it gives the general idea.

Re: Sample Size Question

dlehman1 — Wed, 21 Feb 2024 20:11:33 GMT

Attached is an example comparing the first failure at 30% and 10% probabilities, with a few graphs of the results stored as scripts.

Re: Sample Size Question

chrsmth — Wed, 21 Feb 2024 20:36:49 GMT

Thanks for the sample file, dlehman1. I do understand how you got the charts and the data, but I am still a little confused about how I should use these charts to answer these two questions:

Say it takes 4hrs to inspect a part to see if it will fail. I need to find a failure so that I can do some failure analysis work, but management wants me to give them an estimate of time for how long it will take to inspect parts before I actually find a failure. Let's assume a historical failure rate of 30% and I want to be 95% confident I am giving management a good estimate of time. How many parts should I anticipate inspecting before I find a failure? Then, the same question, but with a historical 10% failure rate.

Re: Sample Size Question

MRB3855 — Wed, 21 Feb 2024 22:20:21 GMT

Hi @chrsmth . You could think about it this way. Prob(finding at least 1)=Prob(x>0), where x is the number of failures in a sample size of n, = 1 - Prob(x=0)^n.
1 - Prob(x=0)^n = 1-(1-p)^n = 0.95.
This implies (1-p)^n = 0.05.

This implies n = ln(0.05)/ln(1-p) where p = 0.3 or 0.1. Then round n up to the nearest integer.

n is then the sample size such that there is 95% chance of at least one of them is a failure.

The thing is, this is a question of probability, not statistics. Problems of statistics involve testing/estimating parameters based on observed data. In your case, you say you know the parameter p (0.3). So, once p is assumed to be known, the sample size question, as you state it, is all about probability. I.e., you know the distribution is binomial(n, p), where p = 0.3. So, once the distribution is known there are no statistical hypotheses to be tested. It is a matter of probability.

Now…if you want to “prove” (I’m using “prove” very loosely here) that p = 0.3, then that is a problem of statistics.

Re: Sample Size Question

dlehman1 — Wed, 21 Feb 2024 22:06:47 GMT

Well, the files gives 95% confidence intervals for the mean number of parts before the first defect shows up. If it takes 4 hours to inspect each part, then multiply the mean number of parts (and/or the confidence interval) by 4 hours. I think I may be missing something in your question, but that is what I would say.

Re: Sample Size Question

chrsmth — Thu, 22 Feb 2024 16:38:02 GMT

Thanks dlehman1 and MRB3855 as both of these do end up giving me what I was looking for as I now have a few tools to add to my JMP toolbox. I appreciate the formula supplied by MRB3855 (Samples to test to be 95% confident you find a reject = ln(0.05)/ln(1-p) where p is the reject rate which was 0.3 (30%) or 0.1 (10%), then round n up to the nearest integer) to get the answer to my basic question (9 and 29, respectively).

The geometric distribution was a new thing for me (thanks again for the additional explanation of that distribution MRB3855) and simulating this distribution as done in the sample file supplied by dlehman1 was an elegant statistics approach to my question which helps join the statistics vs probability component of this question. Taking this approach a little further, I highlighted the data points around the 95% confidence level, then subset that data, which has its own distribution with a Mean approximately equal to the number supplied by the formula above. Note, for anybody following along here, this mean number is much different than the Mean of these two randomly generated Geometric distributions as their Means are based on the whole distribution for how many parts you will need to inspect on average to find a single failure versus how many parts you need to inspect to be 95% confident you will find a single reject.

Perhaps a small challenge to this awesome JMP community would be if somebody can take this information and make a cool add-in script whereby people with a more limited understanding of statistics and probability can answer my initial question, so they can look smart to their bosses. For example, if the user supplies the Defect Level to Detect and the Confidence Level for detecting a defect, while the script simply outputs the Number of parts to Inspect would be a nice Sample Size calculator. Adding in dlehman1 Geometric distribution idea would be even cooler!

Re: Sample Size Question

dlehman1 — Wed, 21 Feb 2024 22:36:43 GMT

Simulation provides a link between probability and statistics. Almost any probability problem can be simulated and statistical analysis of the simulation can provide results that match (closely if enough simulations are used) the probability theoretical results. Since I have always been bad at probability theory, simulation works better for me. It is also easier to explain to people.

Re: Sample Size Question

MRB3855 — Thu, 22 Feb 2024 09:54:09 GMT

Agreed @dlehman1 ! And, for completeness and for everyone's further understanding: For the geometric distribution, the way "Random Geometric (p)" is parameterized in JMP (returns the number of non-failures until a failure occurs, for failures with probability p), the mean is (1-p)/p. So, for p=0.3 the mean is (1-0.3)/0.3 = 2.3. And for p=0.1, the mean is (1-0.1)/0.1 = 9. And, these verify your simulations where you estimated the means as 2.3487 and 8.951, respectively.

https://en.wikipedia.org/wiki/Geometric_distribution

Re: Sample Size Question

calking — Thu, 22 Feb 2024 19:55:44 GMT

Hey @chrsmth!

As it turns out, there does exist a tool in JMP to do the calculation @MRB3855 described for you. It just involves a bit of creative usage.

The Nonparametric Demonstration Reliability explorer (DOE>>Sample Size Explorers>>Reliability>>Nonparametric Reliability Demonstration) can easily perform the calculation you need. Just set Max Failures to 0, Alpha to 1-confidence level (in this case, 0.05), make sure the drop-down menu next to Solve for reads Sample Size, and then put 1-p into the Demonstration Reliability box. It should then give you the Sample Size (rounded up the next integer) you need. See the example below:

This explorer is used to determine how many sample sizes you need to demonstrate a given reliability (1-defect rate) while allowing for a maximum number of failures. It is essentially the complement to your situation, where you want to know how many samples you need to demonstrate a given defect rate while allowing for a minimum number of failures. So, in short, it is directly performing the trick @MRB3855 described.

In general, by putting in the complement of the defect rate and one less than the minimum number of defects you would like in the Max Failures entry, you should get the sample size you're looking for in any variation of your situation. Not exactly the direct tool you may have wanted, but if I have described the procedure well enough, this should still answer your question with little effort on your part (you just have to compute 1-p).

Re: Sample Size Question

chrsmth — Thu, 22 Feb 2024 21:03:08 GMT

Thanks, calking!! I knew JMP had to have this built in somewhere. Now I, along with everybody else who has made it this far in the thread, know how to get the answer to my question using the already built in JMP script.

Re: Sample Size Question

WebDesignesCrow — Fri, 23 Feb 2024 01:23:52 GMT

Hi @chrsmth ,

How about using OC-curve for Hypergeometric / Binomial?

You did not mention the lot size, N.

If N is big, binomial is OK, but if N is small like <1000 maybe you want to use Hypergeometric distribution.

In this case, 95% confidence that it will detect at least 1 defective = Probability of acceptance, Pa=0.05 (in Y-axis of OC curve) because you will accept the sampling when c=0 @ 30% of defective rate (i.e X = 0.3).

During sampling of n sample size, you can detect 1 defective, 2 defective etc.

Usually, I just use Excel to develop OC curve because it's easier for me to understand the reasoning behind the choice of my sample size. But, JMP should be able to simulate that faster. Example as below;

Maybe this research can help: https://www.researchsquare.com/article/rs-3099107/v1

Re: Sample Size Question

chris_dennis — Thu, 30 May 2024 18:07:04 GMT

Mark,

I found this tread and have been reviewing for a similar problem I am facing.

We recently had a failure that we did not find in our normal sampling.

After determining the root cause, we must items to verify that involve sample size.

First item is we want to verify our containment by taking a sample from product produced before and after the event time frame we have defined. We want to determine sample size we should use based on our lot size (250 pieces) and frequency we saw of failures during the event (8 to 30%). Is there a profiler you would recommend helping us model this condition? I am not a statistician.

Second items is adjusting our sample size going forward. Currently we are sampling 1 piece / Lot (destructive test), but only a small portion of products from this type of material. We have already discided to increase the sample size to include all products, but want to understand the confidence of detecting a defect? Can we factor in the samples taken increasing the products from about 20% of the population to 100%? We hope to not have to sample many pieces from each lot, but I don’t know how to make or model this calculation.

Thanks in advance for your support and suggestions.

Re: Sample Size Question

Mark_Bailey — Fri, 31 May 2024 12:27:30 GMT

I think the first question was answered by me at the start of this discussion.

I'm not sure how to address the second question. Have you surveyed the sample-size explorers to see if one suits your purpose?

Re: Sample Size Question

MRB3855 — Fri, 31 May 2024 12:53:41 GMT

Hi @chris_dennis . I confess, and apologize, that neither of your questions are clear to me. What exactly are you assuming, and what exactly do you want to know?

Re: Sample Size Question

chris_dennis — Fri, 31 May 2024 15:26:51 GMT

I have included a process diagram to help explain.

1 What sample size is needed to verify after corrective action that we have correctly contained the event.

2 What should our sample size be for future production to detect internally.

We will need to define confidence level for each sample size (1 &2).