As soon as we start to learn about statistical tools, such as those enabled with JMP, the concept of sampling and sample size becomes important. Regulated industries, such as medical devices, require a justification of sample size determination when making design or process changes, as well as for verification/validation of performance. There is a nearly mystical relationship between sample size and the ability of statistical analysis to answer a question of interest, but this relationship isn't often explained or discussed in great detail. This presentation tries to break the taboo, discussing the general relationship between sample size, power and confidence, and effect size (difference to be detected), at least in the context of statistical inference and hypothesis testing. Along the way, we look at some of the sample size calculators available in JMP 19.

 

 

All right. Hi, everyone. My name is Dale Human. I work at GE HealthCare. Today we're going to talk a little bit about sample size as it relates to statistical hypothesis testing. The topics I'll go through today very briefly are motivation: why are we worrying about sample size in the first place? Then we're going to talk about some terms that you need to be a little bit familiar with if you're worrying about sample size and how sample size affects analysis that you might be doing. That's the "It's all Greek to me." There are a lot of Greek letters involved there. We'll go through what those mean. Of course, then we're going to talk about how to do these things in JMP. I'll go through a particular example of one of the sample size calculators that's available to you in JMP, and we'll finish off with that example. All right?

Okay, so why do we care about sample size? Oftentimes in industrial settings, we want to do comparisons. Very commonly, we want to compare parts that we get from two suppliers as an example. In this particular example, let's imagine you get some parts from a supplier, and they measure the diameter of these parts. Their shafts, let's say.

You want to know if the diameter of the shafts that you're getting from supplier A is less than the diameter of the shafts that you're getting from supplier B, a very typical context for this. Maybe you're a process engineer, and you've instituted a change in your welding process, and you want to be able to show that your change in your process has actually resulted in an improved yield from your welding process. That's also a typical application for these things.

Of course, the way you answer these questions is generally you gather some data from before the change and after the change, or from your two different suppliers, and then you'll typically do some statistical hypothesis test depending on exactly the quantity that you're wanting to compare. It turns out that when you start talking about hypothesis testing, sample size is very important for quantifying and controlling the risk that you have from making your conclusions based on a statistical hypothesis test, and that's what we want to focus on here.

All right, so let me look at a brief example here. We have some data from two different suppliers. If we think about that, parts with different diameters from two suppliers here. We have a data table that we can open. You can see the data here. I've got nine data points from supplier A, where we've measured the diameter of the parts from that supplier, and 12 data points from supplier B. Not the same number of data points in each of the samples. That's okay. We can handle that. We'd like to answer the question, "Is there a difference in these parts?" If we go over to JMP and open up that data table, I have that here. You can see the data here.

First, I want to start off with a graph just because it's always a good idea to graph your data before you start doing any analysis. I'm just going to go to the Graph Builder real quick and put my two supplier diameters on the Y-axis. Because I've got nine or more data points, generally, I like box plots when I'm doing a comparison by groups, in this case, by suppliers. I'm going to turn this into a box plot and look at the diameters based on the supplier, right? Very simple box plot. When you look at this box plot, it looks like maybe supplier B has a little bit higher diameter on average compared to supplier A. That box tends to be shifted a little bit upward, but it's not completely separated. The question becomes, "Should I really expect the diameters of parts from supplier B to really be greater than those from supplier A over the long term?" It's not super clear from this graph, so we'd like to do a t-Test in this case if we want to compare average diameters.

If we want to do a t-Test, then we have to use the Fit Y by X platform in JMP. If we want to use the Fit Y by X platform, of course, our data needs to be in a stacked structure, right? I'm going to stack this data real quick. Just go to the Tables menu, Stack, grab the two columns that I would like to stack, and put them in the Stack Columns field. JMP shows you that preview. You can rename things. I'm just going to rename the data to be Diameter and the label to be Supplier. I'm just going to click OK, keep the table names the same, and all of that. Now we have a new data table, which is the same data, just in this particular structure now, where one column indicates which supplier that row comes from, and then the Diameter column, of course, being the values.

Now we'd like to do a t-Test to compare the average diameters from these two suppliers. I'm going to go to the Analyze, Fit Y by X platform and put our Diameter in the Y field and Supplier in the X field. JMP is indicating it's going to do a One-way analysis because we have one grouping factor, so that's great. I'll click OK. JMP gives us a different graph this time, a dot plot rather than a box plot, but that's okay. Again, we can see that if we look at the center of these groups, they're close. Maybe, again, group B looks like it might be a slightly higher average. We've got this one point down here that's a little weird, but we're going to keep that data in there for now. We don't have any reason to exclude it, so we're just going to compare these. If I want to do a t-Test, I'm just going to go to the red triangle, and I'm going to choose the Means/ANOVA/Pooled t.

Now, if you're familiar with doing hypothesis tests, I might do an unequal variances comparison first just to find out which version of the t-Test might be more appropriate. For now, I'm just going to stick with the pooled t-Test for this particular example. I'll hide all of the ANOVA tables just so we can focus on the t-Test here, as you can see here. From our pooled t-Test, if you recall, the null hypothesis for a t-Test is that there is no difference in the mean value between the groups. The alternative is that there is a difference in mean between the groups.

In our case, because we wanted to know if group B truly does have a greater average compared to supplier A, we want to use this middle p-value here, the probability greater than t. We're doing a one-sided, two-sample t-Test in this case, and it has a p-value greater than 0.05. At 95% confidence, we would say it doesn't look like there's a difference between these two suppliers, right? That's a pretty typical approach to doing an analysis like this. You'll go gather your data, you've looked at your data, and then you run your t-Test analysis in this case, and it says, "No, it looks like there's no difference." If we were looking to say, "Okay, we need a second supplier," and supplier B was a candidate, it looks like they behave just like our current supplier, supplier A. Let's go ahead and add them as a second source. Very common application again here for us.

That's a typical approach to all this. All right, so that's good. I have all of that here that I've gone through with you. The problem becomes that you made this conclusion, right? We looked at this p-value, and we've said, "It looks like there's no difference, and there's nothing wrong with what we've done." We might do a couple of other steps here and there, but in general, you've gathered your data, you've looked at it, you said, "Okay, it looks like there's no difference." The question really becomes, can we trust that conclusion? That's what we really want to be able to answer. Let me talk about how we determine when we can trust that conclusion. In order to do that, we have to talk about these quantities, alpha, beta, and p. We've mentioned them already a little bit, so let's look at that. If you recall, when you're doing a statistical hypothesis test, you have two quantities of interest, alpha and beta. They're two different kinds of risk for making an incorrect conclusion based on your analysis.

Alpha risk is the risk of concluding that things are different when, in fact, they are not different in reality, but your data and your conclusions are telling you they are different. That's called an alpha risk. Then beta risk is the opposite case. Things are different, but your data is telling you there is no difference, so you don't detect a difference that you should be detecting, and that's what we call the beta risk.

In terms of hypothesis testing, you can think of it as there are two possible truths. Either there is no difference, the null hypothesis is true, or there is a difference, which is what we would call the alternative hypothesis. One of these is true, all right? You're trying to decide which one of these two things is true, and it's a binary choice. You have to choose one or the other. What you're going to do... You don't know the truth because you will never measure the entire population from your data, wherever you're gathering your data. Generally speaking, we typically are basing our decisions on a sample, right? Because we're using a sample, we'll never really know the truth. All we're going to do is get a sample of data and then make a decision based on that sample of data.

We have two possible decisions we can make in this case. One decision could be that we decide not to reject the null hypothesis. In other words, we say, "There is no difference: things have not changed." That's a possible decision you can make, or, of course, you could decide the alternative is true, and you reject the null and say, "Yes, there is a change. These two suppliers are different." Those are your two possible decisions that you're going to make, and hopefully you're choosing the correct one based on the truth.

Given that, there are four possible outcomes here. If we focus on the left side, let me say that the null hypothesis is true: there is no difference in the things that you're comparing. That's the truth. You go gather some data, you do your analysis, you run your t-Test, your chi-square test, whatever the case may be, and based on the results of that test, you conclude things are not different. Then you've made the correct decision. That's great. That's really where you want to be. Of course, the alternative could be the case where your data says that, no, there is a difference, but the truth is that there is no difference. But your sample happens to tell you that these things appear to be different.

In that case, you've made an error. We call that a type 1 error or an alpha error, also called a false positive error, and that's because you're saying things have changed when, in fact, they have not, right? That's why it's a false positive. There's a positive change that you're looking at. We quantify that risk by this quantity we call alpha. Because one of these two has to be true, if alpha is the risk of making an error, then 1 minus alpha is the probability that you've made the correct decision. This quantity, 1 minus alpha, is what we call the confidence level, and that is something that, generally, when you're doing a hypothesis test, you choose your confidence level upfront. I want to make this decision with 95% confidence or 90% confidence. You choose your confidence level based on the impact of making the wrong conclusion. That's alpha.

Let's shift gears and go to the other column, and let's say that the alternative is true. In fact, there is a difference between these two suppliers, and that is the truth. Again, you go gather your data, you run your analysis, you look at the conclusions from your analysis, and you make a decision. Again, if your decision is that you reject the null hypothesis, and you conclude things are different, again, you've made the correct decision, and that's where you'd really like to be. I want to make the correct decisions based on my analysis.

But you could make a mistake, and your data gives you the wrong conclusion, and we call that a type 2 error or a beta error, also called a false negative, because you didn't detect a difference that you should have detected because there is a difference in reality, right? Again, if beta is the probability of making this error, we call that beta. Then the probability of making the correct decision in that case is 1 minus beta because one of these two has to be the truth, all right? This quantity 1 minus beta is what we call the statistical power of the test. You have a false positive relating to alpha and a false negative relating to beta, or confidence level and statistical power. These are the two things that relate to risk when you're making a conclusion based on a sample of data. We have to keep that in mind.

Another way to look at this would be to graphically draw pictures here to explain what this really means. Here's our data. We have this data from supplier A, and then we have our data from supplier B, and they have some mean and standard deviations, since we're talking about t-Tests here. Imagine this distribution here. This is the distribution for manufacturer A, supplier A, and it has its average value, which was about 10.33 for our data here. Then we said, we want to make a decision: is supplier B different or not, with 95% confidence? Well, as soon as you choose that 95% confidence, you've determined a value, which we call the critical value for the test that you're doing. That value is chosen based on where I draw a line such that 1 minus my confidence level, or alpha percent, 5% in this case, of my distribution is covered by that line. This blue-colored area here represents 5% of my null population. Once you do that, you're basically saying that if the average of my supplier B is out here in this blue area, so anywhere past this blue line, I'm going to conclude that things are different.

That's the importance of that confidence level choice, or the choice of alpha, the significance level, if you like. It determines where this line is going to be drawn given the distribution of the values that you have for your null population, in this case, supplier A, assuming they're our null population, right? That's the critical value based on alpha. It determines where that line falls. Then after that, you go, and you get your data from supplier B, and you look at its distribution. Here's our distribution of values for supplier B. The mean for supplier B is a little bit higher than supplier A, so that's this light blue line here if you can see that, and it has some amount of variation, a little bit less variation than supplier A, 0.64 versus 0.74, as you can see up there. All right? When we calculate our p-value, the p-value, it turns out, is really just the area underneath the manufacturer A curve to the right of the mean of the manufacturer B distribution. You're basically looking at how much do these two groups overlap. If they overlap by more than that blue area, then I will say these are not different. There's so much overlap that I can't say that these two groups are different from each other. That's graphically what's happening there. That's how that p-value is being calculated behind the scenes.

Typically, this is where you stop, and you say, "All right, I've got a p-value. Is my p-value less than my significance level?" 0.05 most of the time for 5% significance or 95% confidence, and you say, "Is it less than or greater than that value?" Make your decision, and that's the end of it. But there's more to it than that, really, because if you look at that distribution for my light blue curve, which represents my alternative distribution in this case, let's say it's supplier B's distribution, there's this whole area to the left of that blue line, and that area actually represents your beta risk, right? You can think of it as all of this area for my supplier B, it overlaps with my supplier A by so much that I would actually say all of these values belong to supplier A. They don't belong to supplier B. Now, in reality, they do belong to supplier B, but because there's so much overlap, I can't tell that's the case, right?

This area here that I've colored in with these red hash marks represents your beta risk, and that is really controlled by your sample size. How many samples do you have? We'll look at that with these calculators here momentarily. We have to pay attention to this part of the curves that we don't talk too much about. Typically, when you learn about hypothesis test: it's choose your level of confidence, that determines your value of alpha, you calculate your p-value, and then you compare your p-value to your choice of alpha and see what happens. That's how you make a conclusion. But in reality, you want to also think about this beta risk.

That's beta. I put a little caveat here to make sure I don't get too many bad notes from statisticians. I've been drawing this, saying imagine these are the distributions of the real values that you've been measuring. In fact, they're using the sampling distributions of the statistic that you're talking about. In this case, we're talking about the means of these two groups, and typically for that, you would use the t-distribution. These are actually t-distribution curves. They're not the real distributions of the actual data, so I just want to be very clear about that. But you can think about it in your head as being the real distributions. That's how I think about it.

All right. Hopefully that suffices for the technicality there. We'll move on to talk about these other quantities, which I call delta, sigma, and n, some more Greek letters to talk about here. These are really related to, well, how different is different? If we look at this quantity, delta, and in our example, we're comparing the means of two groups, so delta is just simply the difference between those two means. If that delta is a very small number, my means are very close together, then it's going to be hard to say that they are different from each other unless you have a very small variation between your groups. But in general, the size of that difference here determines how easy it's going to be to detect a difference if a difference does actually exist, of course, right?

You can see here, if you recall from the previous page, this area is really large, in this case, because this delta is so small. I got a lot of overlap. I have a high beta risk or a low probability of detecting a difference if there is actually a difference. More than likely, my t-Test will say there's not a difference because I've got so much overlap. But if I were to just take these two and separate them and look at a larger difference, even though they both have the same amount of variation as they do on the left, forgiving my artwork, of course, you can see the beta risk now is very small. It's the area under my alternative distribution, supplier B, that overlaps with supplier A, right? That's much smaller because it's so far away that I have very little overlap. That's the idea of the effect size, or the size of the difference that you're trying to detect, which we call delta. All right? Larger delta equals easier-to-detect differences, lower beta risk.

What about variation? Variation plays a role here as well. If you have a lot of variation, like what I've drawn here, a lot being a relative term, then I've got a lot of overlap. It's hard to detect a difference because there's so much overlap. Even though the means are separated from each other, it's hard for me to tell that the population means are really different from each other because there's so much overlap. Whereas here, if I had these two populations with the same mean. But with lower variation, again, my beta risk is much smaller here because there's less overlap. Even though they have the same mean value, they have lower variations, and so their curves don't overlap so much. So that's another way of reducing your beta risk, if you're looking at things that have lower variation relative to higher variation.

Then finally, getting to the one we're really interested in, which is sample size, which is given by the letter n here. If all I do is take the same two distributions, but I gather more and more data from each of those different distributions, so instead of having nine data points from supplier A and 12 from supplier B, let's say I get 30 or 40 from each of those suppliers, then that actually reduces the variation in these curves also because these curves don't just use the standard deviation, right? They use the standard error of the mean, which is the standard deviation divided by the square root of n. If I increase n, I reduce my standard error, and from a hypothesis test perspective, that shrinks those distributions and reduces the amount of overlap. It's just like reducing the standard deviation.

Your uncertainty in where the true population mean lies goes down as you increase your sample size. You measure more of the population, you have a better idea of where the true mean of that population lies. Just by increasing your sample size, you can also reduce your beta risk or increase your power, which means increasing the likelihood of saying there is a difference if a difference does, in fact, exist.

That's the one we're going to really pay attention to here because we see that there's this relationship between n and beta, and that's really why we want to focus on sample size for hypothesis testing: to reduce that beta risk down to an acceptable level or increase our power, whichever way you'd like to think about it.

All right, so let's look at some calculators here to finish up. JMP has a lot of calculators that you can choose from. They're on the DOE menu. If you go to DOE and then down to the Sample Size Explorers there, you'll see all of these different options. If you're interested in looking at changes in power or beta risk relating to sample size, there's a whole list for that.

There's another list for confidence intervals, which is really just saying, given the size of your sample, what's the range of uncertainty for the quantity that you're evaluating, a mean, a standard deviation, et cetera? You're comparing your uncertainty in the knowledge of a value based on sample size. That would be the confidence interval calculators. There are a couple of others relating to reliability calculations and quality calculations. I'm not going to talk about those here. We're actually going to focus on the power for two independent sample means because we want to figure out, well, what's our power for these tests, or what's our beta risk for these tests? Let me do that.

All right, so we're going to come back to our example. We did this t-Test previously. We found that there is no difference, right? Our p-value is greater than 0.05 for 95% confidence, and we're saying there is no difference. The question becomes, well, do we believe that's true? Is it a trustworthy conclusion? Given the fact that I only had nine data points from supplier A and 12 data points from supplier B, should I really trust the decision that there is no difference between them?

We can do that by going into JMP. Let me go back to our data table here. We want to look at the power of this test. I'm going to go to the DOE menu, and then Sample Size Explorers. Then I'm going to go to Power in this case because I'm focusing on beta risk or statistical power, and we're doing a two-sample t-Test. Let's go to the Power for Two Independent Sample Means calculator. It opens up a window like this. All right. Okay.

It looks a little complicated. This graph probably looks like the Prediction Profiler. If you've done any regression analysis or modeling in JMP, you've probably seen a profiler graph like what we're seeing here. There are some settings that we want to talk through. You have to choose: are you doing a one-sided calculation or a two-sided calculation? In my example, I'm doing a one-sided calculation because I'm curious if supplier B is higher on average than supplier A. That's a one-sided test, so I'll change that to one-sided. I'm using 95% confidence, so I'll keep my alpha value at 0.05. Generally, you're going to keep your Population Standard Deviation Known to be No, because almost all the time you're dealing with a sample; you never have the entire population, fairly rare at least in industrial cases.

All right, so then there's these different things down here, and you notice you have to look at your total sample size, which is simply the sum of the number of samples that you have in each of your groups, in this case, supplier A and supplier B, if you will. How big is the difference to detect? Then what are the standard deviations? All of these terms that we just talked through and showed how they relate to each other in terms of power and beta risk, right? Let's go look at our data real quick and figure out these quantities. I'm going to just go back to my data table, and let's do a distribution analysis. I'm going to go to Analyze, Distribution, and I'm going to put Diameter in the Y field. I want to do this by supplier because I want to know the mean and the standard deviation for each of my suppliers. I'm going to put the Supplier column in the By field here and click OK. Here we see we have some summary statistics for each of the groups. I'm going to hide the Quantiles tables for now just so it's a little easier to scroll back and forth.

If you look at for supplier A, it had a mean of about 10.33 and a standard deviation of about 0.74, and similarly for supplier B, a mean of 10.74, so a little bit higher mean, but a standard deviation of about 0.64. I have nine data points for supplier A and 12 for supplier B here. We can come over to our calculator and actually plug those values right in down here on the profiler. My total sample size is 9+12=21. For 21, I had nine of those in group one, I'll call that supplier A, and then 12 in group two, it automatically changed because I put 21 over here. What is the difference I'm trying to detect in this case? If I go back to my distribution analysis, you really just need to take the difference between the means here, so 10.74 minus 10.33. It turns out that's 0.41 or 0.4114, if you want to get to the fourth decimal place. That's the difference between those. Then I'll put in the standard deviations for the two groups. Again, supplier A had a standard deviation of 0.74, and supplier B had a standard deviation of 0.64. When you just put in those numbers and don't do anything else, you get a power calculation right over here.

This is telling you the power of the t-Test that we have already performed. Given the fact that I only had 21 total samples, 9 for one group and 12 for the other, with this difference in average and these standard deviations, the power of that test is 36%. That means if there is a difference in the average between these two groups, there's only a 36% chance that you're going to detect that difference given this sample of data. Even though the t-Test says there is no difference, there's a pretty good chance that there could be a difference, and you're simply not detecting it with your test. That's the idea of these calculators. It allows you to figure out what is the power of the test that you've done. That's what we looked at here. But you can also go the other way and say, "Well, I want to make sure that I have 90% power because I want to make sure that if there is a difference, I detect that difference at least 90% of the time." We can do that. If you go up here to your power field, you can say, "I want 90% power," or 0.9. I'm accepting a beta risk of 10% or 0.1 in that case.

Then you have to choose something in your drop-down that you want to calculate, right? Given that I want a power of 0.9, I'm going to just go and say, "Give me my total sample size needed in order to make sure that if there is a difference in the average for these two suppliers, that I would detect that difference in my test." Notice that there's also this box here for the ratio of the group sizes. You can change that if you like. I'm going to keep it at 1.33. It turned out that I had 33% more data in group two than I had in group one for no reason. That's just how much data we had available, right? I'm just going to keep that there for now, but you could play with that number if you'd like. I'm going to click Go. When I click Go, JMP goes and calculates, well, if you want to make sure you have 90% power for your test, then you need a total sample size of 103, which is the ceiling of this number here. You have to round up to the next highest integer because you can't have a partial sample, presumably.

All right, so I need a total of 103, and if I have a ratio of number of data points between the groups of 1.33, then I need 44 data points from supplier A, and I need 59 data points from supplier B. If I go get that much data, and again, I'm trying to detect this size difference with these standard deviations, if those still hold true for those larger groups, then my t-Test has a 90% chance of detecting a difference if the difference does exist. That's how you can go both directions. We initially said, "I've got this data, what's my power?" You can do it proactively and say, "I want to make sure I have this much power in my test. How many samples do I need?" You can go either way there, and you can play with different quantities. If I don't want to look at total sample size, maybe you want to say, "Hey, given the number of samples I have, how small of a difference can I actually detect with 90% power?" You might calculate that and say, "Oh, well, it didn't change much because I used those numbers previously." But you can really calculate any one of these quantities based on the values of the other quantities for a given level of power that you're shooting for.

Reactively, you already have some data, and you want to evaluate what's the power of a t-Test given this data, or proactive, determine how many samples you need in order to make sure that your test is powerful enough to detect the change you're interested in.

That is one way to use these calculators. Again, JMP has a lot of other calculators. They're all very similar. Of course, they have different parameters in them. Whether you're comparing proportions or means, you're going to have slightly different parameters to worry about, but they all work in basically the same way. That's a way you can use these sample size calculators in JMP to evaluate. In this case, we focused a lot on power to control the risk of a wrong decision, and that's very useful from our perspective, so make sure you check those out in JMP. Thank you very much; appreciate it.

0 Comments
Presented At Discovery Summit 2026

Presenter

Skill level

Beginner
  • Beginner
  • Intermediate
  • Advanced

Files

Published on ‎07-15-2026 03:39 PM by Community Manager Community Manager | Updated on ‎10-30-2026 03:34 PM

As soon as we start to learn about statistical tools, such as those enabled with JMP, the concept of sampling and sample size becomes important. Regulated industries, such as medical devices, require a justification of sample size determination when making design or process changes, as well as for verification/validation of performance. There is a nearly mystical relationship between sample size and the ability of statistical analysis to answer a question of interest, but this relationship isn't often explained or discussed in great detail. This presentation tries to break the taboo, discussing the general relationship between sample size, power and confidence, and effect size (difference to be detected), at least in the context of statistical inference and hypothesis testing. Along the way, we look at some of the sample size calculators available in JMP 19.

 

 

All right. Hi, everyone. My name is Dale Human. I work at GE HealthCare. Today we're going to talk a little bit about sample size as it relates to statistical hypothesis testing. The topics I'll go through today very briefly are motivation: why are we worrying about sample size in the first place? Then we're going to talk about some terms that you need to be a little bit familiar with if you're worrying about sample size and how sample size affects analysis that you might be doing. That's the "It's all Greek to me." There are a lot of Greek letters involved there. We'll go through what those mean. Of course, then we're going to talk about how to do these things in JMP. I'll go through a particular example of one of the sample size calculators that's available to you in JMP, and we'll finish off with that example. All right?

Okay, so why do we care about sample size? Oftentimes in industrial settings, we want to do comparisons. Very commonly, we want to compare parts that we get from two suppliers as an example. In this particular example, let's imagine you get some parts from a supplier, and they measure the diameter of these parts. Their shafts, let's say.

You want to know if the diameter of the shafts that you're getting from supplier A is less than the diameter of the shafts that you're getting from supplier B, a very typical context for this. Maybe you're a process engineer, and you've instituted a change in your welding process, and you want to be able to show that your change in your process has actually resulted in an improved yield from your welding process. That's also a typical application for these things.

Of course, the way you answer these questions is generally you gather some data from before the change and after the change, or from your two different suppliers, and then you'll typically do some statistical hypothesis test depending on exactly the quantity that you're wanting to compare. It turns out that when you start talking about hypothesis testing, sample size is very important for quantifying and controlling the risk that you have from making your conclusions based on a statistical hypothesis test, and that's what we want to focus on here.

All right, so let me look at a brief example here. We have some data from two different suppliers. If we think about that, parts with different diameters from two suppliers here. We have a data table that we can open. You can see the data here. I've got nine data points from supplier A, where we've measured the diameter of the parts from that supplier, and 12 data points from supplier B. Not the same number of data points in each of the samples. That's okay. We can handle that. We'd like to answer the question, "Is there a difference in these parts?" If we go over to JMP and open up that data table, I have that here. You can see the data here.

First, I want to start off with a graph just because it's always a good idea to graph your data before you start doing any analysis. I'm just going to go to the Graph Builder real quick and put my two supplier diameters on the Y-axis. Because I've got nine or more data points, generally, I like box plots when I'm doing a comparison by groups, in this case, by suppliers. I'm going to turn this into a box plot and look at the diameters based on the supplier, right? Very simple box plot. When you look at this box plot, it looks like maybe supplier B has a little bit higher diameter on average compared to supplier A. That box tends to be shifted a little bit upward, but it's not completely separated. The question becomes, "Should I really expect the diameters of parts from supplier B to really be greater than those from supplier A over the long term?" It's not super clear from this graph, so we'd like to do a t-Test in this case if we want to compare average diameters.

If we want to do a t-Test, then we have to use the Fit Y by X platform in JMP. If we want to use the Fit Y by X platform, of course, our data needs to be in a stacked structure, right? I'm going to stack this data real quick. Just go to the Tables menu, Stack, grab the two columns that I would like to stack, and put them in the Stack Columns field. JMP shows you that preview. You can rename things. I'm just going to rename the data to be Diameter and the label to be Supplier. I'm just going to click OK, keep the table names the same, and all of that. Now we have a new data table, which is the same data, just in this particular structure now, where one column indicates which supplier that row comes from, and then the Diameter column, of course, being the values.

Now we'd like to do a t-Test to compare the average diameters from these two suppliers. I'm going to go to the Analyze, Fit Y by X platform and put our Diameter in the Y field and Supplier in the X field. JMP is indicating it's going to do a One-way analysis because we have one grouping factor, so that's great. I'll click OK. JMP gives us a different graph this time, a dot plot rather than a box plot, but that's okay. Again, we can see that if we look at the center of these groups, they're close. Maybe, again, group B looks like it might be a slightly higher average. We've got this one point down here that's a little weird, but we're going to keep that data in there for now. We don't have any reason to exclude it, so we're just going to compare these. If I want to do a t-Test, I'm just going to go to the red triangle, and I'm going to choose the Means/ANOVA/Pooled t.

Now, if you're familiar with doing hypothesis tests, I might do an unequal variances comparison first just to find out which version of the t-Test might be more appropriate. For now, I'm just going to stick with the pooled t-Test for this particular example. I'll hide all of the ANOVA tables just so we can focus on the t-Test here, as you can see here. From our pooled t-Test, if you recall, the null hypothesis for a t-Test is that there is no difference in the mean value between the groups. The alternative is that there is a difference in mean between the groups.

In our case, because we wanted to know if group B truly does have a greater average compared to supplier A, we want to use this middle p-value here, the probability greater than t. We're doing a one-sided, two-sample t-Test in this case, and it has a p-value greater than 0.05. At 95% confidence, we would say it doesn't look like there's a difference between these two suppliers, right? That's a pretty typical approach to doing an analysis like this. You'll go gather your data, you've looked at your data, and then you run your t-Test analysis in this case, and it says, "No, it looks like there's no difference." If we were looking to say, "Okay, we need a second supplier," and supplier B was a candidate, it looks like they behave just like our current supplier, supplier A. Let's go ahead and add them as a second source. Very common application again here for us.

That's a typical approach to all this. All right, so that's good. I have all of that here that I've gone through with you. The problem becomes that you made this conclusion, right? We looked at this p-value, and we've said, "It looks like there's no difference, and there's nothing wrong with what we've done." We might do a couple of other steps here and there, but in general, you've gathered your data, you've looked at it, you said, "Okay, it looks like there's no difference." The question really becomes, can we trust that conclusion? That's what we really want to be able to answer. Let me talk about how we determine when we can trust that conclusion. In order to do that, we have to talk about these quantities, alpha, beta, and p. We've mentioned them already a little bit, so let's look at that. If you recall, when you're doing a statistical hypothesis test, you have two quantities of interest, alpha and beta. They're two different kinds of risk for making an incorrect conclusion based on your analysis.

Alpha risk is the risk of concluding that things are different when, in fact, they are not different in reality, but your data and your conclusions are telling you they are different. That's called an alpha risk. Then beta risk is the opposite case. Things are different, but your data is telling you there is no difference, so you don't detect a difference that you should be detecting, and that's what we call the beta risk.

In terms of hypothesis testing, you can think of it as there are two possible truths. Either there is no difference, the null hypothesis is true, or there is a difference, which is what we would call the alternative hypothesis. One of these is true, all right? You're trying to decide which one of these two things is true, and it's a binary choice. You have to choose one or the other. What you're going to do... You don't know the truth because you will never measure the entire population from your data, wherever you're gathering your data. Generally speaking, we typically are basing our decisions on a sample, right? Because we're using a sample, we'll never really know the truth. All we're going to do is get a sample of data and then make a decision based on that sample of data.

We have two possible decisions we can make in this case. One decision could be that we decide not to reject the null hypothesis. In other words, we say, "There is no difference: things have not changed." That's a possible decision you can make, or, of course, you could decide the alternative is true, and you reject the null and say, "Yes, there is a change. These two suppliers are different." Those are your two possible decisions that you're going to make, and hopefully you're choosing the correct one based on the truth.

Given that, there are four possible outcomes here. If we focus on the left side, let me say that the null hypothesis is true: there is no difference in the things that you're comparing. That's the truth. You go gather some data, you do your analysis, you run your t-Test, your chi-square test, whatever the case may be, and based on the results of that test, you conclude things are not different. Then you've made the correct decision. That's great. That's really where you want to be. Of course, the alternative could be the case where your data says that, no, there is a difference, but the truth is that there is no difference. But your sample happens to tell you that these things appear to be different.

In that case, you've made an error. We call that a type 1 error or an alpha error, also called a false positive error, and that's because you're saying things have changed when, in fact, they have not, right? That's why it's a false positive. There's a positive change that you're looking at. We quantify that risk by this quantity we call alpha. Because one of these two has to be true, if alpha is the risk of making an error, then 1 minus alpha is the probability that you've made the correct decision. This quantity, 1 minus alpha, is what we call the confidence level, and that is something that, generally, when you're doing a hypothesis test, you choose your confidence level upfront. I want to make this decision with 95% confidence or 90% confidence. You choose your confidence level based on the impact of making the wrong conclusion. That's alpha.

Let's shift gears and go to the other column, and let's say that the alternative is true. In fact, there is a difference between these two suppliers, and that is the truth. Again, you go gather your data, you run your analysis, you look at the conclusions from your analysis, and you make a decision. Again, if your decision is that you reject the null hypothesis, and you conclude things are different, again, you've made the correct decision, and that's where you'd really like to be. I want to make the correct decisions based on my analysis.

But you could make a mistake, and your data gives you the wrong conclusion, and we call that a type 2 error or a beta error, also called a false negative, because you didn't detect a difference that you should have detected because there is a difference in reality, right? Again, if beta is the probability of making this error, we call that beta. Then the probability of making the correct decision in that case is 1 minus beta because one of these two has to be the truth, all right? This quantity 1 minus beta is what we call the statistical power of the test. You have a false positive relating to alpha and a false negative relating to beta, or confidence level and statistical power. These are the two things that relate to risk when you're making a conclusion based on a sample of data. We have to keep that in mind.

Another way to look at this would be to graphically draw pictures here to explain what this really means. Here's our data. We have this data from supplier A, and then we have our data from supplier B, and they have some mean and standard deviations, since we're talking about t-Tests here. Imagine this distribution here. This is the distribution for manufacturer A, supplier A, and it has its average value, which was about 10.33 for our data here. Then we said, we want to make a decision: is supplier B different or not, with 95% confidence? Well, as soon as you choose that 95% confidence, you've determined a value, which we call the critical value for the test that you're doing. That value is chosen based on where I draw a line such that 1 minus my confidence level, or alpha percent, 5% in this case, of my distribution is covered by that line. This blue-colored area here represents 5% of my null population. Once you do that, you're basically saying that if the average of my supplier B is out here in this blue area, so anywhere past this blue line, I'm going to conclude that things are different.

That's the importance of that confidence level choice, or the choice of alpha, the significance level, if you like. It determines where this line is going to be drawn given the distribution of the values that you have for your null population, in this case, supplier A, assuming they're our null population, right? That's the critical value based on alpha. It determines where that line falls. Then after that, you go, and you get your data from supplier B, and you look at its distribution. Here's our distribution of values for supplier B. The mean for supplier B is a little bit higher than supplier A, so that's this light blue line here if you can see that, and it has some amount of variation, a little bit less variation than supplier A, 0.64 versus 0.74, as you can see up there. All right? When we calculate our p-value, the p-value, it turns out, is really just the area underneath the manufacturer A curve to the right of the mean of the manufacturer B distribution. You're basically looking at how much do these two groups overlap. If they overlap by more than that blue area, then I will say these are not different. There's so much overlap that I can't say that these two groups are different from each other. That's graphically what's happening there. That's how that p-value is being calculated behind the scenes.

Typically, this is where you stop, and you say, "All right, I've got a p-value. Is my p-value less than my significance level?" 0.05 most of the time for 5% significance or 95% confidence, and you say, "Is it less than or greater than that value?" Make your decision, and that's the end of it. But there's more to it than that, really, because if you look at that distribution for my light blue curve, which represents my alternative distribution in this case, let's say it's supplier B's distribution, there's this whole area to the left of that blue line, and that area actually represents your beta risk, right? You can think of it as all of this area for my supplier B, it overlaps with my supplier A by so much that I would actually say all of these values belong to supplier A. They don't belong to supplier B. Now, in reality, they do belong to supplier B, but because there's so much overlap, I can't tell that's the case, right?

This area here that I've colored in with these red hash marks represents your beta risk, and that is really controlled by your sample size. How many samples do you have? We'll look at that with these calculators here momentarily. We have to pay attention to this part of the curves that we don't talk too much about. Typically, when you learn about hypothesis test: it's choose your level of confidence, that determines your value of alpha, you calculate your p-value, and then you compare your p-value to your choice of alpha and see what happens. That's how you make a conclusion. But in reality, you want to also think about this beta risk.

That's beta. I put a little caveat here to make sure I don't get too many bad notes from statisticians. I've been drawing this, saying imagine these are the distributions of the real values that you've been measuring. In fact, they're using the sampling distributions of the statistic that you're talking about. In this case, we're talking about the means of these two groups, and typically for that, you would use the t-distribution. These are actually t-distribution curves. They're not the real distributions of the actual data, so I just want to be very clear about that. But you can think about it in your head as being the real distributions. That's how I think about it.

All right. Hopefully that suffices for the technicality there. We'll move on to talk about these other quantities, which I call delta, sigma, and n, some more Greek letters to talk about here. These are really related to, well, how different is different? If we look at this quantity, delta, and in our example, we're comparing the means of two groups, so delta is just simply the difference between those two means. If that delta is a very small number, my means are very close together, then it's going to be hard to say that they are different from each other unless you have a very small variation between your groups. But in general, the size of that difference here determines how easy it's going to be to detect a difference if a difference does actually exist, of course, right?

You can see here, if you recall from the previous page, this area is really large, in this case, because this delta is so small. I got a lot of overlap. I have a high beta risk or a low probability of detecting a difference if there is actually a difference. More than likely, my t-Test will say there's not a difference because I've got so much overlap. But if I were to just take these two and separate them and look at a larger difference, even though they both have the same amount of variation as they do on the left, forgiving my artwork, of course, you can see the beta risk now is very small. It's the area under my alternative distribution, supplier B, that overlaps with supplier A, right? That's much smaller because it's so far away that I have very little overlap. That's the idea of the effect size, or the size of the difference that you're trying to detect, which we call delta. All right? Larger delta equals easier-to-detect differences, lower beta risk.

What about variation? Variation plays a role here as well. If you have a lot of variation, like what I've drawn here, a lot being a relative term, then I've got a lot of overlap. It's hard to detect a difference because there's so much overlap. Even though the means are separated from each other, it's hard for me to tell that the population means are really different from each other because there's so much overlap. Whereas here, if I had these two populations with the same mean. But with lower variation, again, my beta risk is much smaller here because there's less overlap. Even though they have the same mean value, they have lower variations, and so their curves don't overlap so much. So that's another way of reducing your beta risk, if you're looking at things that have lower variation relative to higher variation.

Then finally, getting to the one we're really interested in, which is sample size, which is given by the letter n here. If all I do is take the same two distributions, but I gather more and more data from each of those different distributions, so instead of having nine data points from supplier A and 12 from supplier B, let's say I get 30 or 40 from each of those suppliers, then that actually reduces the variation in these curves also because these curves don't just use the standard deviation, right? They use the standard error of the mean, which is the standard deviation divided by the square root of n. If I increase n, I reduce my standard error, and from a hypothesis test perspective, that shrinks those distributions and reduces the amount of overlap. It's just like reducing the standard deviation.

Your uncertainty in where the true population mean lies goes down as you increase your sample size. You measure more of the population, you have a better idea of where the true mean of that population lies. Just by increasing your sample size, you can also reduce your beta risk or increase your power, which means increasing the likelihood of saying there is a difference if a difference does, in fact, exist.

That's the one we're going to really pay attention to here because we see that there's this relationship between n and beta, and that's really why we want to focus on sample size for hypothesis testing: to reduce that beta risk down to an acceptable level or increase our power, whichever way you'd like to think about it.

All right, so let's look at some calculators here to finish up. JMP has a lot of calculators that you can choose from. They're on the DOE menu. If you go to DOE and then down to the Sample Size Explorers there, you'll see all of these different options. If you're interested in looking at changes in power or beta risk relating to sample size, there's a whole list for that.

There's another list for confidence intervals, which is really just saying, given the size of your sample, what's the range of uncertainty for the quantity that you're evaluating, a mean, a standard deviation, et cetera? You're comparing your uncertainty in the knowledge of a value based on sample size. That would be the confidence interval calculators. There are a couple of others relating to reliability calculations and quality calculations. I'm not going to talk about those here. We're actually going to focus on the power for two independent sample means because we want to figure out, well, what's our power for these tests, or what's our beta risk for these tests? Let me do that.

All right, so we're going to come back to our example. We did this t-Test previously. We found that there is no difference, right? Our p-value is greater than 0.05 for 95% confidence, and we're saying there is no difference. The question becomes, well, do we believe that's true? Is it a trustworthy conclusion? Given the fact that I only had nine data points from supplier A and 12 data points from supplier B, should I really trust the decision that there is no difference between them?

We can do that by going into JMP. Let me go back to our data table here. We want to look at the power of this test. I'm going to go to the DOE menu, and then Sample Size Explorers. Then I'm going to go to Power in this case because I'm focusing on beta risk or statistical power, and we're doing a two-sample t-Test. Let's go to the Power for Two Independent Sample Means calculator. It opens up a window like this. All right. Okay.

It looks a little complicated. This graph probably looks like the Prediction Profiler. If you've done any regression analysis or modeling in JMP, you've probably seen a profiler graph like what we're seeing here. There are some settings that we want to talk through. You have to choose: are you doing a one-sided calculation or a two-sided calculation? In my example, I'm doing a one-sided calculation because I'm curious if supplier B is higher on average than supplier A. That's a one-sided test, so I'll change that to one-sided. I'm using 95% confidence, so I'll keep my alpha value at 0.05. Generally, you're going to keep your Population Standard Deviation Known to be No, because almost all the time you're dealing with a sample; you never have the entire population, fairly rare at least in industrial cases.

All right, so then there's these different things down here, and you notice you have to look at your total sample size, which is simply the sum of the number of samples that you have in each of your groups, in this case, supplier A and supplier B, if you will. How big is the difference to detect? Then what are the standard deviations? All of these terms that we just talked through and showed how they relate to each other in terms of power and beta risk, right? Let's go look at our data real quick and figure out these quantities. I'm going to just go back to my data table, and let's do a distribution analysis. I'm going to go to Analyze, Distribution, and I'm going to put Diameter in the Y field. I want to do this by supplier because I want to know the mean and the standard deviation for each of my suppliers. I'm going to put the Supplier column in the By field here and click OK. Here we see we have some summary statistics for each of the groups. I'm going to hide the Quantiles tables for now just so it's a little easier to scroll back and forth.

If you look at for supplier A, it had a mean of about 10.33 and a standard deviation of about 0.74, and similarly for supplier B, a mean of 10.74, so a little bit higher mean, but a standard deviation of about 0.64. I have nine data points for supplier A and 12 for supplier B here. We can come over to our calculator and actually plug those values right in down here on the profiler. My total sample size is 9+12=21. For 21, I had nine of those in group one, I'll call that supplier A, and then 12 in group two, it automatically changed because I put 21 over here. What is the difference I'm trying to detect in this case? If I go back to my distribution analysis, you really just need to take the difference between the means here, so 10.74 minus 10.33. It turns out that's 0.41 or 0.4114, if you want to get to the fourth decimal place. That's the difference between those. Then I'll put in the standard deviations for the two groups. Again, supplier A had a standard deviation of 0.74, and supplier B had a standard deviation of 0.64. When you just put in those numbers and don't do anything else, you get a power calculation right over here.

This is telling you the power of the t-Test that we have already performed. Given the fact that I only had 21 total samples, 9 for one group and 12 for the other, with this difference in average and these standard deviations, the power of that test is 36%. That means if there is a difference in the average between these two groups, there's only a 36% chance that you're going to detect that difference given this sample of data. Even though the t-Test says there is no difference, there's a pretty good chance that there could be a difference, and you're simply not detecting it with your test. That's the idea of these calculators. It allows you to figure out what is the power of the test that you've done. That's what we looked at here. But you can also go the other way and say, "Well, I want to make sure that I have 90% power because I want to make sure that if there is a difference, I detect that difference at least 90% of the time." We can do that. If you go up here to your power field, you can say, "I want 90% power," or 0.9. I'm accepting a beta risk of 10% or 0.1 in that case.

Then you have to choose something in your drop-down that you want to calculate, right? Given that I want a power of 0.9, I'm going to just go and say, "Give me my total sample size needed in order to make sure that if there is a difference in the average for these two suppliers, that I would detect that difference in my test." Notice that there's also this box here for the ratio of the group sizes. You can change that if you like. I'm going to keep it at 1.33. It turned out that I had 33% more data in group two than I had in group one for no reason. That's just how much data we had available, right? I'm just going to keep that there for now, but you could play with that number if you'd like. I'm going to click Go. When I click Go, JMP goes and calculates, well, if you want to make sure you have 90% power for your test, then you need a total sample size of 103, which is the ceiling of this number here. You have to round up to the next highest integer because you can't have a partial sample, presumably.

All right, so I need a total of 103, and if I have a ratio of number of data points between the groups of 1.33, then I need 44 data points from supplier A, and I need 59 data points from supplier B. If I go get that much data, and again, I'm trying to detect this size difference with these standard deviations, if those still hold true for those larger groups, then my t-Test has a 90% chance of detecting a difference if the difference does exist. That's how you can go both directions. We initially said, "I've got this data, what's my power?" You can do it proactively and say, "I want to make sure I have this much power in my test. How many samples do I need?" You can go either way there, and you can play with different quantities. If I don't want to look at total sample size, maybe you want to say, "Hey, given the number of samples I have, how small of a difference can I actually detect with 90% power?" You might calculate that and say, "Oh, well, it didn't change much because I used those numbers previously." But you can really calculate any one of these quantities based on the values of the other quantities for a given level of power that you're shooting for.

Reactively, you already have some data, and you want to evaluate what's the power of a t-Test given this data, or proactive, determine how many samples you need in order to make sure that your test is powerful enough to detect the change you're interested in.

That is one way to use these calculators. Again, JMP has a lot of other calculators. They're all very similar. Of course, they have different parameters in them. Whether you're comparing proportions or means, you're going to have slightly different parameters to worry about, but they all work in basically the same way. That's a way you can use these sample size calculators in JMP to evaluate. In this case, we focused a lot on power to control the risk of a wrong decision, and that's very useful from our perspective, so make sure you check those out in JMP. Thank you very much; appreciate it.



Start:
Wed, Oct 21, 2026 02:00 PM EDT
End:
Wed, Oct 21, 2026 02:45 PM EDT
Key Ballroom 10
Attachments
0 Kudos