For this third installment of my "Learning from my mistakes" blog series, I’d like to wrap up my thoughts on the preliminaries of planning a test. We’ve already talked about what we can expect from a t-test in general. So, we now know that we need to establish a confidence level, establish a target for improvement as compared to current condition, and establish a direction for the improvement (greater than or less than the current condition). Now let’s talk about some other test considerations, and then wrap up with Ask the Complete Question.
Other Considerations Prior to Test Planning
Recall from the original scenario that the engineer decided to draw three samples from each of the populations to determine whether the population mean changed between the New Treatment and the Control. Why did he choose three samples? Why not five? Or 10? Or 2? Perhaps it was because “We’ve always done it that way.” Perhaps there were other considerations that haven’t yet been described.
From a purely statistical/mathematical perspective, more is always better when choosing a sample size. After all, if we could sample ALL parts from a population, then we wouldn’t have to make any assumptions about the mean or variance of the population. We would know it!
Unfortunately, there are many practical considerations in our daily work that affect sample size selection. And even more unfortunately, I have myself been guilty of not considering all of the constraints and impacts of executing a given test. It would be much wiser to consider these factors before beginning the test, rather than waste time out of haste.
Below I offer a few things that might affect how you plan your test:
- What is the budget for this test?
- Management is always interested in budget!
- When do we need an answer?
- If an answer is needed by the end of the week, but the test will take at least a month to complete, then we need to rethink the question that we are trying to answer, and the methodology that we are using to answer it.
- How many test samples can we afford to build?
- This might involve cost of the parts themselves, as well as time and scheduling to build them. Are we shutting down a production line to build these parts?
- Is “Batch” going to be important to us (assuming parts are made in batches)?
- If so, consider sampling across multiple batches. Of course this impacts the time to produce the initial test parts, and the time until we have our final results.
- How many samples can we actually test?
- Is the lab that tests the samples busy with other things? Does it cost money to send the samples out to an external lab?
- Are we really planning to test the right parameter, in the right fashion? In our scenario, we want to increase the “strength” of our parts. There are many ways to measure strength (e.g., pull tests, repeated cycling, impact testing). Or perhaps we really need to be measuring elastic modulus, or durometer, or some other parameter. Don’t just rush in and start measuring!
- Can we even measure the “important difference” that we want to detect, using our current test equipment?
- A Measurement System Analysis helps understand your gauge variation. I’ll cover more about this in my next blog post, including what to do if your gauge isn’t good enough.
- Do you know the normal part-to-part strength variation in your Control parts? Do you expect the new parts to have the same variation? Is the variation expected to be normally distributed?
- These questions play an important role when we plan the test and analyze results.
- Besides improving the average part strength, is maintaining existing part-to-part variation important to the product? Does the variation also need to be improved? Conversely, can it be expanded?
- If we improve the average strength of the parts but increase the spread (variation) of the parts, will we actually be making more scrap?
- Usually we are worried about processes that are out of control, and/or making bad parts. But if we have an in-control process that is making consistently good parts, how much are we spending to maintain that control? Could we tolerate larger variation in the New Treatment parts and save some money in process controls?
All of these considerations fold into selecting the sample size for our hypothetical t-test. (I’m sure you can think of many more considerations that need to be discussed before the test is planned as well.)
So, I’ll call this mistake: Not adequately considering all factors that could affect test planning.
What Is the Complete Question That We Are Trying to Answer?
You can mitigate all of the above discussion and mistakes by setting aside time to “Ask the Complete Question.” (Here, I need to give a shout-out to Bill Kappele, my mentor and friend, who taught this method to me many years ago.)
“Ask the Complete Question” means going through all of the discussion above (and in Parts 1 and 2) and ensuring that everything has been considered, so we are measuring the right quantity, defining constraints on our testing, defining how much difference is important to us, etc.
In our scenario, let’s say that we have talked to all interested parties and stakeholders, and we rewrite the question as follows:
“With 95% confidence, can we say that Treatment A improves the average strength of the part by more than 1 unit, as compared to the current process, without increasing variation of the process?”
Following this, we would list any constraints. A few of these might be:
- We can afford to build and test a total of 40 parts (total of all parts from Treatment A + Control <=40)
- We can ignore the effect of Batch
- We need the answer in one month (just enough time to complete this blog series!)
Confirm That We Have Asked the Complete Question
Over my career, I’ve seen engineers spend a lot of time and resources in the lab trying to get to the bottom of some issue, only to find that no one cares about the results. The problem is not in the initial thinking and test planning. The problem is not in the test execution. The problem is that the engineers didn’t check to make sure ahead of time that the right/complete question was being answered.
So, once we’ve done all this forward thinking and test planning, it is a great idea to do one last thing: Ask your manager if he/she agrees that this is the question that needs to be answered. In fact, if the test takes a good bit of time to run, it’s a good idea to periodically check in with management to make sure conditions haven’t changed. No one wants to waste their time running a useless test!
Coming Up
I’m planning to touch on all of the following subjects as we go along. Hope you can join me for the whole series!
- Part 1: Laying Out a Typical Scenario
- Part 2: What does the t-test really tell us?
- Part 3: Other test considerations, and Ask the Complete Question (this post)
- Part 4: Is Your Gauge Capable (and what to do if it isn’t)?
- Part 5: Planning and Running the Test: From Sample Sizes to Best Practices
- Part 6: Analyzing the Results (including Garbage In = Garbage Out)
- Part 7: Summary