topic Re: A Cringeworthy Statistic Statement: #2 in a series in Discussions

A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Fri, 09 Jun 2023 00:41:44 GMT

Thanks again to all who joined in my first "Cringeworthy" discussion topic. Very interesting discussion

points were contributed! I have published a blog post on why I cringe when I hear that first statement.

Here is a second scenario and statement that makes me cringe:

A company experienced a problem with yield on a production line. The working hypothesis was that it was related to a difference in subassemblies received from two different suppliers. An engineer is asked to do a t-test, from which the p-value comes back as 0.001. The manager immediately says "That is more than 95% significant! Go figure out why they are different. We have to get these two suppliers to be providing the same parts!"

What is wrong with the manager's thinking?

#Cringeworthy

Re: A Cringeworthy Statistic Statement: #2 in a series

P_Bartell — Mon, 08 Nov 2021 13:59:29 GMT

This one is soooooooooo familiar. My last dance at Eastman Kodak Company was as a supplier quality manager. Here's a few thoughts...

If the problem is yield, what is the practical functional relationship between the subassemblies and yield? Is one supplier's material more problematic wrt to yield? What are the specifications that have been communicated to the suppliers? Are they identical? The fact that the subassemblies MAY be different does that make either one or both problematic wrt to yield? Right now it's a '...working hypothesis...'. On what basis has this working hypothesis been established? I've seen all these:

a. Urban manufacturing legend.

b. Because John says so.

c. We saw this yield issue last year...turns out it was the supplier's fault.

d. We ran a designed experiment that was confirmed through subsequent study.

Re: A Cringeworthy Statistic Statement: #2 in a series

statman — Mon, 08 Nov 2021 14:55:33 GMT

I'll let the rest of the group argue the statistical issues associated with the Manager's statement.

Similar to Pete's thinking, here are my thoughts:

1. How is yield measured? Is the measurement system adequate? What is actually failing? What is the yield? Is the "problem" with yield consistent, rare or patterned?

2. Yield is a terrible measure if you are trying to understand failure mechanisms or causal structure. It can be too aggregate to get to the fundamental issues.

3. I don't see a hypothesis? There is no explanation as to WHY there are differences in supplied subassemblies, nor why those differences would impact yield.

4. t-test of what? What comparisons, over what time period, what samples...etc.?

One of my favorite quotes from Dr. Deming:

“Analysis of variance, t-test, confidence intervals, and other statistical techniques taught in the books, however interesting, are inappropriate because they provide no basis for prediction and because they bury the information contained in the order of production. Most if not all computer packages for analysis of data, as they are called, provide flagrant examples of inefficiency.”

Deming, W. Edwards (1975), On Probability As a Basis For Action. The American Statistician, 29(4), 1975, p. 146-152

But I will say, when you've got issues, (I'll paraphrase from Brian Joiner and add the last bullet), you can:

1. Improve the system,

2. Distort the system,

3. Distort the data, or

4. Blame someone.

Re: A Cringeworthy Statistic Statement: #2 in a series

Craige_Hales — Mon, 08 Nov 2021 15:14:45 GMT

I'm enjoying this discussion and learning something. Thanks to all contributors!

(I liked the blog post...the vials part...brought it together for me.)

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Mon, 08 Nov 2021 15:27:12 GMT

Thanks @Craige_Hales ! Glad you are enjoying the discussion. (I certainly am!)

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Mon, 08 Nov 2021 15:27:55 GMT

Excellent points @statman ! I enjoy reading your replies!

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Mon, 08 Nov 2021 15:28:38 GMT

Always good to hear your point of view, @P_Bartell ! Keep 'em coming!

Re: A Cringeworthy Statistic Statement: #2 in a series

ih — Tue, 09 Nov 2021 16:03:50 GMT

I think there is some good news here: clearly the manager and team is interested in letting the data help them make decisions. With some coaching maybe they can ask a few more questions and make a positive impact on their process. They established that the parts are different: just asking the manager 'which one should we standardize to' might be all it takes to prompt more evaluation:

Is the yield better from one supplier than the other? If so what is different so you can standardize on the better part.
Does yield get better in campaigns of a single supplier? Are there processes or systems that need to be adjusted based on supplier?
Can the people on the manufacturing line tell the difference between the parts? What do they think is different about them and why does that matter?

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Wed, 10 Nov 2021 18:44:36 GMT

Thoughtful reply, @ih ! Thank you!

Re: A Cringeworthy Statistic Statement: #2 in a series

markschwab — Wed, 10 Nov 2021 19:36:31 GMT

They should try and rule out confounding factors - if products processed on Subassembly A usually process through Machine X, and products processed on Subassembly B usually process through Machine Y, the actual driver could be that Machine X and Machine Y are not matched.

Without a properly controlled sampling, that tests Subassembly A and Subassembly B with all other factors identical, (or failing that, at least with any differences properly accounted for and subtracted out,) it would be premature to conclude the subassemblies are the driving factors.

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Tue, 16 Nov 2021 13:50:15 GMT

Excellent point, @markschwab ! Thanks for the reply!

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Tue, 16 Nov 2021 21:17:03 GMT

Thanks to all that replied to this discussion thread. (And if you are just now seeing it, feel free to continue the discussion!)

If you are interested in my take on why I cringe when I hear things like this, see my latest blog post.

And if you liked this one, check out my 3rd "cringeworthy" statement discussion!

Re: A Cringeworthy Statistic Statement: #2 in a series

MarkDayton — Tue, 16 Nov 2021 22:42:26 GMT

Perhaps I’m missing something, it seems the manager is equating significantly different yields with the sub-assemblies being different. As has been pointed out, there are many factors that affect yield which have nothing to do with form, fit, or functional differences in the assemblies.

Re: A Cringeworthy Statistic Statement: #2 in a series

JerryFish — Wed, 17 Nov 2021 14:00:49 GMT

Thanks for the reply, @MarkDayton . Yes, my thought when writing this statement was to point out that people jump at a "Significantly Different" result without thinking through what that means, whether it is important, etc.