A Cringeworthy Statistic Statement: #2 in a series

[#1]

Thanks again to all who joined in my first "Cringeworthy" discussion topic.  Very interesting discussion

points were contributed!  I have published a blog post on why I cringe when I hear that first statement.  

Here is a second scenario and statement that makes me cringe:  

A company experienced a problem with yield on a production line.  The working hypothesis was that it was related to a difference in subassemblies received from two different suppliers.  An engineer is asked to do a t-test, from which the p-value comes back as 0.001.  The manager immediately says "That is more than 95% significant!  Go figure out why they are different.  We have to get these two suppliers to be providing the same parts!"

What is wrong with the manager's thinking?

JMP #Cringeworthy

[#2]

This one is soooooooooo familiar. My last dance at Eastman Kodak Company was as a supplier quality manager. Here's a few thoughts...

If the problem is yield, what is the practical functional relationship between the subassemblies and yield? Is one supplier's material more problematic wrt to yield? What are the specifications that have been communicated to the suppliers? Are they identical? The fact that the subassemblies MAY be different does that make either one or both problematic wrt to yield? Right now it's a '...working hypothesis...'. On what basis has this working hypothesis been established? I've seen all these:

a. Urban manufacturing legend.

b. Because John says so.

c. We saw this yield issue last year...turns out it was the supplier's fault.

d. We ran a designed experiment that was confirmed through subsequent study.

[#3]

Always good to hear your point of view, @P_Bartell !  Keep 'em coming!

[#4]

I'll let the rest of the group argue the statistical issues associated with the Manager's statement.

Similar to Pete's thinking, here are my thoughts:

1. How is yield measured?  Is the measurement system adequate? What is actually failing?  What is the yield?  Is the "problem" with yield consistent, rare or patterned?

2. Yield is a terrible measure if you are trying to understand failure mechanisms or causal structure.  It can be too aggregate to get to the fundamental issues.

3. I don't see a hypothesis?  There is no explanation as to WHY there are differences in supplied subassemblies, nor why those differences would impact yield.

4. t-test of what?  What comparisons, over what time period, what samples...etc.?

One of my favorite quotes from Dr. Deming:

“Analysis of variance, t-test, confidence intervals, and other statistical techniques taught in the books, however interesting, are inappropriate because they provide no basis for prediction and because they bury the information contained in the order of production. Most if not all computer packages for analysis of data, as they are called, provide flagrant examples of inefficiency.”

Deming, W. Edwards (1975), On Probability As a Basis For Action. The American Statistician, 29(4), 1975, p. 146-152

But I will say, when you've got issues, (I'll paraphrase from Brian Joiner and add the last bullet), you can:

1. Improve the system,

2. Distort the system,

3. Distort the data, or

4. Blame someone.

[#5]

Excellent points @statman !  I enjoy reading your replies!

[#6]

I'm enjoying this discussion and learning something. Thanks to all contributors!

(I liked the blog post...the vials part...brought it together for me.)

[#7]

Thanks @Craige_Hales !  Glad you are enjoying the discussion.  (I certainly am!)

[#8]

I think there is some good news here: clearly the manager and team is interested in letting the data help them make decisions.  With some coaching maybe they can ask a few more questions and make a positive impact on their process.   They established that the parts are different: just asking the manager 'which one should we standardize to' might be all it takes to prompt more evaluation:

• Is the yield better from one supplier than the other?  If so what is different so you can standardize on the better part.
• Does yield get better in campaigns of a single supplier? Are there processes or systems that need to be adjusted based on supplier?
• Can the people on the manufacturing line tell the difference between the parts?  What do they think is different about them and why does that matter?

[#9]

Thoughtful reply, @ih !  Thank you!

[#10]

They should try and rule out confounding factors - if products processed on Subassembly A usually process through Machine X, and products processed on Subassembly B usually process through Machine Y, the actual driver could be that Machine X and Machine Y are not matched. 

Without a properly controlled sampling, that tests Subassembly A and Subassembly B with all other factors identical, (or failing that, at least with any differences properly accounted for and subtracted out,) it would be premature to conclude the subassemblies are the driving factors.