A Cringeworthy Statistic Statement: #3 in a series

[#1]

Welcome back for the third Cringeworthy discussion topic!  Many thanks to those that contributed to the first two topics.  Please feel free to join this discussion as well!

Today's Cringeworthy Statement should be obvious, though you all have surprised me with great insights in the past that I hadn't considered!  Here it is:

An engineer is tasked with running a test on a particular process output.  The goal is to learn about the average performance of the process.  The engineer decides to measure the process output three times.

After completing the test, the engineer reports the following in a stakeholder meeting:

"The process output is 11.327384, based on averaging 3 independent measurements."

What is cringeworthy about this statement?

#cringeworthy #measurements #msa

[#2]

For starters:

• Unit of measurement is not given
• How are the measurements independent?
• Measure of dispersion is missing. Perhaps reproducibility / repeatability does really only have an effect 5-6 digits after the comma, but one could easily and would intuitively doubt a result like this. 

[#3]

Units! Somebody came back from a vacation in Germany several years ago. I asked about filling up the tank in the rental car. "Gas was about half as expensive in Germany as the USA." Comparing Germany's 1.653 EUR/Litre vs USA's 3.763 USD/Gallon, I guess it did seem like that.

(No time to do the math? The reverse is true. At least in late 2021. Your mileage may vary.)

[#4]

If the measurement instrument is accurate enough to report the result to 6 decimal places, then it is surely enough to provide the degree to which the 3 measurements differed.  So, yes, I want to see some measure of dispersion at a minimum.  Then, of course, all the additional caveats (how do we know they are independent?  how are the measurements taken?  were the conditions the same?  etc.).

[#5]

More fun...there are many issues with the statement.

Head in the oven, feet in the freezer, on average OK...

An average without the context of variation is a worthless statistic.  Not even sure if it is the appropriate measure of central tendency.  

How does reporting this statistic have anything to do with the "goal".  There is no description of over what changing factors in the process is this measure indicative of. There is no sampling plan provided (or description of the process associated with when the 3 measures were taken).  

There is no estimate of measurement precision.  Just because there are many decimal places reported means nothing (it could simply be  a reporting issue, we don't know the measurement unit). we have no knowledge of the  accuracy, precision, discrimination or stability of the measurement process.

Not sure what you can do with 3 measurements unless they possibly indicate an issue so as to motivate you to investigate.

[#6]

"...3 independent measurements." Something that isn't clear is sample size wrt to the process. Is it 1? Is it 3? Not stated.

[#7]

What is the process output measuring, what is the target, and what are the units?  What are the conditions, e.g. dates, shifts, and other relevant conditions? What are the sample sizes for the three measurements?  Can each sample mean be considered to be different from the others or not (cannot reject the null hypothesis in Student t test or do the 95% confidence intervals for the sample means overlap?).  Only after "cannot be considered different" can you justifiably pool the samples and calculate a grand mean (with an appropriate number of decimal places).  Then should included the 95% confidence interval in the statement to stakeholders so they understand the significance of the statement.

[#8]

not much you can say with a confidence interval based on n of 3. A very wide interval indeed!  Same thing if we were making an inference on a specific proportion of the population ("Reliability") at a particular level of Confidence (e.g. 95%).   "Law of small numbers".  The smaller the sample size the lower the precision we have on the estimate of a sample statistic. 

Also, the confidence level we choose should be based on risk which should be based on our practical assessment of the problem and the potential consequences of failure to meet a given criterion or set of criteria. 

[#9]

@JerryFish I'm hoping the blog covers this independent thing. Right now, I'm thinking it means three separate rulers, but I'm not sure why that would be a good thing.

[#10]

As others pointed out, there are many details missing. Without the unit, boundary conditions and other details the statement is simply useless. It also would be an important information before doing the thest for the output, what the number is needed for.