Learning from my mistakes in experiments and data analysis -- Part 1: The scenario

[#1]

I hope to help you avoid some mistakes in data analysis.I’ve been a practicing engineer for many years now. Along the way, I’ve seen many tests performed, and many different ways to analyze data. More often that I care to admit, I’ve been on teams that have made one or more mistakes on the job. The end result was a cost to the company in time and money.

I thought some readers might like to learn from my mistakes, so I came up with a fictitious scenario that embodies the various problems I’ve come across in my work.

Although my scenario is a mechanical engineering example, it is both simple and generic enough that you should find it easy to translate to your particular discipline. We will explore this scenario in this and upcoming blog posts.

The Scenario

We are trying to improve the strength of a product. We have identified a treatment “A” that we believe will improve strength. Our management wants us to prove it before implementing the change in production.

We create three samples with Treatment A and measure the strength of each. We also collect three standard (or Control) samples and measure the strength of each. Here is the data:

Using the Fit Y by X platform in JMP (see below), we find that the average strength from the Treatment A sample is slightly higher than the Control. However, the t-test says that we cannot say (at 95% confidence) that the population mean of Treatment A will be any different than the population mean of the Control. We stop the test and start looking for another way to improve product strength.

This seems like a simple test with simple-to-interpret results, but I contend that mistakes are being made in the above scenario, and that we might be throwing away a perfectly good solution. 

What do you think they might be? Stay tuned for my next post in this series, coming in about a week!

[Peter_Bartell]

What do we know about measurement system variation? Perhaps much of the variation is due to the measurement system and NOT a difference in the properties of the two populations...making distinguishing the true but unknown population values for each problematic.

[JerryFish]

Excellent @Peter_Bartell !  This is one of the mistakes commonly made.  I'll talk more about it in a future post.  Keep the ideas coming!

[Peter_Bartell]

OK...one more...Often I was the statistician on a new product development team and was the 'go too' resource for all things DOE related to the team's work. Usually we'd design our experiments including a discussion of all the logistical stuff associated with the experiment in team meetings. Then a subset of the team would be tasked with running the experiment themselves, or sometimes handing off the design (my JMP data table on a piece of paper usually) to some uninvolved (in the planning) group of people to actually conduct the experiment.

On more than one occasion the folks actually running the experiment would ignore things like randomization (Gee...it's a pain to switch between low and high temperatures for the next 8 hours overnight when I'm actually running this experiment...I'll just run all the lows together, then all the highs...What can it hurt? Plus I'll be more productive because I don't have to wait for temps to equilibrate from run to run.)

Or take it upon themselves to NOT use the levels for a specific factor because, "Well every other time I run an experiment for these folks they use these levels...they must have made a mistake on the spreadsheet".

Or something goes wrong...like a bag of raw materials from a supplier is determined to be not useable...so they change to a different supplier's 'identical' stuff that happens to be in the storage room. "What they don't know won't hurt them."

And on and on with all sorts of 'improvements'.

I bet you can make up some others?

So I learned pretty quickly that the best way to insure these sorts of procedural failure modes didn't occur was to insist on being physically present during the conduct of the experiment to observe...and maybe even lend a hand. Usually I found the technicians very welcoming of this sort of interest and help in their work. Plus I got to learn alot about their jobs and our processes by actually "Going to Gemba". A lean concept which, loosely translated means , 'go see where the work is done'. Sure it meant for some late nights...running experiments on graveyard shift and all...but at the end of the day we were convinced this was the best way to run our projects.