Statistically Speaking: Answers to audience questions on design of experiments
Created:
The event attracted a lot of interest, judging by the number of questions the panelists received. While they did a great job addressing many of them, it was not possible to answer all in the time allowed. We’ve, therefore, decided to answer them in this post.
Q: Would you please relate sample size with possibilities of having a reliable innovation?
A: We can do better than that. The new Sample Size Explorers, introduced in JMP 16 and JMP Pro 16, were created to encourage exploration of the relationship between power, sample size, and other key metrics involved in determining how best to ensure your experiments are a success, as explained in this blog post by Caleb King (@calking). We recommend anyone interested in allocating resources or budgets between development projects, planning timelines, or simply understanding sample size calculations to check out this new feature.
Q: Do you have recommendations on resources to tackle the most challenging part of DOE: choosing factors and defining ranges?
A: This is a topic of much interest and for good reason: Choosing factors that affect your response and ranges large enough to detect an effect are the critical first steps in planning and executing a successful DOE. At this point, it would also be helpful to define success. A DOE is successful if it results in a model that encompasses enough understanding of the process to satisfy a business need: “How can we set our factors to achieve this level of quality in our output?” for example.
A DOE is also successful if it identifies significant factors, quantifies the variation not captured by the model, and points the way towards a follow-up experiment (an augmented DOE). An experiment that fails to identify a single significant factor would constitute a failure, although the learning would be that the wrong factors were chosen, or the ranges weren’t set wide enough. Any good resource on DOE includes some discussion on how to choose factors and ranges, although it is often not much more than “choose factors that are likely to affect your response, and ranges that are wide enough to see an effect if one exists, but not so wide as to cause the runs to fail.” This is because the choice of factors and ranges is entirely dependent on the specifics of the situation in question. Therefore, the best people to determine these are the process experts, rather than statisticians with great understanding of DOE but limited understanding of the application. Having said all this, take a look at this discussion on the JMP User Community, which includes much useful insight on the matter.
Q: How do you secure the reliability of "historical" data or data that are coming from "external sources"? Someone once said "never analyze data you haven't collected yourself." How do you overcome this in order to be sure your historical data are reliable, meaningful, the measuring system used is capable to assess the process variations to a minimum, etc.?
A: I don’t know who said that about data you haven’t collected yourself, but I don’t agree with it. In her keynote, Dame Sally Davies discussed the multitude to ways data is being generated and collected, as well as the risks for analysts “working in silos” using different data sets that are “not informing each other.” In her words, there is currently “greater engagement between scientists and data than ever before,” and limiting our analysis to data we collect ourselves would severely impede progress.
Of course, we should always be skeptical of data, just as we are skeptical of the models generated from the data. But treating something with skepticism is not equivalent to dismissing it outright. Nothing is perfect, and this certainly true for data and models, as George Box succinctly expressed in his famous aphorism. It does not follow that because something isn’t perfect or can’t be trusted that nothing can be learned from it. We are right to ask questions about the source or accuracy of the data. Any conclusions drawn from its analysis should be considered alongside our own process knowledge and subject matter expertise, and never accepted blindly.
Q: Would you be willing to elaborate on the common mistakes that new DOE users should be aware of?
A: One of the mistakes people make is over-focusing on the design rather than why they’re doing a DOE in the first place. It happens far too often that someone will begin by first choosing a design, and then compromising their experimental needs to fit that design. The Custom Design platform in JMP lets you create optimal designs tailored to any situation. Additionally, CPI’s Rachel Findlay gave some great advice during the event: “Share your learning and your mistakes” with colleagues. The more we promote DOE within our organization, the more the organization will benefit.
Q: What is the most important factor to keep in mind when selecting a DOE?
A: There are many important factors in selecting the right DOE, beginning with the business question we are trying to answer. What exactly are we aiming to accomplish by executing a DOE in the first place? Do we want to minimize defects or increase yields? Maybe we want to incorporate material from a new supplier into our process, figure out how to speed production or decrease costs. DOE is often an iterative process of incremental learning, and understanding our target objective is critical when determining whether to experiment further. We also need to ask how many variables might be affecting our process, as well as our timeline and budget. Once all of this is clear, we can begin the process of choosing a design.
For example, if we already know that a small number of variables are significant and we’re looking for the best combination of settings to maximize output, we may choose to opt for a Response Surface Experiment. Alternatively, a large number of factors and a tight budget may point the way towards a Definitive Screening Design. Well-defined answers about our situation, our constraints, limitations, and what we are trying to solve will point the way towards a design, or series of designs, that ensure our requirements are satisfied. It should also be mentioned that the Evaluate Design and Compare Design platforms are useful tools to help make the best possible decision.
Q: DOE can be a hugely powerful technique… when used correctly…. but I have equally seen too many examples of heavily overfit models which are not predictive, and this often isn’t obvious to novice statisticians/biologists. How do we better help novices to understanding the tools they are using and what the outputs actually mean?
A: There are plenty of resources to help new experimenters get up to speed with design of experiments very quickly. They also cover the various ways of removing insignificant terms from a model and using validation to avoid overfitting. Ultimately, the best proof of a model is verification with independent data: Is the proposed prediction as the solution determined from our model supported by verification runs? An excellent place to start is the Statistical Thinking of Industrial Problem Solving online course, which includes a module on DOE. There also several textbooks available here. Those new to both JMP and DOE can go through the New Users Welcome Kit and Design of Experiments Intro Kit. The Mastering JMP webinars include DOE as well. Finally, the JMP User Community is a powerful resource where anyone with a specific question can post it as a discussion topic; questions are usually answered by one of the 30k community members within a few hours.
Q: How can artificial intelligence complement DOE?
A: The topic of artificial intelligence (AI) was discussed in a previous Statistically Speaking event. To say that it is a very promising field does not quite do it justice. It has already delivered on many of its promises. However, the greatest risks arising from AI relate to our over-reliance on it to solve all problems, or to blindly accept the solutions it purports to present. The adage that “correlation does not mean causation” highlights AI’s shortcomings. To separate the two and get to the heart of understanding of any process, DOE is needed. Of course, DOE is not able to drive a working process autonomously. I would say there are opportunities for both AI and DOE to complement each other.
To learn more about how DOE can speed innovation, please sign up for these free events in November:
- English: www.jmp.com/doe-en
- French: https://bit.ly/FormationDOE
- German: https://jmp.com/doe-de
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.