At JMP, we are fans of Dennis Lendrem, his books, his LinkedIn posts and newsletter, Apes in Lab Coats, his passion for advocating better science, and his delightful British wit. We had more questions than time to answer them in this Statistically Speaking episode, so Lendrem kindly provided answers for this blog post. We are delighted to emphasize his message of needing better science and faster problem solving.
In your collaborations with scientists showing them how DOE can make their experiments more efficient and effective, have any of them been so blown away by the power of DOE that they were motivated to learn about it and use it on their own?
A few. But don’t get your hopes up. For most, it’s like showing fire to a caveman. There’s initial awe, but the next day they’re back rubbing sticks together. Why? Because it’s not habit. It’s not instinct. It’s not how they were trained. The ones who do convert tend to be the ones who already suspected something was wrong. They’d spent years foraging and getting nowhere.
We see low-density foraging in engineering, software testing, and other fields – should we start teaching smart experimentation in high school and finally displace the one-factor-at-a-time approach?
Yes. In fact, we should apologize for not doing it 30 years ago.
One-factor-at-a-time is scientific junk food. It’s simple, satisfying, and utterly misleading. By the time students get to university, it’s already baked in. They don’t even question it. We need to acknowledge that the world is complex, and that single-variable thinking is a lie we tell ourselves to stay comfortable. It’s not about statistics – it’s about curiosity, structured discovery, and structured exploration. That’s a mindset worth instilling early. (See also the related supplementary question below.)
To your point about kind vs. unkind learning spaces, with increasing complexity across so many domains, do you see more opportunities for combinatorial testing and covering array designs? For example, software is becoming more ubiquitous – think self-driving cars.
Absolutely. The learning space is no longer kind – it’s high-dimensional, sparse, and full of traps. You can’t rely on intuition or single-variable fiddling. That’s how you miss the lethal interaction. Combinatorial testing and covering arrays aren’t optional anymore, they’re survival tools. Software testing figured that out. Engineering is catching up. Science is still in denial. And yes, self-driving cars are the poster child with millions of possible input combinations. You either design your tests, or you let the real world do it for you – with pedestrians.
The $5.4 billion outage/crash in 2024 wasn’t a bug; it was a breakdown. One missing input check and BOOM, flight cancellations and hospitals offline. This is why combinatorial testing isn’t luxury, it’s risk‐mitigation. Crowdstrike is facing a $500m lawsuit from Delta Airlines with more to follow.
One variable at a time? What about blocking and the robots, as well as fractional factorial DOE and confounding?
We need a law prohibiting one-variable-at-a-time experimentation.
Blocking is key, even in robotics, because equipment can fail and things can go wrong. See Murphy’s Law in Gorillas in Suits.
Fractional factorials significantly reduce resources while making a modest sacrifice in terms of information loss.
And both blocking and selecting fractions help guard against confounding.
In three different industries, I've experienced product development that aligns the team to a set of boundaries and target timeframe, and the collective thinking ultimately works against being able to effectively use designed experiments. The reason being that the rapid collection of data points (without design, the scientist determined conditions) had that self-fulfilling prophecy and became the lead and only approach. I've had great statistical science support, whether internal or external, but I'm keen to hear other ideas of how to have greater influence on scientific thinking, especially in the product development realm.
R&D is broken; flawed experimentation is endemic. Know that you are not alone. Keep the faith. Be part of the New Wave. Seek out the company of those who know there is a better way through events like these. And see below.
Is your data set for random experimentation vs. scientific foraging available online (JMP Live?) I'd love to use it as an example within my own group (fully credited of course!).
Not yet. It will be published as an Appendix to the third book in the Apes in Lab Coats trilogy, Apes & Anoraks, which has a launch date of Nov. 1. But for a more complete and up-to-date copy of the full data set, please contact Harrold Fellerman, who has a wealth of new data, at Newcastle University.
I think one of the misconceptions about the DOE approach is that it "costs" more or that it requires more "conditions" (something that I've heard over and over). I have found the exact opposite! It is the most efficient way to the result because you can model and predict the effect of factors on responses without running every possible combination. It is iterative! We don't start with a full factorial – we end with it!
Firstly, if you want to minimize the number of conditions and resources consumed while maximizing the amount of information about what is happening in a multidimensional space, then the only way to do that is using a designed approach. That’s what designs do. Period. The only way to do it with fewer resources is by sacrificing information. End of story.
Secondly, if you look at how many resources are consumed in the field using traditional scientific foraging, then it consumes more resources in order to generate poorer information. What’s more the resources you need to make available are more unpredictable. Sometimes you stumble upon something that kind of works and you can shove it out the door quickly. Other times, you end up floundering around until you run out of time before shoving it out of the door. In both cases, chances are that the information is incomplete and that your solution falls over at the drop of a hat, which is a nightmare scenario for R&D management. See Gorillas in Suits.
Do you find that it is hard to deliver this message in person to a scientist who wants to trust his/her intuition? Talking about it in the abstract seems easier than saying that the scientist has to change their approach.
Yes. Not everyone gets it. We run with the scientists who do. They self-select as lead users. We’re just asking them to suspend their disbelief while we work them through that first study. And we work with them to generate the successful case studies that will sell design to the organization. Nothing sells better than success. And DOE is extraordinarily successful.
Is there any work underway to develop an objective measure of the application of multidimensional DOE and statistical best practices in a study? In other words, a rating system that does not try to rate the outcomes or conclusions of the study, but a rating of whether the study objectivity used multidimensional DOE and statistical best practices.
This is a really good idea. Currently, it’s patchy. The EQUATOR network (e.g., STROBE, STARD, CONSORT) have been proactive, though they tend to be clinically oriented. And there have been efforts in the past that are not in the public domain, for example, GSK produced a series of single-page DOE Cheat Sheets for internal use by its research scientists.
The need for more collaboration with scientists and people knowledgeable about DOE is clear. How can we encourage more of it?
Creating a kind learning environment, in which people are given opportunities to learn in safety, is key. Events such as webinars, workshops, conferences bring together like-minded people to help them develop a shared understanding of real-world applications. Other ways include shared collaborative software tools; easy visualization of results and mapping of the multidimensional spaces of interest; helping users to lead the change in the mindset in their organization.
I feel convinced that I should be using DOE. But I am not sure where to start! Any advice?
Keep it simple. Find someone who has done it before. Get them to hold the nail when you wield that hammer for the first time. Be conservative. Stick with a relatively safe design. And remember Moore’s Observation: Almost any design is better than no design at all.
There are so many founders in the biotech industry and principal investigators in academia that have built their whole careers and reputations using scientific foraging. They are still in power. How do we break through? It seems impossible from the "bottom up." Thoughts?
All change begins with one person. And all change is bottom-up until that one person rises to the top to drive change top-down. Remember The Story of Imo, the juvenile Japanese macaque that introduced the troop to washing sand from sweet potatoes prior to eating them. The new trend swept through the ranks, but it was the dominant males who were the last to adopt the behavior.
I'm curious about your opinion on how best to remedy the situation of underteaching DOE in science? Sneak it into high school curriculums? Target Ph.D.-level courses? Deans? Or forget it and let industry fill in the blank?
Right now, we need to do all of those. We need the next generation NOW. Until then: High schools? Yes. Taught component of Ph.D. courses? Yes. Deans? Too late for most, but one or two deans for research and innovation might bite. Industry? Yes.
We need to drive DOE earlier in the development life cycle – to children in schools. Kids get it. Nobody has yet told them that they should stick to one variable at a time. They like DOE teaching aids – like helicopters and catapults. Show them things like the Paradox of the Self-Fulfilling Prophecy and they will laugh when you tell them that most scientists use one variable at a time approaches. We need to teach kids that the world is complex and that single-variable thinking is a lie that adults tell themselves to stay comfortable and to avoid being overwhelmed. It is about building a scientific mindset: exploiting natural curiosity, providing opportunities for structured exploration, and encouraging scientific discoveries.
I completely agree that DOE is starting to percolate through to industrial biosciences but is much rarer in academia. Part of this might be due to the high-cost threshold that remains for robots, automation, and consumables. How can academics and students with limited budgets more effectively approach DOE?
You don’t need robots. If you have access, great, but you don’t need robots. If you want to minimize consumables, then DOE maximizes information while minimizing costs. Scientific foragers underestimate the cost associated with stumbling around in multidimensional darkness.
Can you talk about how sequential experimentation can be helpful, but also a bit dangerous, based on your example of scientists wanting to jump to conclusions quickly?
Sequential experimentation can be valuable when the cost of each experiment is totally outrageous. But parallel experimentation is the ideal, which is why robots can be so useful. But in science, the longer the sequence, the more vulnerable to confounding with changes over time, the more opportunities for analytical drift and other trends, and the more likely that Murphy’s Law will strike. (See the chapter on Murphy’s Law in Gorillas in Suits.) Keep sequences short, exploit blocking, check for run effects.
Did Dennis reference Ronald Fisher’s book on DOE?
No longer in print, but of historical interest:
- Fisher RA 1935 The Design of Experiments Publisher: Oliver & Boyd, Edinburgh.
But for a more modern and relevant treatment of DOE:
- Montgomery DC 2020 Design & Analysis of Experiments Publisher: Wiley, New York.
Do you expect the result from JMP DOE to be different from any other app? If so, why?
Rather than allow software to be an obstacle, we used to run DOE training in the client’s chosen software. As a result, we got to know most of the DOE packages out there (Modde, DesignExpert, SPSS, BMDP and various R-based DOE packages). While computationally JMP DOE yields similar results to other packages, it excels in user experience: ease-of-use, data visualization, and interpretation.
We thank Dennis for not only answering the questions we didn’t have time to answer in the livestream but also expounding on questions he did answer. We hope you’ll check out the enlightening and entertaining on-demand Statistically Speaking. We are looking forward to the third book in the Apes in Lab Coats trilogy, due in November.