Meet the Author: Doug and Brad

In this episode of Meet the Author, Anne Milley interviews world-renowned DOE experts Doug Montgomery (Arizona State University) and Bradley Jones (JMP). Ten years in the making, their recently published book, Design of Experiments: A Modern Approach takes a new look at the powerful data analysis tool. Unlike many DOE textbooks, Montgomery and Jones’ doesn’t focus on the technical or mathematical parts of experimental design, but on the critical tasks of planning and designing your experiment. Every chapter starts with a “textbook design” experiment, included to show how important a thoughtful design is in conducting a successful experimental design effort. Though not the focus of the book, JMP is featured for many of the design and analysis discussed.

Anne Milley	We are super excited to have to design of experiments rock stars with us today.
Anne Milley	We have Professor Doug Montgomery, who is the region's professor of industrial engineering and statistics at Arizona State University, and we have our own Brad Jones principal Research Fellow in the junk division of SAS. Welcome to you both.
Brad Jones	To be here.
Doug Montgomery	Thanks, thanks, nice to be here.
Anne Milley	We're excited about your new book, but let's start at the very beginning, what is design of experiments and how is it a unique approach to experimentation.
Brad Jones	Let me start out by saying what it isn't. It's not like the experiments that you saw, or did in a science lab where you're actually doing a demonstration of something that you already know. Instead, when we do designed experiments in industry, for instance.
Brad Jones	It's because we don't know the answer that we want to find and we want to learn something about how a system or a process behaves
Brad Jones	And so what we do is we do a number of tests where we change the inputs of the system or process in a very special structured way in order to find out how each of those inputs affects whatever output. We're interested in
Anne Milley	Perfect. Good. Well, please share with us the motivating factors that led you to co author your new book design of experiments and modern approach and Doug, let's start with you.
Doug Montgomery	Well, I've, I've been teaching
Doug Montgomery	For about 15 years
Doug Montgomery	At the university level. And I've also talked a lot of designers premise courses in companies and I I saw the, the technology in the field of experimental design changing and
Doug Montgomery	Brad and I talked about this quite a bit for quite a long time. And we felt that there was an opportunity for a for a different kind of textbook or a different kind of book about design.
Doug Montgomery	Many, many of you know that I am the author of it is
Doug Montgomery	In fact, it's in the 10th edition.
Doug Montgomery	And it's been pretty successful some something like 65% of the experimental design courses that are taught
Doug Montgomery	At the university level use my textbook, which is gratifying that means that there's a lot of instructors out there that like it.
Doug Montgomery	But what Brandon, I knew is that there were a lot of people. In fact, we found out that it's about 25% of the people that teach carnivals enforces that use know book at all.
Doug Montgomery	They, they work with slides and notes and God knows what. And, and I think this is a disservice to their students because I, I cannot imagine teaching a course like this with the amount of content, the content in it.
Doug Montgomery	Without a textbook that students can read study and master the details. So we felt that there was a need for a different kind of book. And so that's what started the planning for this and
Doug Montgomery	By the way, it took us seven years to complete this so so we we
Doug Montgomery	We put a lot of thought into into what we were doing and how we were going to go about doing that right. Is that about right, Brad.
Brad Jones	Yeah, I would. I thought I made tell a story about 10 years ago. You and I were at a we're giving talks at the joint statistics meetings.
Brad Jones	Right in Washington, DC.
Brad Jones	And the title of the session was, is there a new paradigm for design of experiments. And what I talked about at the very beginning of this was the various paradigms that design of experiments has gone through from our a fisher at the very beginning to modern software nowadays.
Brad Jones	And there's been some resistance over time to the idea of using software as your major way to create designed experiments.
Brad Jones	But one of the things that I thought was problematic. Back then 10 years ago was that there was no textbook that was built around the idea of using software as the way to get to generate designs.
Brad Jones	And so john saw, who's the co founder of SAS and and also started JUMP JUMP project really wanted a textbook that that felt that would fill that gap and so
Brad Jones	Now here we are 10 years later we finally have that book.
Doug Montgomery	It took a while. But we got it done.
Anne Milley	Yeah where's and we're so glad you did.
Anne Milley	So what are the unique features of this book and Doug, let's start with you again.
Doug Montgomery	Well, I think, I think there's a couple of things that are really
Doug Montgomery	Unique about the book is in
Doug Montgomery	Every, every chapter has an introductory example and in many cases the. The example is is a as a standard design what what Brad, sometimes called a textbook design.
Doug Montgomery	And then we show in many in most cases, how that how that textbook design is is actually an optimal design and you can create this with software and then we
Doug Montgomery	Start looking at, well, what, what if the problem was actually different, you know. Suppose suppose there were there were constraints or there were
Doug Montgomery	Some situation where that standard design didn't exactly fit and let me actually show how how straightforward. It is to create a design that that
Doug Montgomery	Deals with that situation very, very nicely. I think another unique thing in the book is that is that we don't spend a lot of time.
Doug Montgomery	On the details of statistical analysis.
Doug Montgomery	A lot of a lot of design books spend at least half of the book.
Doug Montgomery	Talking about the mechanics of the analysis of variance and presenting the you know the summation formulas and all sorts that we don't do that. We rely on the computer.
Doug Montgomery	To actually do the analysis because in practice. Nobody sits down with a calculator and does summation formula is to do an analysis of variance or or or if they if they're doing that they're misguided so
Doug Montgomery	So, so we try to get right straight to the to the to the use of the computer from from from the very beginning to constructing the design all the way through to the, to the end.
Doug Montgomery	Where you're analyzing the data and presenting results in a way that somebody that's interested in the problem can understand which is
Anne Milley	very streamlined, good bread, anything to add to that, about what makes the to the book.
Brad Jones	Well, it is unique that jump is throughout the book.
Brad Jones	The jump analyses and also the jump designs and at the end, there's an appendix that shows jump scripting commands that you can use to find out.
Brad Jones	quintiles of the normal distribution or the chi square distribution or the F distribution so that so that you don't have to have
Brad Jones	You know, all these indices at the end of the book that with all these tables of that are normally shown and textbook for design. And I think another thing that's that's good about this book is that oftentimes in textbooks, not yours. Doug, but
Brad Jones	Other textbooks.
Brad Jones	The design part is almost treated as if it's a given like the designs come down from God. And, you know, they're the only possible thing that you could actually do
Brad Jones	For this particular problem and and as a result it it makes a student think that there's something magical and mystical about the designs.
Brad Jones	And, you know, and that you must follow these very, you know, flexible design approaches in the future and that is that's totally the wrong thing to do.
Anne Milley	That's been the recipes are always the same.
Brad Jones	Exactly.
Brad Jones	Sometimes you have to change the recipe that may not have one of the ingredients that you need.
Doug Montgomery	Yeah, if the stuff you have in the refrigerator doesn't fit the recipe, you've got a problem.
Anne Milley	Exactly. Good. Good. Thank you both. I'm sure this is going to be so well received, for all the reasons you were just explaining
Anne Milley	Five years ago, Doug, you gave a really interesting plenary it jumped discovery summit where you talked about how design of experiments was practically declared dead 30 years ago. So what did these naysayers get wrong. And why is designed to experiment still so underutilized.
Doug Montgomery	Well, this, this is a really isn't a little interesting backstory to this.
Doug Montgomery	Around 1990 I collaborated with a couple of people from the university for john Cornell was one of them late john Cornell and Andre curry and we wanted to
Doug Montgomery	Yeah, either one of these NSF workshops that where we you basically invite all of the sort of presentations work.
Doug Montgomery	And it's sort of a window you invite lots of people
Doug Montgomery	In, you know, including the people who are going to be the speakers and leaders, so you get a pretty broad spectrum of people. So we run this proposal for this UNICEF.
Doug Montgomery	Workshop and these things are usually pretty automatic to get to get supported by NSF because it's one way they they build their research portfolio so anyway.
Doug Montgomery	The review came back and and they declined to fund the workshop and the principal reviewer said, well, why on earth do we want to
Doug Montgomery	Fund, an effort to see what research needs to be done in the field all the research that needs to be done has been done, the field is basically dead.
Doug Montgomery	And this this disturbed me a lot because I had a pretty active consulting practice in those days.
Doug Montgomery	And and i was out working with a lot of organizations and finding lots of applications of design and lots of problems that that that were
Doug Montgomery	unique and interesting and that hadn't really been explored very much and you know one of, one of the things that was that I was seeing just for instance was
Doug Montgomery	The emergence of split plot and in the industrial world that split plots have been around and hang experiments for years, but there was
Doug Montgomery	There was all this stuff going on in the industrial world who are split plotting was necessary, and the problem scenarios, they're
Doug Montgomery	Often different than they are in agriculture. And so that was that was lots of stuff that really needed to be addressed. And it was astounding to me that somebody who was an expert in the field could could think that all the interesting problems whir whir
Doug Montgomery	whir whir solved it was
Doug Montgomery	It was just
Doug Montgomery	I think design is still under utilized in the science and industrial world.
Doug Montgomery	I think a lot of it goes back to education.
Doug Montgomery	I think the way statistics is taught to scientists and engineers is is not as modern approach as as it could be.
Doug Montgomery	I used to tell my engineering colleagues, the way that we really prove this in engineering education is to have more funerals.
Doug Montgomery	We, you know, we basically need to have a lot of the whole garb that
Doug Montgomery	But then I think things will get better because, you know, for example, my
Doug Montgomery	experimental design course that I teach your Asus is a master's level course 500 level course.
Doug Montgomery	And I very often have well over 100 people in it and they're from all the different engineering disciplines and these people are getting PhDs and he
Doug Montgomery	And in material science and chemical engineering and they're, they're gonna go happen a lot. I'm going to become faculty or they're going to become
Doug Montgomery	Research engineers and research scientist in companies and they're there, they're going to want to use this material. They're going to want to use it. And so it's an it's an education. I think, I think it's an education thing.
Anne Milley	Yeah, I think there's also a
Doug Montgomery	big job.
Doug Montgomery	To do in terms of
Doug Montgomery	that haven't been exposed to somebody. It was just a big group of people out there that you know
Doug Montgomery	Middle. Middle level scientists and engineers people that went through the old approaches and well they think about running experiments they think about either the best guests or one factor at a time. And that's the way to do things, but Brad. I probably have some insight on that too.
Anne Milley	Yeah, that actually is a good segue into what I wanted to ask, Brad. Brad. So the book that you co authored
Anne Milley	With Peter cost optimal design of experiments has actually been used as a continuing education tool for many organizations. And I was curious if your new book with Doug. If you think that will also be used.
Anne Milley	In industry, as continuing education, maybe we can get some of this old guard to see things in a new way.
Brad Jones	Well, it's very possible that
Brad Jones	This this book that Doug and I have just done would be used again in industrial applications.
Brad Jones	I remember that.
Brad Jones	One of my friends from Eastman Chemical Company Kevin white bought his entire statistics team a copy of that book and then they took, he took turns teaching each other for from each chapter. So
Brad Jones	This book, could you be used in the same same way the there are 12 chapters and we can you could make each chapter be the subject of a of a one hour talk in a in a course, maybe a few day course and in modern statistics for experimentation.
Anne Milley	Okay. Very good. Thank you.
Doug Montgomery	And by the way, that the eat the book has got a PowerPoint slides that support every one of the chapters and so you know you using the book in the way that brand describes would be very straightforward for you.
Anne Milley	Oh good, that's useful to know and we have a URL to share with the audience at the end.
Anne Milley	But before the next question I want to read something. So back in 2012 200 was introducing you Brad at Discovery summit and I want to make sure I get right so I'm gonna read it.
Anne Milley	He said, as an old timer. Who's taught design of experiments for years and years and years. I want to testify to the fact that over the last several years. The art of experimental design has changed profoundly
Anne Milley	And if I had to teach courses. Now in the university or in industry, I would have to teach it in a profoundly different manner and a lot of that is due to Brad Jones.
Anne Milley	And I wanted to mention this because I think it relates to what you said earlier, Doug, about how statistics isn't taught well
Anne Milley	And it's been criticized for a long time, but I will. Thankfully, I will say there's evidence that it's changing, so that's good.
Anne Milley	But I wanted to more granular Lee go into, what do you think the rate of changes and how design of experiments is taught. Is it being taught more modern Lee, or do you see evidence of it changing as well.
Anne Milley	And I, you know, whoever wants to start actually dug, we'll start with you, you because you're still in academics.
Doug Montgomery	Well, I think, I think it is changing.
Doug Montgomery	I think, I think.
Doug Montgomery	More and more, you see courses on experimental design taught in in programs that are outside of statistics.
Doug Montgomery	For example, art, our
Doug Montgomery	Experiment design course here. And we actually have three courses that experimental design. We have the 500 level course we had a 600 level course and then
Doug Montgomery	PhD level course. And then we have a 500 level response surface course. It covers response surfaces and mixtures and a few other topics and
Doug Montgomery	More and more. Yeah, I think you're starting to see those courses taught outside of statistics their their their their top maybe taught like here, they're taught engineering. Here are our stats department does teach some work on experimental design, but
Doug Montgomery	And I don't want these people do good work. Okay, don't, I'm not saying that they're not doing the right but what they they teach is of course that's aimed at the geologists
Doug Montgomery	You know who the apologists are
Doug Montgomery	Psychologists the biologists. The sociologist, the what and the these folks need to do statistics.
Doug Montgomery	As as part of the research and so it's more of course on analyzing the data. Okay. And, and I won't even let my students go take the course because it's it's a big SAS template.
Doug Montgomery	They show them how to run pocket Nova for every, every conceivable situation on Earth. And because that's what those folks need to do you know and you know but but some of them come over and take and take our course. I get people from biology.
Doug Montgomery	I don't, I don't get people from from sociology or some of the other colleges, because of the math backwards.
Anne Milley	Okay.
Doug Montgomery	Yeah, they, they, we engineering we, you know, if you have, if you don't have a capitalist prerequisite. And of course, you know, it's, it can't be engineering. So
Doug Montgomery	So, so that's that's
Doug Montgomery	I but I do think by and large the education in experimental design. He's getting there.
Anne Milley	Well, and I think that experimentation is becoming more popular because more people are working with data and they're finding out that they're
Anne Milley	faster ways to learn from that data, but Brad. Did you want to add anything about your observations and how do E's being taught how that might be changing.
Brad Jones	Almost gratified to find out from our academic group and jump that there are over 150 different do eat courses already being taught using jump as the generator of the DMV and this textbook, then can be useful to all those courses because it makes it it makes
Brad Jones	Each section. Each new kind of design.
Brad Jones	Easy in a structured way, it introduces them from, from the very simplest designs to substantially more complex design like designs for nonlinear models.
Brad Jones	So we start, we start with one factor designs and two factor designs and then screening designs and so forth. And we also have
Brad Jones	Discussions of random blocks and split plots and so forth. So there's a lot of material in this book.
Anne Milley	Good. Well, and I just wanted to add on to that. So just even in your interactions with customers over the years. I think you've seen scientists and engineers, much more eagerly embrace monitor design of experiments because they see how it can really help them.
Brad Jones	Yes, I think that the semiconductor companies have really bought into modern design of experiments and even more so the pharmaceutical companies, the pharmaceutical applications have are complex and they require a lot of thought and and the old school designs just aren't up to it.
Anne Milley	Right there. Yeah.
Doug Montgomery	A lot of a lot of the consulting work that
Doug Montgomery	I've done has been with biotech firms and medical device companies.
Doug Montgomery	And these, these people are becoming and have become very big users.
Doug Montgomery	One of the things, and I discovered in in my consulting practices that people would would ask me to go. Could you come and teach a course.
Doug Montgomery	And so I would, but I would tell them that I'm only going to teach a course if if you have people that come to the course that actually have a project or a problem.
Anne Milley	Where they can use this
Doug Montgomery	And and so as part of the way I taught the course I would. I would spend like three days, teaching them a course and then I would come back.
Doug Montgomery	A couple of months later, maybe three months later. And we have another couple of days, there will be some again more new material, but I would have all these people talk about their projects. What have they done
Doug Montgomery	And most of them were very successful in using design. And what I found is it only took a few people to have a big success for the word to spread throughout these organizations and suddenly now everybody wants to do this. So, so, yeah, that was gratifying to see that.
Anne Milley	Right, we'd love it when good things go viral, good things.
Anne Milley	So the last question before we wrap up, Doug, I wanted to ask you, because I've heard you talk about some really fun and interesting examples of do we so if you have a favorite you want to share, that'd be great.
Doug Montgomery	Oh yeah, my, my all time favorite okay and and
Doug Montgomery	Any, any of you that know me, know that that I like to drink wine. Okay. And the reason I like to drink wine is I can't drink beer beer makes me sick.
Doug Montgomery	Physically I become part of the entertainment and so some years ago when I was at the University of Washington in Seattle. I was doing consulting work for a company down in Oregon.
Doug Montgomery	And the guy worked with. There was a really sharp chemical engineering guy and he can I got to be pretty good friends. And so we're out having
Doug Montgomery	drinks and dinner one night and and you know he says you know I've always wanted to be a winemaker. And I said, Well, what's stopping you. And he said well you know how to make a small fortune and wine. And I said, no, he said, you start with a large fortune and
Doug Montgomery	And he said, I don't have that. And they said, but but I am thinking about forming a partnership, because I've got this. I just bought this lovely
Doug Montgomery	Vineyard property in just outside of a little town called newburgh Oregon, and he said, I'd like to turn that into a producing vineyard and then start making wine.
Doug Montgomery	And I said, well, sign me up. I'll be a partner. And so he ended up putting 17 partners together and and so we had this property we trusted we planted and then we
Doug Montgomery	Sort of
Doug Montgomery	came to the realization that we didn't really know how to make one
Doug Montgomery	Which we could grow the grapes and we were going to sell the grapes to some folks but we
Doug Montgomery	Didn't know how to carry and then any said, Well, I'm a chemical in
Doug Montgomery	My first job as a chemical engineer. I figured out how to do it. And he said, Doug, you're going to help us that our company here.
Doug Montgomery	run experiments to solve some of the problems we have with our castings. Let's do that here. So we set out to develop a winemaking process using design experiments and we did fractional factorial experiments every year for five years.
Doug Montgomery	Wow and and we always started off with a long list of fat, we must have started off with 20 factors.
Doug Montgomery	We scoured the literature we we took all of the the
Doug Montgomery	You know the folklore from our friends in France that make Burgundy, we were going to make peanut war and we started running 16 one fractional factorial experiments because we had 16 little test barrels.
Doug Montgomery	That we had had made. And so
Doug Montgomery	We started doing this and I wouldn't let us run more than eight factors per year because eight factors 16 runs as a brand new resolution for so at least you can figure out the main effects.
Doug Montgomery	And so we did this every year for five years trying to figure out what the right
Doug Montgomery	Factors were and the last year we decided that, okay, we've got enough wine that we made this year.
Doug Montgomery	That we could have a commercial release so so we we decided to do that. And we also decided to enter the one that we had the one that we had made from our from our process.
Doug Montgomery	In a competition to see how it how it get
Doug Montgomery	And. And by the way, there's a there's a
Doug Montgomery	There's a secret to getting an award for your wine. Okay, here's the secret. You take your wine to the most obscure
Doug Montgomery	Read that county in your state. Okay. I have a generic name for this dog breath County.
Doug Montgomery	Okay. He grew up the dog breath COUNTY AND YOU ENTER THE LINE IN THE FAIR AND NOBODY KNOWS ANYTHING. Okay. And nobody makes anything it's any good. So if your stuff is at least drinkable. You're going to get the award.
Doug Montgomery	And trust me, you know, go to wineries, look at the awards on the wall, see where they're from. There are a lot of them from dog breath counting. So we went to the international wine competition okay in Indianapolis, Indiana 2000 lines, guess what we got a gold medal.
Anne Milley	Of course you did. Because excellent process.
Doug Montgomery	It was good stuff.
Anne Milley	I'm sure it was with Doug. We're out of time, but
Doug Montgomery	We built. We've built a successful business out of this. And yes,
Anne Milley	Yes, and
Anne Milley	Success. And yes, I've had the pleasure of sharing a glass of
Anne Milley	Wine with you. Thank you.
Anne Milley	Well, I want to thank you both. Brad, Doug. You guys are such innovators in this space. Thank you for all that you've done and for all the other innovation that you are enabling so thank you and Julian back to you.

Anne Milley We are super excited to have to design of experiments rock stars with us today. Anne Milley We have Professor Doug Montgomery, who is the region's professor of industrial engineering and statistics at Arizona State University, and we have our own Brad Jones principal Research Fellow in the junk division of SAS. Welcome to you both. Brad Jones To be here. Doug Montgomery Thanks, thanks, nice to be here. Anne Milley We're excited about your new book, but let's start at the very beginning, what is design of experiments and how is it a unique approach to experimentation. Brad Jones Let me start out by saying what it isn't. It's not like the experiments that you saw, or did in a science lab where you're actually doing a demonstration of something that you already know. Instead, when we do designed experiments in industry, for instance. Brad Jones It's because we don't know the answer that we want to find and we want to learn something about how a system or a process behaves Brad Jones And so what we do is we do a number of tests where we change the inputs of the system or process in a very special structured way in order to find out how each of those inputs affects whatever output. We're interested in Anne Milley Perfect. Good. Well, please share with us the motivating factors that led you to co author your new book design of experiments and modern approach and Doug, let's start with you. Doug Montgomery Well, I've, I've been teaching Doug Montgomery For about 15 years Doug Montgomery At the university level. And I've also talked a lot of designers premise courses in companies and I I saw the, the technology in the field of experimental design changing and Doug Montgomery Brad and I talked about this quite a bit for quite a long time. And we felt that there was an opportunity for a for a different kind of textbook or a different kind of book about design. Doug Montgomery Many, many of you know that I am the author of it is Doug Montgomery In fact, it's in the 10th edition. Doug Montgomery And it's been pretty successful some something like 65% of the experimental design courses that are taught Doug Montgomery At the university level use my textbook, which is gratifying that means that there's a lot of instructors out there that like it. Doug Montgomery But what Brandon, I knew is that there were a lot of people. In fact, we found out that it's about 25% of the people that teach carnivals enforces that use know book at all. Doug Montgomery They, they work with slides and notes and God knows what. And, and I think this is a disservice to their students because I, I cannot imagine teaching a course like this with the amount of content, the content in it. Doug Montgomery Without a textbook that students can read study and master the details. So we felt that there was a need for a different kind of book. And so that's what started the planning for this and Doug Montgomery By the way, it took us seven years to complete this so so we we Doug Montgomery We put a lot of thought into into what we were doing and how we were going to go about doing that right. Is that about right, Brad. Brad Jones Yeah, I would. I thought I made tell a story about 10 years ago. You and I were at a we're giving talks at the joint statistics meetings. Brad Jones Right in Washington, DC. Brad Jones And the title of the session was, is there a new paradigm for design of experiments. And what I talked about at the very beginning of this was the various paradigms that design of experiments has gone through from our a fisher at the very beginning to modern software nowadays. Brad Jones And there's been some resistance over time to the idea of using software as your major way to create designed experiments. Brad Jones But one of the things that I thought was problematic. Back then 10 years ago was that there was no textbook that was built around the idea of using software as the way to get to generate designs. Brad Jones And so john saw, who's the co founder of SAS and and also started JUMP JUMP project really wanted a textbook that that felt that would fill that gap and so Brad Jones Now here we are 10 years later we finally have that book. Doug Montgomery It took a while. But we got it done. Anne Milley Yeah where's and we're so glad you did. Anne Milley So what are the unique features of this book and Doug, let's start with you again. Doug Montgomery Well, I think, I think there's a couple of things that are really Doug Montgomery Unique about the book is in Doug Montgomery Every, every chapter has an introductory example and in many cases the. The example is is a as a standard design what what Brad, sometimes called a textbook design. Doug Montgomery And then we show in many in most cases, how that how that textbook design is is actually an optimal design and you can create this with software and then we Doug Montgomery Start looking at, well, what, what if the problem was actually different, you know. Suppose suppose there were there were constraints or there were Doug Montgomery Some situation where that standard design didn't exactly fit and let me actually show how how straightforward. It is to create a design that that Doug Montgomery Deals with that situation very, very nicely. I think another unique thing in the book is that is that we don't spend a lot of time. Doug Montgomery On the details of statistical analysis. Doug Montgomery A lot of a lot of design books spend at least half of the book. Doug Montgomery Talking about the mechanics of the analysis of variance and presenting the you know the summation formulas and all sorts that we don't do that. We rely on the computer. Doug Montgomery To actually do the analysis because in practice. Nobody sits down with a calculator and does summation formula is to do an analysis of variance or or or if they if they're doing that they're misguided so Doug Montgomery So, so we try to get right straight to the to the to the use of the computer from from from the very beginning to constructing the design all the way through to the, to the end. Doug Montgomery Where you're analyzing the data and presenting results in a way that somebody that's interested in the problem can understand which is Anne Milley very streamlined, good bread, anything to add to that, about what makes the to the book. Brad Jones Well, it is unique that jump is throughout the book. Brad Jones The jump analyses and also the jump designs and at the end, there's an appendix that shows jump scripting commands that you can use to find out. Brad Jones quintiles of the normal distribution or the chi square distribution or the F distribution so that so that you don't have to have Brad Jones You know, all these indices at the end of the book that with all these tables of that are normally shown and textbook for design. And I think another thing that's that's good about this book is that oftentimes in textbooks, not yours. Doug, but Brad Jones Other textbooks. Brad Jones The design part is almost treated as if it's a given like the designs come down from God. And, you know, they're the only possible thing that you could actually do Brad Jones For this particular problem and and as a result it it makes a student think that there's something magical and mystical about the designs. Brad Jones And, you know, and that you must follow these very, you know, flexible design approaches in the future and that is that's totally the wrong thing to do. Anne Milley That's been the recipes are always the same. Brad Jones Exactly. Brad Jones Sometimes you have to change the recipe that may not have one of the ingredients that you need. Doug Montgomery Yeah, if the stuff you have in the refrigerator doesn't fit the recipe, you've got a problem. Anne Milley Exactly. Good. Good. Thank you both. I'm sure this is going to be so well received, for all the reasons you were just explaining Anne Milley Five years ago, Doug, you gave a really interesting plenary it jumped discovery summit where you talked about how design of experiments was practically declared dead 30 years ago. So what did these naysayers get wrong. And why is designed to experiment still so underutilized. Doug Montgomery Well, this, this is a really isn't a little interesting backstory to this. Doug Montgomery Around 1990 I collaborated with a couple of people from the university for john Cornell was one of them late john Cornell and Andre curry and we wanted to Doug Montgomery Yeah, either one of these NSF workshops that where we you basically invite all of the sort of presentations work. Doug Montgomery And it's sort of a window you invite lots of people Doug Montgomery In, you know, including the people who are going to be the speakers and leaders, so you get a pretty broad spectrum of people. So we run this proposal for this UNICEF. Doug Montgomery Workshop and these things are usually pretty automatic to get to get supported by NSF because it's one way they they build their research portfolio so anyway. Doug Montgomery The review came back and and they declined to fund the workshop and the principal reviewer said, well, why on earth do we want to Doug Montgomery Fund, an effort to see what research needs to be done in the field all the research that needs to be done has been done, the field is basically dead. Doug Montgomery And this this disturbed me a lot because I had a pretty active consulting practice in those days. Doug Montgomery And and i was out working with a lot of organizations and finding lots of applications of design and lots of problems that that that were Doug Montgomery unique and interesting and that hadn't really been explored very much and you know one of, one of the things that was that I was seeing just for instance was Doug Montgomery The emergence of split plot and in the industrial world that split plots have been around and hang experiments for years, but there was Doug Montgomery There was all this stuff going on in the industrial world who are split plotting was necessary, and the problem scenarios, they're Doug Montgomery Often different than they are in agriculture. And so that was that was lots of stuff that really needed to be addressed. And it was astounding to me that somebody who was an expert in the field could could think that all the interesting problems whir whir Doug Montgomery whir whir solved it was Doug Montgomery It was just Doug Montgomery I think design is still under utilized in the science and industrial world. Doug Montgomery I think a lot of it goes back to education. Doug Montgomery I think the way statistics is taught to scientists and engineers is is not as modern approach as as it could be. Doug Montgomery I used to tell my engineering colleagues, the way that we really prove this in engineering education is to have more funerals. Doug Montgomery We, you know, we basically need to have a lot of the whole garb that Doug Montgomery But then I think things will get better because, you know, for example, my Doug Montgomery experimental design course that I teach your Asus is a master's level course 500 level course. Doug Montgomery And I very often have well over 100 people in it and they're from all the different engineering disciplines and these people are getting PhDs and he Doug Montgomery And in material science and chemical engineering and they're, they're gonna go happen a lot. I'm going to become faculty or they're going to become Doug Montgomery Research engineers and research scientist in companies and they're there, they're going to want to use this material. They're going to want to use it. And so it's an it's an education. I think, I think it's an education thing. Anne Milley Yeah, I think there's also a Doug Montgomery big job. Doug Montgomery To do in terms of Doug Montgomery that haven't been exposed to somebody. It was just a big group of people out there that you know Doug Montgomery Middle. Middle level scientists and engineers people that went through the old approaches and well they think about running experiments they think about either the best guests or one factor at a time. And that's the way to do things, but Brad. I probably have some insight on that too. Anne Milley Yeah, that actually is a good segue into what I wanted to ask, Brad. Brad. So the book that you co authored Anne Milley With Peter cost optimal design of experiments has actually been used as a continuing education tool for many organizations. And I was curious if your new book with Doug. If you think that will also be used. Anne Milley In industry, as continuing education, maybe we can get some of this old guard to see things in a new way. Brad Jones Well, it's very possible that Brad Jones This this book that Doug and I have just done would be used again in industrial applications. Brad Jones I remember that. Brad Jones One of my friends from Eastman Chemical Company Kevin white bought his entire statistics team a copy of that book and then they took, he took turns teaching each other for from each chapter. So Brad Jones This book, could you be used in the same same way the there are 12 chapters and we can you could make each chapter be the subject of a of a one hour talk in a in a course, maybe a few day course and in modern statistics for experimentation. Anne Milley Okay. Very good. Thank you. Doug Montgomery And by the way, that the eat the book has got a PowerPoint slides that support every one of the chapters and so you know you using the book in the way that brand describes would be very straightforward for you. Anne Milley Oh good, that's useful to know and we have a URL to share with the audience at the end. Anne Milley But before the next question I want to read something. So back in 2012 200 was introducing you Brad at Discovery summit and I want to make sure I get right so I'm gonna read it. Anne Milley He said, as an old timer. Who's taught design of experiments for years and years and years. I want to testify to the fact that over the last several years. The art of experimental design has changed profoundly Anne Milley And if I had to teach courses. Now in the university or in industry, I would have to teach it in a profoundly different manner and a lot of that is due to Brad Jones. Anne Milley And I wanted to mention this because I think it relates to what you said earlier, Doug, about how statistics isn't taught well Anne Milley And it's been criticized for a long time, but I will. Thankfully, I will say there's evidence that it's changing, so that's good. Anne Milley But I wanted to more granular Lee go into, what do you think the rate of changes and how design of experiments is taught. Is it being taught more modern Lee, or do you see evidence of it changing as well. Anne Milley And I, you know, whoever wants to start actually dug, we'll start with you, you because you're still in academics. Doug Montgomery Well, I think, I think it is changing. Doug Montgomery I think, I think. Doug Montgomery More and more, you see courses on experimental design taught in in programs that are outside of statistics. Doug Montgomery For example, art, our Doug Montgomery Experiment design course here. And we actually have three courses that experimental design. We have the 500 level course we had a 600 level course and then Doug Montgomery PhD level course. And then we have a 500 level response surface course. It covers response surfaces and mixtures and a few other topics and Doug Montgomery More and more. Yeah, I think you're starting to see those courses taught outside of statistics their their their their top maybe taught like here, they're taught engineering. Here are our stats department does teach some work on experimental design, but Doug Montgomery And I don't want these people do good work. Okay, don't, I'm not saying that they're not doing the right but what they they teach is of course that's aimed at the geologists Doug Montgomery You know who the apologists are Doug Montgomery Psychologists the biologists. The sociologist, the what and the these folks need to do statistics. Doug Montgomery As as part of the research and so it's more of course on analyzing the data. Okay. And, and I won't even let my students go take the course because it's it's a big SAS template. Doug Montgomery They show them how to run pocket Nova for every, every conceivable situation on Earth. And because that's what those folks need to do you know and you know but but some of them come over and take and take our course. I get people from biology. Doug Montgomery I don't, I don't get people from from sociology or some of the other colleges, because of the math backwards. Anne Milley Okay. Doug Montgomery Yeah, they, they, we engineering we, you know, if you have, if you don't have a capitalist prerequisite. And of course, you know, it's, it can't be engineering. So Doug Montgomery So, so that's that's Doug Montgomery I but I do think by and large the education in experimental design. He's getting there. Anne Milley Well, and I think that experimentation is becoming more popular because more people are working with data and they're finding out that they're Anne Milley faster ways to learn from that data, but Brad. Did you want to add anything about your observations and how do E's being taught how that might be changing. Brad Jones Almost gratified to find out from our academic group and jump that there are over 150 different do eat courses already being taught using jump as the generator of the DMV and this textbook, then can be useful to all those courses because it makes it it makes Brad Jones Each section. Each new kind of design. Brad Jones Easy in a structured way, it introduces them from, from the very simplest designs to substantially more complex design like designs for nonlinear models. Brad Jones So we start, we start with one factor designs and two factor designs and then screening designs and so forth. And we also have Brad Jones Discussions of random blocks and split plots and so forth. So there's a lot of material in this book. Anne Milley Good. Well, and I just wanted to add on to that. So just even in your interactions with customers over the years. I think you've seen scientists and engineers, much more eagerly embrace monitor design of experiments because they see how it can really help them. Brad Jones Yes, I think that the semiconductor companies have really bought into modern design of experiments and even more so the pharmaceutical companies, the pharmaceutical applications have are complex and they require a lot of thought and and the old school designs just aren't up to it. Anne Milley Right there. Yeah. Doug Montgomery A lot of a lot of the consulting work that Doug Montgomery I've done has been with biotech firms and medical device companies. Doug Montgomery And these, these people are becoming and have become very big users. Doug Montgomery One of the things, and I discovered in in my consulting practices that people would would ask me to go. Could you come and teach a course. Doug Montgomery And so I would, but I would tell them that I'm only going to teach a course if if you have people that come to the course that actually have a project or a problem. Anne Milley Where they can use this Doug Montgomery And and so as part of the way I taught the course I would. I would spend like three days, teaching them a course and then I would come back. Doug Montgomery A couple of months later, maybe three months later. And we have another couple of days, there will be some again more new material, but I would have all these people talk about their projects. What have they done Doug Montgomery And most of them were very successful in using design. And what I found is it only took a few people to have a big success for the word to spread throughout these organizations and suddenly now everybody wants to do this. So, so, yeah, that was gratifying to see that. Anne Milley Right, we'd love it when good things go viral, good things. Anne Milley So the last question before we wrap up, Doug, I wanted to ask you, because I've heard you talk about some really fun and interesting examples of do we so if you have a favorite you want to share, that'd be great. Doug Montgomery Oh yeah, my, my all time favorite okay and and Doug Montgomery Any, any of you that know me, know that that I like to drink wine. Okay. And the reason I like to drink wine is I can't drink beer beer makes me sick. Doug Montgomery Physically I become part of the entertainment and so some years ago when I was at the University of Washington in Seattle. I was doing consulting work for a company down in Oregon. Doug Montgomery And the guy worked with. There was a really sharp chemical engineering guy and he can I got to be pretty good friends. And so we're out having Doug Montgomery drinks and dinner one night and and you know he says you know I've always wanted to be a winemaker. And I said, Well, what's stopping you. And he said well you know how to make a small fortune and wine. And I said, no, he said, you start with a large fortune and Doug Montgomery And he said, I don't have that. And they said, but but I am thinking about forming a partnership, because I've got this. I just bought this lovely Doug Montgomery Vineyard property in just outside of a little town called newburgh Oregon, and he said, I'd like to turn that into a producing vineyard and then start making wine. Doug Montgomery And I said, well, sign me up. I'll be a partner. And so he ended up putting 17 partners together and and so we had this property we trusted we planted and then we Doug Montgomery Sort of Doug Montgomery came to the realization that we didn't really know how to make one Doug Montgomery Which we could grow the grapes and we were going to sell the grapes to some folks but we Doug Montgomery Didn't know how to carry and then any said, Well, I'm a chemical in Doug Montgomery My first job as a chemical engineer. I figured out how to do it. And he said, Doug, you're going to help us that our company here. Doug Montgomery run experiments to solve some of the problems we have with our castings. Let's do that here. So we set out to develop a winemaking process using design experiments and we did fractional factorial experiments every year for five years. Doug Montgomery Wow and and we always started off with a long list of fat, we must have started off with 20 factors. Doug Montgomery We scoured the literature we we took all of the the Doug Montgomery You know the folklore from our friends in France that make Burgundy, we were going to make peanut war and we started running 16 one fractional factorial experiments because we had 16 little test barrels. Doug Montgomery That we had had made. And so Doug Montgomery We started doing this and I wouldn't let us run more than eight factors per year because eight factors 16 runs as a brand new resolution for so at least you can figure out the main effects. Doug Montgomery And so we did this every year for five years trying to figure out what the right Doug Montgomery Factors were and the last year we decided that, okay, we've got enough wine that we made this year. Doug Montgomery That we could have a commercial release so so we we decided to do that. And we also decided to enter the one that we had the one that we had made from our from our process. Doug Montgomery In a competition to see how it how it get Doug Montgomery And. And by the way, there's a there's a Doug Montgomery There's a secret to getting an award for your wine. Okay, here's the secret. You take your wine to the most obscure Doug Montgomery Read that county in your state. Okay. I have a generic name for this dog breath County. Doug Montgomery Okay. He grew up the dog breath COUNTY AND YOU ENTER THE LINE IN THE FAIR AND NOBODY KNOWS ANYTHING. Okay. And nobody makes anything it's any good. So if your stuff is at least drinkable. You're going to get the award. Doug Montgomery And trust me, you know, go to wineries, look at the awards on the wall, see where they're from. There are a lot of them from dog breath counting. So we went to the international wine competition okay in Indianapolis, Indiana 2000 lines, guess what we got a gold medal. Anne Milley Of course you did. Because excellent process. Doug Montgomery It was good stuff. Anne Milley I'm sure it was with Doug. We're out of time, but Doug Montgomery We built. We've built a successful business out of this. And yes, Anne Milley Yes, and Anne Milley Success. And yes, I've had the pleasure of sharing a glass of Anne Milley Wine with you. Thank you. Anne Milley Well, I want to thank you both. Brad, Doug. You guys are such innovators in this space. Thank you for all that you've done and for all the other innovation that you are enabling so thank you and Julian back to you.