Virtual Classrooms Are Here to Stay. How Do You Conduct a Practical DOE Exercise? (2021-US-30MP-842)
Ledi Trutna, JMP Education, SAS
Diana Ballard, Statistician, Brewer Science
The challenge of teaching remote DOE classes is in the live exercise. Students put the pieces of DOE together by actually designing an experiment together, gathering data, analyzing it, generating models and running confirmation runs. But how do we, as instructors, manage these exercises when we are so physically far apart from our students? After presenting a practical DOE conducted with students in a remote setting, we offer suggestions on how you could manage and run similar experiments for any size of group.
Auto-generated transcript...
Speaker | Transcript |
| Olivia Lippincott | So he started the recording so verifying this is virtual classrooms are here to stay, how do you conduct a practical do we exercise. |
| The right talk Okay, and that you understand that this is being recorded for use in the job discovery summit conference and will be available publicly in the JMP user community do you give permission for this recording and use. | |
| Yes. | |
| Olivia Lippincott | i'm going to go off camera and mute and once I do you guys can get started. |
| Ledi Trutna | Hello, I'm Ledi Trutna. |
| Diana Ballard | And I'm Diana Ballard from Brewer Science. |
| Ledi Trutna | I'm with SAS Education. |
| Diana Ballard | We both live in Austin, Texas, and have known each other for many years. |
| Ledi Trutna | How long have you been using JMP in teaching DOE, Diana? |
| Diana Ballard | I started using JMP around 1996 and teaching DOE in 1994. How about you? |
| Ledi Trutna | I started teaching DOE in 1987 and I started using JMP in 1996. |
| Diana Ballard | So that gives us 50 collective years of teaching DOE. |
| Are we that old? | |
| Ledi Trutna | Yes, we're really that old. So we want to have a conversation today about our experiences conducting experiments in the classroom and teaching DOE and how to transition those experiments to a virtual classroom. |
| Diana Ballard | Not every DOE class actually has data collection, so when did you start doing this kind of experiment in the classroom? |
| Ledi Trutna | When I was at Texas Instruments, I started teaching designed...Toguchi design of experiments actually and i've got a catapult to demonstrate to doubting engineers how a designed experiment can work. |
| And I developed this exercise using the catapult, where a team would brainstorm factors, figure out the measurement process, collect data, analyze it, and run confirmation runs, all in the classroom setting. | |
| And we were throwing a little plastic golf ball, measuring where it lands. And it was so much fun and it made DOE come to life to many of the engineers participating, and I used it for years in DOE, and even in some SPC classes. | |
| Diana Ballard | When I saw the benefit of it when we started teaching together, when we were both consulting, we both had assignments at Simitech but at different times. And then in the early 2000s, we started consulting for the same company and teaching classes together. |
| Ledi Trutna | And that company really believed in hands-on exercises, so we use paper helicopters and Lego toy cars and also with airplanes and nerf guns. |
| Diana Ballard | I remember going shopping trying to buy all the smallest Lego cars I could find at Walmart or Target, because they didn't last very long and sometimes we lost pieces. And those catapults? Not great for travel, but the tiny Lego cars, those were good and they were hard to find. |
| Ledi Trutna | So what all these experiments had in common was a measurable response, like distance or time, and the data collection or run of an experiment |
| was quick. We didn't want to spend three hours collecting data. I wanted the data collection to be able to be completed in 30 minutes, so something that was quick to take a measurement. | |
| Diana Ballard | Which brings us to the virtual classroom. We're teaching over Zoom or Teams and our students are scattered all across the country. How can we do an experiment with them in a classroom environment while we are so physically distant? |
| Ledi Trutna | So Diana and I had a brainstorming session last summer to talk about this, on how could you conduct an experiment in this virtual world. And there might have been alcohol involved in this brainstorming session but |
| we could assume, at least at that time, that most everybody was working from home. So what did they have at home where we could experiment and what would the goals be for this experiment? | |
| So we wanted it to be complex enough so that no one knows the answer, but simple enough that the data collection doesn't need to take | |
| forever. And so what we decided to do was boil water. And so we had the assumption that everyone would have access to a stove, pans and a stopwatch, and we were going to measure how long it took the water to boil. | |
| Diana Ballard | Whoever said a watched pot never boils was wrong. We brainstormed lots of factors, |
| which led to lots of questions about how to run the experiment. So I designed one in JMP and I ran it. | |
| And I did learn a lot about choices. How hot do I turn the stove on? What is the order of operation? | |
| When do I start the timer? What does boiling mean? What happens if it never boils? I had one instance where I thought I might ruin my pan, because I was about to boil it dry, but it never really boiled to our definition of boiling water. | |
| Ledi Trutna | So after Diana did her experiment, I had to do one also and then we were able to agree on some things about the process of running the experiment. |
| Diana Ballard | For example, you have to start with a cold stove and regular water, not pre heated water. And to minimize the time required, we don't want to boil large amounts of water. We started with a range of one to four cups, |
| but I had some runs that took over 15 minutes to boil and some that never boiled. | |
| So we learned that the choice of a | |
| wide pan | |
| does not work well and we decided to eliminate that wide pan. | |
| Ledi Trutna | We eliminated some of the factors to keep the experiment size small. For example, we have not tested anything with the lid on or off, or the fan on or off. We haven't experimented with those. |
| Diana Ballard | And we figured out that we have different definitions of add salt. I was thinking teaspoons; |
| you were thinking dashes. | |
| But we were both curious if adding salt would make the water bowl faster. | |
| Ledi Trutna | So the real challenge was agreeing on the definition of boiling. So on my induction stove, the water heats up very quickly and starts boiling. So what we decided we should do is have three measurements. |
| First is the time to little bubbles on the bottom of the stove. So this is going to show you a little teeny tiny bubbles. You can see it on this left-hand side of the pan, the bubbles coming up from the bottom of the stove. | |
| And then we're also going to measure when can you hear the water boiling. And then we're lastly going to measure the big bubbles. So now, this is the the big bubbles. So do you | |
| click the stopwatch at the start of the big bubbles or how long does it have to be going before you're going to say that's boiling? | |
| Diana Ballard | This was the problem that I had with the flat pan, when I only had a little bit of water in it, it never got to that big bubble stage. |
| I boiled all the water away but I | |
| never stopped the timer because it never achieved big, visible bubbles. So that's when we decided the pan type factor should only be two levels, big and small, not including the flat. | |
| Ledi Trutna | And we had similar sizes and types of pan, so these were the...these were my big and small pans, and these were Diana's big and small pans. |
| Diana Ballard | We were guessing that there might be an interaction between the size of the pan and the size of the burner, because we know from cooking experiences that the big pan on the small burner might not heat up as quickly as the big pan on the big burner. |
| So after we ran both experiments, both ran our experiments, and our results shared with each other, we wanted to try it on some willing participants (victims). | |
| We have a group of statisticians here in Austin, Texas, who were once affiliated with Simitech, and the group has expanded to being a statistical nerds who have lunch together two or three times a year (at least, we used to pre-Covid). | |
| Ledi Trutna | And so that experiment is here in this outline box called lunch experiment, but what we did is scheduled a virtual lunch, where we asked everyone to help us collect some data. |
| We had seven participants who agreed to gather data and design an experiment and assigned them each two runs, | |
| assuming they have one burner on the stove and more than one pan they can use. And we knew we would have a variety of types of stoves, both gas and electric, and maybe not an equal number of them, | |
| so we created a method of running experiments so everyone would do it the same way. And we said, this is...this is how we want you to run the experiment. | |
| Diana Ballard | I think we designed it as a fixed block with two runs per block, so that each person would collect data on two runs, |
| easy to achieve in about 15 minutes, which is our timing goal, assuming you set the temperature to max. Then we could gather the data into a table and share it back with everybody. | |
| Ledi Trutna | So let's show you how to design this in JMP. At the bottom of the journal, I've got the responses table and the factors table, so i'm going to go to DOE and custom design |
| and load those factors and responses. So the factors table was the last one I touched, so I can load my factors and then I'm going to find my responses table and I'll go load my responses. Now, Diana, you wouldn't do it this way. | |
| Diana Ballard | I would not. In a real classroom, I really want my students to have the experience of designing it themselves. |
| Ledi Trutna | So we're doing this for the sake of time now, where we're loading the responses and loading the factors but we haven't finished it yet. So here's my three responses that I want to minimize. |
| I have four of my factors in, but I don't have my blocking factor so we're going to have two runs per person. So when we did it, we created this as a fixed block with two runs per person. And I'm going to call this block "person stove." | |
| And then we have to think about what terms go in the model. So for the main effects only model, I need 14 runs, but what if I want some of those interactions? | |
| This 14-run design would work with for seven people just perfectly, but what if I want to look at some of those interactions? | |
| I can go and click the RSM button and add the higher order terms, as in all the two-factor interactions and the quadratic effects for my two continuous factors. | |
| But now I'm up to 30 runs and I can't afford to do 30 runs. So what can I do, I'm going to go and take all of my higher order terms and ask JMP to, | |
| if possible, give me the ability to estimate those, and then i've got the 14 runs | |
| in my design. | |
| Diana Ballard | Most of my classes, seven people is a dream. There's no way we would get that many. |
| So what if I only have three people participating or two? I mean I like the idea of the, if possible, on the interactions but still, even if I go from minimum number of runs here with 10, that's still assuming I have, what, five people? | |
| I can see my students being optimistic about their ability to do three or even four pans of water. I've got four partners, I can do it. That might take a bit longer, so that would maybe double the class time, maybe three quarters of an hour instead of only 15 minutes. | |
| I learned the hard way in my data collection that trying to boil four pots on four burners is a terrible idea. I could not keep track of it. | |
| If we're crunched for time, so I really want to keep it down to | |
| 15 minutes or less than 30 minutes for data collection, I'd consider dropping one or more of our factors from my really tiny classes. | |
| Ledi Trutna | And it was still be a great experiment when doing that, and then you can reduce the number of runs because the point is all about collecting the data. I'm going to go ahead and click make design here, but have you ever seen people make mistakes when an running experiment? |
| Diana Ballard | Yes. I've had them assingned to gather a huge DOE, and then give up entirely as it being too big and them taking too long, when they were only halfway through. |
| I've had them try to add a factor when they'd already started the data collection. I've had them try to add a response when they'd already started the data collection. | |
| I've had them decide, oh this random DOE table, that makes no sense. It'd be...it'd be better if I did Run 1 and then Run 5 and then Run 7 so we don't have to change this hard to vary factor. | |
| Of course you can't generate analyzable results when you do that. You're running it as a split plot, but you didn't design it as a split plot. This happens so frequently. | |
| It's a great learning experience to happen in the classroom instead of on the manufacturing floor with real data and a real DOE that could cost thousands or hundreds of thousands of dollars. What about you? | |
| Ledi Trutna | I've seen the same thing that you just mentioned.I did have one other one, which was a team that designed a very small experiment thinking they're going to finish early and get to go home early. |
| But the results didn't meet the goals of the experiment, so they had to do a follow on experiment and widen their factor ranges to meet their goals and they weren't super happy about having to complete two experiments, instead of one. | |
| Diana Ballard | Planning, planning, planning. Many groups of students surprise me by deciding that what I see as a continuous factor, like water or salt here, |
| Ttey decide, no, no those are categorical factors, and then they don't include center points and they don't include replication. | |
| And this is also a wonderful mistake to make in a classroom rather than the real world, because the analysis drives home the value of continuous factors and center points and replication wherever you possibly can. | |
| So what if I have a really small class, like say two people, and I don't have time to do this whole experiment in the alloted class time and we still want to do this, these factors, | |
| no dropping factors? | |
| Ledi Trutna | I have a class this week with two students and |
| this will be a great experiment to assign as homework. So we're teaching these classes in more half-day kind of opportunities, or if it's a multiple-day class, | |
| class ends on the first day and each person goes away with two or four runs that they're expected to complete. | |
| And they come back the next day with their measurements and you can do the analysis. | |
| And that would work if you had some people in the office and some at home. Just assign the runs as homework. It won't take long and the point is to minimize the time required to complete the runs. | |
| Of course, if they're really ambitious, they can do the whole experiment themselves. | |
| Diana Ballard | Of course, we did the whole experiments ourselves, and we learned how long it takes both the pan and the burner to cool down to start the next run. I found the cooldown time to be up to 30 minutes so distributing the runs to multiple people, if at all possible, is kind of a better idea. |
| All right, let's talk about key lessons that we want students to take away. First, it's okay to make mistakes; that's when you learn what not to do. | |
| Ledi Trutna | And you need to think about the measurement process and improve it as much as possible before starting the data collection. |
| Diana Ballard | Think about the logistics of the data collection, where can and will things go wrong? Try to anticipate everything we can think of. Do a data collection, that will hopefully protect our conclusions from anything we cannot anticipate. |
| Ledi Trutna | So the steps required to be successful in conducting an experiment. You've got to think about what is the problem. |
| You've got to brainstorm your factors and responses and choose your factor levels. | |
| Design the experimen. Conduct the experiment. Analyze the results and predict the response. All of those are things that can be completed in this classroom exercise and, if you want to analyze our experiment that we conducted with our lunch group, the table is here in this journal behind | |
| the outline box that says lunch experiment. You can see our results where most of the measured values were 10 minutes or less, and we had seven people participating. | |
| Well, thank you for listening and we hope you'll try something like this with your own groups, and if you do, please tell us how it went in the comments section when you watch this video. | |
| And if you come up with other ideas about conducting experiments in the virtual classroom, we'd love to hear about them. Thanks. | |
| Olivia Lippincott | Thanks. I'm going to go ahead and stop the recording. That was good. |
