A Year of Implementing Learnings: How to Design and Analyse Experiments with Pass/Fail Responses

We're here to talk about my presentation for the JMP Summit in Berlin, and it's called A Year of Implementing Learnings: How to Design and Analyse Experiment with Past/Fail responses. It's called so because it's actually kind of a response to your presentation, Don, that was last year in Manchester because that was called how to design, and analyse experiment with past failure responses. Because I've watched that recording again and again, and I sincerely hope someone will find this recording just as usual as I found yours.

Just to give some context on what is this design in made on, and it's actually made out of tube sealer. Now I realized that most people would not know what a tube sealer is. Really, it's just scientific hair straightener. We take silicon tube that we have had vaccine run through, and we then take a hair straightener and seal it. Now, because this is pharmaceutical production, we are a little more scientific about it. But in principle, it's just two heating bodies that presses over a tube and makes a seal.

Now the problem is, we're looking into buying or [inaudible 00:01:21] is looking into buying new thermal sealers for our polymer tubes, and it's paramount that when we make a seal, it stays sealed because that will be our active product that is transferred through, and if it breaks open, we'll have that product on the floor. There's all sorts of issues with that. No patient risk here at all, but we are in the risk of losing a batch of vaccines, which of course is not something we want.

Before taking into production these new sealers, we came with the idea of making this design experiment. First, what is the definition of a success? When have we actually sealed the tube? Now, today it's a visual inspection, so they look at the tools. You can see here in my prior slide that I have like a few pictures of seals. They will visually assess these tubes and give it a pass fail. That's actually all there is.

I really tried to ask, well, can we do something else? Can we make somehow a continuous test of this? No, but we could have another pass fail response, which was the pressure. We were able to put some pressure on the tube, put it underwater. If bubbles come through, well, then it's not very sealed, so kind of what you would do with a bike tire.

I really tried to find a continuous response because it's just so much easier, but then is when I thought of the presentation I went to in Manchester, which was exactly on this topic. Then we went about to setting up the design. I made this DOE because I knew it was relatively easy to take new tubes and make new tests. I really wanted to try... Well, first of all, be mindful about the operator's time and not make more designs or more experiments than needed.

I really had the kind of sequential DOE in mind. It was to start as small as I could, and then if needed, I could put more experiments on it, making it larger and larger, and hopefully going towards having a DOE that explains or gives me an answer to my problem, which is defining a process range for a lot of factors where I will get successful seals every time, or at least the success of failure is very, very low. It will not... Actually, statisticians know, it will never be zero, but we wanted it to be approaching zero.

Now when I was setting up the design, I really wanted to try and infuse this DOE with all the knowledge there was about this sealer because I realized there are people, operators and subject-matter experts, that knows way more than me. Just to make clear, like, I've never actually touched a sealer, and I've yet not touched a sealer.

I need someone in the room that can tell me whether I'm what I'm saying is making sense because there's a chance it's not. I wanted to really try and get that knowledge into the DOE, and we will come back to this later, how I did that and what that actually means to infuse that knowledge into my design.

When we were setting up the design, we had the problem that some of the combination of our factors was not possible. In JMP, it's actually relatively easy to take parts of your design space and cut off. Here you can see I have my cooling temperature and I have cooling type, one is continuous, and the other one is a 2-level categorical.

When we have that kind of scientific hair straightener or tube sealer, it's possible to attach external cooling. It has an internal cooling system, but as you might expect, if you have an external compressor, it's much more effective than an internal compressor because it's relatively small. When we have an external compressor, we are able to change the cooling temperature down. After the tube has been sealed, for that seal to stay closed, you can imagine it has to then get cold, so it will start a cooling process.

Now, as an added benefit, we actually in the end figured out that when we have this external cooling system, it actually significantly reduces the ceiling time. Something we weren't looking at, at all, doing this just realized we should always use external cooling because that's just really cool. But when we use the internal cooling, we cannot have any temperatures lower than 65.

Setting this up is easier than you think, and I have actually made a YouTube video that explains how to do this. If you are watching this recording, there's a QR code you can scan and see how to make this kind of design space.

Final design, 24 combinations, 3 repetitions per combination, giving me 72 welds in total. Thank you will come later, so let's try and look at it. It was really done in three steps. I had an original DOE. Then we did the first augmentation, adding 12 more runs, and then we did a third augmentation adding eight more runs or six more runs. We'll go through the steps of why we did it and the learnings as we go along.

Here's the original design just to show you 24 runs, and you can see here I have three that have failed. I have a total of 3, and then how many failed visual and how many failed at pressure. For this run, you can see whatever setting it had. It had three runs with zero visual fails or zero pressure fails. Now why did I end up with 24? What I forgot to tell you is that this DOE has five, six categorical factors of two levels, four continuous variables factors, and one categorical factor of three levels, so there's really a lot of factors going here, and I ended up just doing it for 24 runs. How did I come to this to that?

Let me show you how that came about. This is where we come back to that idea of infusing my DOE because what I did, and I can cancel this. Let's see if I still have it. Here you can see I have all of my factors. That's a fairly good factor list. This is done in the custom DOE. If I press the response surface model, I will get all their effects by themselves, like main effects. I will get all continuous factors and their curvature effects, so if possible, I will do these four also as a curve. Then I have all their interactions.

Now as a minimum, I would need to do 82 runs to make this design. Now I was not about to do 82 runs with 3 repetitions, giving me being a laughingstock, like 200 and what is that, 246. I'll leave it there with the math. But then what I did is that we went through this list and that took some time and every time they said, you know what, the product and sealing time, no, we don't need that. We kept doing that until we got to this model, which was then our final model, where you go... I need to press back.

You can see we significantly reduced it because they were able to say, you know what, all of those interactions [inaudible 00:09:59] really unlikely that there's going to be there. I still set up the design, so we could identify them if like add runs to identify them later. We really came down to just wanting the main effects and then these like other effects, and that just gave us gave it that minimum of 24.

I was brave, like we're going to go with just that minimum again because I knew I was able to add more runs later. That's why how I came to the 24 number. The next step was then I got this data, and I wanted to be all scientific about it, and I then made a model in the Generate platform. I was really proud of that model, and I'll show you how cool it is.

This was made by following Tom's video and then doing what he was doing while I... Doing what he was doing while watching the video. What we're seeing here is a sealer head, which is, I had an old sealer head or a new sealer head. Gammel is old and new is new. New is new. Then there's two different media we can add. There's two different tube materials. Avantaflex is one and C-Flex is the other.

Then we have a like cutting temperature than the dimension. What I really wanted you to focus on is this tube material, the difference between C-Flex and Avantaflex. Because already at this setting, which you can see 165 temperature and a ceiling time of 100, which is a total, this would be a normal setting and saying, for C-Flex we have a 0.00 probability of failing, and it might... It's as high as 0.04.

But for Avantaflex, you can see how that uncertainty is much, much higher. You can also see what happened to the other uncertainty bands when I change over to Avantaflex that they kind of explode. If I have Avantaflex, and I'm using an old sealer head, it can be really almost all over the place.

The first thing I realized is that there are custom combinations where we're not sure what's happening there. Like, we're not sure that we fully understand what's happening here. But what I did see is, look how beautiful that uncertainty is at higher degrees, so why don't we just turn this up to 180 and everything is fine?

You can see that all of these are now totally flat, and that was my first conclusion. Like spoiler alert, I was wrong, but that was my initial conclusion just looking at the profiler. I think I maybe did data analysis mistake number one. I didn't actually visualize this data enough. Now let me show you what that means.

I came back to them and said, "You know what? If you just turn this up to 180, you're fine." They said, "Yeah, Christian, that sounds good. We want test that." That was the output of the first DOE. I was cocky, said 180, then you're fine. Let's see what happened. Let's see what actually happened. This was the graph that I showed them. What we have here is a ceiling temperature and ceiling time.

As I said, what we really want to get out of this is a range where we have good seal. A blue dot is where the 3 repetitions have all been successful. If just one of the repetitions have been a failure, it will become red. We really want to be in a space where there's only blue. I brought this graph to a meeting and jumped, and I said, you know what, let's just together define that new process range.

We really don't want to be at 140 Celsius. Let's remove that. That's looking better. And 200 is too warm. Some cases it's fine, but we have cases that are red, so let's just turn down that temperature. Then we only have one red dot left, and that has this ridiculously low ceiling time, so if you just not do that, but then we say, well, I actually don't have a lot of data left to prove that that range is a good range. We're trusting the model prediction, but what we then did is, okay, within this space, like, we're zooming in, that looks good. We don't have any data in that space, but it looks promising. That was the next design.

Let's look at that. Now you can see I have the original Dewey, and I have 12 more. If we look at the same plot as we saw just before, no, not that one. This one, we can see that I have these 12 dots here, and we can see one of them came out as red. I was like really hoping everything would be blue. It would be okay if they were red because it was an optimistic design space, but we wanted like, if we could do that, that could be super cool. But one came out as red.

While in that meeting, we played around with a graph builder and made this plot instead. What this plot showed us is that, actually, most of the... What we're looking at is temperature and time as before, but I've divided it into the material and the dimensions. You can see C-Flex and Avantaflex, and then tube dimension half inch, 3 to 3 by 5 inch, and 3 times 9 16 inch.

What we saw is like, wait a minute. Have we never had a successful run with this combination? But, Christian, you said everything was fine above 180. I'm like, yes, I did. But actually, we saw, well, I can't say even if my prediction is telling you this is okay. We've never actually seen that. That was what happened.

One thing I was a little bit proud of was that if we look at the prediction of failure, if you look at all the new ones here and don't see how clear that is, you have 0.00, almost all the way down except for two. If I were to point at just one of these, it would be this one. I know it's only saying a 13 percent chance of failure, but this is like still a relatively small dataset.

I was able to pinpoint, and I did so. I said, If there's going to be a field, it's going to be that one. That was the one that failed. It told me that, I'm not totally off of my prediction.

Third augmentation. The third augmentation or the third run, but second augmentation, then added this new dataset. Now I have the original, the Augment 1, and the Augment 2. If we again look at that same space and a funny thing happened, I don't know if you realized it. I went totally away from making models and just looking at this graph, and if we're getting blue dust, we're seeing what that looks good.

I really should go back and make a model on it, but it was just so visual that looking at the original 24 data points, there's 24 dots, and we can see if I'd made this plot first, I could see, well, I can't say everything is good above 180. I hadn't seen that. Like, both of these dots are above 180, but they're both a failure. But then we added this one, and that's when it dawned on us. You see, all are blue except for one are red.

Then we added the third one, which is exactly in the range that they that vented, the producer of this, has said, I think this point is exactly where they're saying it should be, and then we have only blues. We're saying, canning. These five are relatively easy to seal. If we just not use ridiculously low temperatures, we actually only have two dots except for ridiculously low times. If you just do that, all of this, that's a lot of combinations, a lot of repeats that only gives good results.

The only weird one out is this one combination. The takeaways here is, can we just not use this? Because everyone was telling me it's really difficult to use. If you can just use this one, it would be best, and we're not there yet, but that was my... I'm going to hand it over to you, Don, but that was where we were with this design. We took something that was purely visual inspection, and we made it into data, and we're able to make visualization on that data to draw database decisions. All right, I'll stop sharing. Give it over to you.

Thanks, Christian. I'm going to switch gears a little bit. I'm going to talk about how... Let me just make sure. Let me just do a check. I assume that we are seeing my screen, my journal. Fantastic. Switch gears a little bit. We're going to talk about how I might set something like this up from the beginning to make sure I'm able to have enough runs, experimental runs, and trials to be able to tell whether what I deem is practically important or practically significant, whether that is significantly significant.

Our goal, is to design an experiment. Again, I want to make it large enough, have enough runs, have enough trials to be able to tell whether or something practically significantly significant.

Alright, now, before I begin, I'm going to define a couple of things. Let me talk about my two definitions here, and I want to separate. I talk about trials and I talk about runs. When I talk about a treatment condition or a run, I'm really talking about a row in my experimental design data table, so that's a unique set of treatment combinations or factor settings that I'm going to run.

A trial is, think of it as a replicate. I'm not calling it a replicate because it's really mathematically dealt with a little bit differently. It only appears on one row when we start to do the analysis. Usually when I'm dealing with replications, they're in separate rows in my data table. I want to be able to not only know the number of replications, the number of trials, but I also need to know the number of treatment combinations, the number of experimental runs. We need to separate those two out.

How do we design something that's going to meet my goals? Well, the first part, you should already be familiar. You start with a good design, and we saw that in Christian's example. He got together with subject-matter experts. They discussed how many factors, which factors they think are important, how many of the two factor interactions need to be included, quadratic effects, things like that. That is always the beginning, so no different than how you would handle this if your response was continuous.

All right, now here's where things get a little bit different. You want to be able to say, given your design, given what JMP creates, let's say in the custom designer, you want to know if you have enough trials to be able to tell whether your treatment conditions allow you to distinguish from your baseline. You want to say, "Can I be able to tell whether my baseline is statistically different than my target value?"

In order to do that, what I need to do is I need to calculate a baseline. I need to know what my failure rate or what my event rate is at my baseline, and I also want to be able to calculate or decide what an important change is. For example, my baseline might be something like I'm getting failure 75% of the time, but it really only makes sense for me to distinguish if I have a change that takes me to 50% or 20%, so you want to decide, this a priori.

Once you have those two pieces of information, and if you've designed your experiment in JMP, you have all the tools that you need under the hood to be able to right size this experiment. What you're going to do is you're going to use simulation to ask the question, "Can I get from where I am, my baseline, to where I want to go, my target?"

All right, now let's talk about how I would do that. Again, you start with the DOE, and I've got 1 queued up here similar to what Christian created. In this case, I've got 10 factors, 6 of which are categorical. They all start with CA. Five of those 6 categorical factors are two levels. One is a 3-level categorical factor. I have 4 continuous factors.

When we're looking at the model, all main effects, a subset of my two factor interactions, and a subset of my quadratic effects. Again, similar to what Christian designed. You start out, all talking with subject-matter experts, you decide what it is you want for your design. Use a lot of the same tools, for your design evaluation. When you design your experiment, you might want to look at the color map to say, "Well, I want something with this little correlation, between my effects as possible."

What you don't do, however, is you don't go here. This is not the correct power analysis. We're going to talk about... I'm going to show you how to set up the correct power analysis, and really the only way at it with a pass fail experiment is to do simulation.

Now that I've got my design set up and let's say I'm happy with what I've created, I'm going to do one more thing before I generate my table. I'm going to go under my hotspot, and I'm going to turn on simulate responses. This is one of those tools under the hood that you might not have been aware of that can help you with the simulation.

This is already turned on. I'm going to go ahead and generate my design table, and you'll see what happens is I get two things. I get this box that allows me to say, how big are my effects, or how big do I want my effects, and I get my design table.

Before I get started, I'm going to do a couple of things. I'm going to scroll over to the right to show you what happens here. I am going to reset my coefficients to zero. I'm going to go down here under distributions, and I'm going to click where it says binomial.

I'm going to decide first shot, let's do a simulation. Again, I'll be consistent with what Christian did. Let's say we do, 3 trials. I'm going to click Apply, and you'll notice that what happens is when I click Apply, it generates two different columns for me. One of these columns is a formula column, and we're going to dive in a little bit more deeply into that formula column in a bit to see how I set that up to do my simulation. Another is a fixed column that just gives me the number of trials.

By the way, if you ever, for whatever reason, let's say, close this or forget to open it, when you generate your DOE, that button is always right here under DOE simulate. You always have access to that as long as you don't delete that table script.

All right, now the question is, I've got my design, I've set it up, I've got my columns in each, my binary variable, how do I get my estimates? Now, without going into a whole lot of mathematical detail, which I'm going to leave for a document that I plan on attaching with the rest of these materials, for the discovery talk, I've created a table to make this simple.

What we're going to do is we're going to specify what we think the baseline error rate is, the baseline event rate, and let's say in this case, let's not make it 0.9, let's make it, let's say, 0.75. My baseline error rate is 0.75, and let's say the target I'm interested in meeting is 0.2. It's going to give me an estimate for my baseline, it's going to give me an estimate for a target.

What I'm going to do is, I'm going to copy this baseline, and I'm going to paste it. Let me reset these to 0 again. I'm going to paste that as my intercept, so my baseline is my intercept. I am going to copy my target.

Don, can I ask you a really quick question?

You bet.

Super quick. As I understand it, you are currently at 75, and you will deem a significant change if you go down to 0.2.

That's correct. That is how large of a change I want to be able to see.

Something less than that, you don't care. But going-

That's correct. Absolutely correct.

Thanks.

I've copied my target estimate. I'm going to go in. I'm going to start with the continuous responses. I'm going to paste them in for their continuous responses. I'm going to hold off on the categorical responses because I want a little bit more control. Before I do all that though, let me specify the assumption I'm making here. I'm assuming when I run my DOE, when I have a baseline condition, that's going to be my continuous factors at the center of the design, and my categorical factor is at a specified level.

I can get that for my continuous factors by just pasting them in here, and let's go ahead and paste these in. They're my four continuous factors, my quadratic effects associated with those continuous factors, my two factor interaction associated with my two continuous factors. I'm going to click make sure I've got binomial there, set my binomial to 3 again, and I'm going to click Apply. I'm just making sure that that Y Simulate column changes, and in fact it does.

Now I want to go in, and I want to make some changes to those categorical factors. I want to identify which of my levels I consider my baseline levels for my categorical factors. I'm going to open up my formula that JMP has created. Notice that what it's doing is it's generating a random binomial variable so that every time I run this, I get a different set of values.

Don't get bogged down in all of the details here. The important thing is that here are my factor effects. In this case, this is my factors, my categorical factors 1 through 6. I need to specify, for example, in categorical factor 1, where is my baseline? I want to make sure that my baseline value is 0. Let's say my baseline is in.

What I'm going to do is, I'm going to paste my effect size and out. Again, let's say up is my baseline, so we'll paste that in down. In this case, CA3 has got three levels, so I'm going to leave 1 as my baseline, the other two are going to get my effect, let's say small and large, and so on. I do that with all of my categorical factors where I pick my baseline level as well as any 2 factor interaction involving a categorical effect.

Again, this is 3 levels, so I'll pick two of my levels, and leave this third one as the baseline. Before leaving, I'm going to hit this Apply button just to make sure that that function, the formula gets evaluated. Now I'm set to go. Now I've set my design app so that I can run my simulation. Here's the idea from what I'm going to do, is I'm going to run the analysis once, and then I'm going to use JMP Pro's one-click simulate tool to be able to do my simulation.

Going under Analyze, Fit Model. In the last year's discovery talk, which Christian has mentioned a couple of times, I talk a little bit more detail in terms of how you set the analysis up, what options you have. I'll talk a little bit more detail in the documentation I'm going to provide, but I'm going to skip that for now. I'm going to just set up my design. In this case, you need to remove what JMP puts in there. You need to put in your two columns, your simulated column versus your number of trials.

I'm going to choose generalized regression, and, I'm using the binomial distribution. I like to keep my dialog open in case I have to come back to it. Then I'm going to click run. At this point, I've got to decide, by default, JMP is going to put in all of my effects. Let's do some kind of model reduction. Again, it's staying consistent with what Christian did, we'll pick the Lasso. Occasionally, what happens is the Lasso terminates a little bit early if you force heredity, so I'm going to just turn that off, and I'm going to click Go.

What you find is that that runs 1 iteration. What if I want to do this a thousand times or 10,000 times, collect how many times each of my effects was significant, and determine how often did it correctly ID something as being significant. With JMP Pro, you could do that by hovering over this probability column, this P-value column. You right-click, and towards the bottom, you've got the Simulate option.

What you'll see highlighted is you'll see the formula column highlighted. I'm not going to do 2,500 runs because it would take a couple of minutes, so let's just cut that down to 250. I picked the number of runs I want to do, 2,500 or 10,000, however much, you've got time to afford, and I click Okay. Now what JMP will do is it will rerun that formula and redo the analysis with the Lasso effects heredity turned off, and with all the settings that I picked.

Now, the wonderful thing is that all of this gets collected in a single data table. The power analysis is available in the script that gets embedded in the table, and my power analysis are down here. It does a power analysis that says, "Well, given if I picked an alpha of 0.01, an alpha of 0.05, an alpha of 0.1, what would my simulated power be?"

What we'll see if we look at alpha 0.05 here, my simulated power is somewhere between 33% and 45%, so kind off on the low side. If I want to collect these into 1 table, again, very easy to do. I'm going to right-click, and I'm going to say Make Combined Data Table, and that collects all of my power settings, all my simulated power values for all of my different effects in 1 table.

I've created a little workflow. If currently not on JMP version 16 or 17, you can create workflows, and what I do is I have this workflow which I will also provide. I use that to clean up my table, so we only have to look at the Alpha 0.05. What you'll notice is that, it looks like my powers and by and large, I've got a couple of cases where my powers are middling, 0.5 or so, but nothing really, really great. Nothing 0.8 or higher.

Again, this is, done with 2,500 runs. I actually repeated this with a couple of different conditions to show you what I want to talk about next, and that is what things affect the power that I can get. Obviously, higher power is better. Power is my ability to discriminate an effect when it is in fact real.

Let me close that. Let me clean up my desktop a little bit, and let me go to my... I've collected results. I've written in this case, I think I have 2,500 runs I did for the simulation just to give you a feel. Here's our baseline. Here's what we looked at. This is 1 simulation run. They'd be slightly different if I did this again. 24 runs in my experimental design, 3 trials, just like Christian did, all of my effects significant. I'm going to assume that everything is significant, and my comparison here was... My baseline was 0.75, and my targeting at 0.2.

You'll notice that I've got the intercept, which you often don't care about, significant. I've got one effect that's up towards point A, but everything else is lower. What if I were to have changed that 3 trials to 5 trials? How does that affect my... You'll notice that by going from 3 trials to 5 trials, that does improve my significance.

What if I go from my full model, figure out well, all of my effects are going to be significant, to a subset? Only, yes, I'm going to put all these things in my design. I'm going to want to be able to estimate these, but let's say only a subset, and in this case I picked the subset of 2 continuous and 1 categorical factors, and again things improve a little bit more.

These are these kind of what if scenarios that you can do by running these simulations. Let's try 1 other thing. Let's say my initial design, instead of doing 24 runs, again, let's go to 36 run and it gets even better. What if I wanted to look at a difference, let's say my baseline was not 0.75, but it was 0.90. Again, and by the way, these asterisks are the effects that I specified in the model as being significant.

Again, these are all what if scenarios that you can use JMP to simulate to say, "Okay, this looks good. This is good enough for me to go with this number of runs, this number of trials." Or, maybe I need to try to add some more runs. Maybe I need to add more trials and so on because my power is too low, or I'm good to go where I am.

Let's recap. What things affect power? Obviously, the more experimental runs I have, my power goes up. The number of trials goes up, my power goes up. The difference between my baseline and where they start, that is going to affect my power. Obviously, the more separated they are, the easier they are to distinguish, and the closer they get to the ends, the easier they are to distinguish. As my significant effects goes down, my power goes up. All right, so those are the things that affect power.

Again, let me recap, I wrap things up. You already have your subject-matter experts, you already are thinking about a design. The way you right size these designs is start with a good design. Start with a good design, decide what you want to look at, try to minimize things like the correlation between your effects because that's going to help you with power.

The second thing is decide what your baseline is. Determine what that baseline value is, determine where you want to go, how big of a change is practically significant for you. Put those values, simulate your responses, make sure you set your distribution to binomial, and set up your formula to be able to specify the size of the effects that are important, and then just run your JMP, run your one-click simulation to be able to estimate powers.

If things don't work out as well as you want, if those powers are low, change something like the number of runs, change the number of trials, change what effects you think might be significant and so on. Do what ifs. The great thing is, you always hear you can't get something for nothing. In essence, this is something for nothing. These are experimental runs that don't cost you nearly as much time, certainly not as much money, and they're free for the taking because the tools are already in JMP.

Alright. That's what I wanted to say. I want to thank Christian for actually running one of these designs. It's always nice to see a pass failed experiment in the wild. If anyone, thank you for watching this, and I hope you got something out of it, and if you have any questions, please, post them to the appropriate site in the Discovery Talk or post them to the JMP community. We would be happy to engage you in more conversation and to answer your questions. Thank you very much.

Thank you, Don.