Hello. My name is Bradley Jones. I lead the team of DoE and reliability in JMP, and I want to talk to you today about a family of orthogonal main effects screening designs for mixed level factors. This is a subject which I'm really excited about. We've just submitted a revision to the paper for this and I'm hoping that it will get accepted so we can include it along with definitive screening designs in the DoE platforms in JMP. L et's get started.
My collaborators for this work are Chris Nachtsheim, who's a Professor at the University of Minnesota Carlson School of Business, and Ryan Lekivetz, who's a member of my DoE team at JMP.
Here's my agenda. I'm going to start with a little bit of history and some technical preliminaries. Then I'm going to describe three different constructions for these orthogonal mixed level screening designs. T here are three different ways that we can make them.
I'll show you the JMP scripting language for creating these design sets. That will only be necessary until we can get them built into JMP itself. Then I'll spend a little bit of time looking at the design properties for designs constructed under these three methods. I'll discuss data analysis for these designs and show an example in JMP, and then I'll make a summary and some recommendations at the end. Let's start with some history and motivation.
The first screening designs were fractional factorial designs like apartment designs, or non regular fractional factorial designs or regular fractional factorial designs. For these designs, every factor was at two levels only. En gineers that I have talked to have felt uncomfortable about these designs because they felt that the world tends to be nonlinear and two levels just isn't sufficient to capture nonlinearity in the effect of a factor in a response.
Then in 2011, definitive screening designs arrived and here all the factors were assumed to be continuous and each factor was at three levels, which allows you to fit curves to the relationship between factors and responses. A t the bottom there is the citation for or the reference for the paper that first introduced these designs in 2011.
Now, there are some pros for our initial implementation of DSDs and also some cons. Let's go through the pros first. The pros are that at least in our original implementation, six factor, eight factor and 10 factor definitive screen designs had orthogonal main effects, but we were unable to get orthogonal main effects for more factors. It turns out that a year later, somebody published a nice way of getting orthogonal designs for definitive screening experiments for every even number of factors for which conference matrices were available. That was a big advance.
Another good thing about DSDs is that the main effects are orthogonal two- factor interactions so that the estimate of a main effect will never be biased by any active two- factor interaction.
The really exciting aspect about DSDs is that all the quadratic effects were estimable, which is never possible with screening designs. Even with centerpoints, you can detect that there is a non linear effect, you just don't know where it's coming from.
Then finally, in DSDs, if there are six factors or more, and only three of the factors turn out to be important, then at the same time you do screening, you can also do response surface optimization. You could maybe get, if you're lucky, a screening design and a response surface design t o optimize a process in one shot. You have to be lucky, of course, you have to have three or fewer active effects or factors.
The cons of the initial implementation of definitive screening designs is that first, they couldn't accommodate categorical factors. Secondly, they couldn't accommodate any blocking. Thirdly, some researchers have pointed out that for detecting small quadratic effects, that is, quadratic effects where the size of the effect is about the same order as the error standard deviation, there is low power for these estimates, detecting them successfully. Of course, if the quadratic effect is big, then of course you can detect it, especially if it's three times as big as the error standard deviation.
Now, after the original publication of DSDs, we were well aware that it was a problem that we couldn't accommodate two level categorical factors. So we wrote a new paper in Journal of Quality Technology in 2013 that showed how to incorporate 2- level categorical factors.
Then in 2016, we wrote another paper in technometrics that showed how to block definitive screen designs using orthogonal blocks, blocks that are orthogonal to the factors. W e were trying step- by- step to address the cons that were associated with original implementation of this methodology. Another thing that we noticed was that people were having a little bit of trouble knowing how to analyze definitive screening designs. We invented an analysis technique that was based on our understanding of the structure of a DSD. It took particular advantage of the special structure of a DSD to make the analysis sensitive to that structure. Rather than trying to use some generic model selection tool like stepwise or lasso or one of those.
This made it possible for a non-expert in model selection to use this out of the box automated technique for analyzing a definitive screening design. That was in 2017, but there's still problems.
First, we did write that paper that added categorical factors to a DSD, but if you had more than three, the quality of the design went down, and that was undesirable. In fact, if you had too many categorical factors, things didn't look good at all. That was an issue. A gain, quadratic effects have been pointed out to have low power if they're small. The purpose of this talk is to introduce a new family of designs that addresses these issues. H ere we go.
First, I have to do some technical preliminaries to explain what we need to have the ability to do in order to build these designs. I'm going to start out by talking about conference matrices. The conference matrix is the tool that the second paper that discussed DSDs in 2012 and introduced all the orthogonal DSDs for twelve factors, 14 factors, 16 factors, and so on. They use conference matrices to make that happen.
I need to show you what a conference matrix is. You can see here, conference matrix, that is four factors with four runs, and there are zeros on the diagonal elements and ones and minus ones off the diagonal. T he cool thing about a conference matrix is that if you multiply the transpose of conference matrix with the conference matrix, you get an identity matrix times the one minus the number of rows in the design. I t's an orthogonal design.
Now, conference matrices exist when the number of rows and columns is equal. They're square matrices and they only exist when the number of rows and columns is an even number. There's a conference matrix for every even number of rows and columns from two rows or and two columns to 30 rows and 30 columns, except for the case where the number of rows and columns is 22. It's actually been proven that the case where there are 2 2 rows and colum ns, the conference matrix has been proven not to exist. So there's no way to construct one, sadly, although I can't prove that result myself.
Okay, the next thing I need to talk about is something called a Kronecker product, which uses that circular symbol with an X in the middle of it, so that when you see that in an equation, it means you want to make a Kronecker product. The Kronecker product is also called a direct product, and in fact, JMP scripting languages, language JSL makes Kronecker products of matrices using the direct product command, not the Kronecker product command.
The Kronecker product of a vector one stacked on top of negative one with a conference matrix stacks C on top of negative C as below. What the Kronecker product does is for every element in the first matrix, it substitutes that element times the second matrix. So one times the conference matrix is just the conference matrix, and negative one times the conference matrix is negative of the conference matrix.
Basically, a Kronecker product of one minus one with a conference matrix just stacks the conference matrix on top of itself. So here's a case where I did just that. You have a four by four conference matrix on top of its fold , which is also four by four, and if you were to add a row zero, you'd have a four factor definitive screening design.
Conference matrices are useful, Kronecker products are also very useful for constructing designs, as it turns out. I have a few more preliminaries to go over. Let me talk a little bit about Hadamard matrices. Hadamard matrices are also square matrices, but they're constructed of ones and minus ones. We have Hadamard matrices built into JMP for every multiple of four runs or four rows and four columns up to 668 rows and 668 columns. So every multiple of four, we can support a Hadamard matrix.
That well known. Hadamard designs are the Plackett–Burman designs and the two level fractional factorial designs. These are both Hadamard matrices. Hadamard was a French mathematician who lived in the late 19th century, and he invented this idea. I f I had my matrix as m rows, it's transposed times itself. Is m times the identity matrix.
T hat means the Hadamard matrix is orthogonal, number one, and number two, it has the greatest possible information about the rows and the columns in the matrix that's possible, given that you're using numbers between negative one and one. They're very valuable tools for constructing designs.
Now we have everything that we need to show how to construct these new designs. We call them orthogonal mix level designs or OMLs. They're mixed level because half of the columns or half of the factors are three levels. Therefore, the three level continuous factors and half of the columns or factors are two levels, and they're for categorical factors or for continuous factors for which we're not worried about nonlinear effects.
Here's the first method for constructing one of our OMLs. If C sub K is a k by k conference matrix and H sub 2k is a 2k Hadamard matrix, so H sub 2K is a Hadamard matrix which has twice as many rows and columns as the conference matrix has.
Then if we stack a conference matrix on top of its foldover and then replicate that, we get this matrix DD, which is just C negative C, C negative C, all stacked on top of each other. DD is two DSDs stacked above each other, minus the two center runs that DSDs normally have.
Now, HH is because DD ends up having 4k runs because there are four conference matrices with K rows and columns stacked on top of each other. So there are 4k rows in this design and K columns. HH is just Hatamard matrix stacked on top of its fold over design.
Since H has two k rows and columns already, stacking it on top of itself makes it have 4k rows just like the DD matrix has. It turns out that you can just concatenate these two matrices horizontally to make an orthogonal multilevel design. The DD part of it all has three levels per factor. And the HH part of it has two levels per factor. And you can see that there are k, three level factors and two k two level factors. Therefore, a 4k row design you can have as many as 3k columns.
For example, if your design K was six, you'd have 24 rows and 18 columns. Six of the factors would be three levels and twelve of them would be two levels. Now you have way more two- level factors and you haven't lost any of the features of the definitive screening design. The main effects of this design are orthogonal to each other.
Here's an example where I constructed an OML from a 6 by 6 conference matrix and a 12 by 12 Hadamard matrix. You can see there are 24 rows in this matrix and 18 columns. The first six of them are the six, three- level columns, and the next twelve are the twelve two- level columns.
Now, of course, you don't need to use every column of this design. You could still use this design even if you had say, four or five three level factors and seven two level factors. You just remove five of the two level factors and a couple of the three level factors, and it's sort of arbitrary which ones you might remove.
Here's the second construction approach here. C sub K is a K by K conference matrix, and H sub K is a K by K Hadamard matrix. Now we're going to create DD the same way we did before. DD is just a definitive screening design stacked on top of itself, minus the two synergies. HH is a replicated Hadamard matrix on top of the same Hadamard matrix folded over twice.
Now if you look at the two columns of ones and minus ones, you might notice that those two vectors are orthogonal to each other. That's what makes this particular construction really, really powerful. In this case, the design has 4K rows and only two K columns. K of the factors are at three levels and K factors are at two levels.
The number of runs in this design is twice the number of columns. But that's still a very efficient number of runs given the number of factors. It's the same effect of definitive screening designs in fact. Definitive screening designs have twice as many, twice as many plus one runs than factors.
Here's an example created using a 4 by 4 conference matrix and a 4 by 4 H adamard matrix. When you stack them on top of each other four times, you get eight columns and 16 rows. Columns A through D, you can see are three levels because you can see those zeros and there are four zeros in every column.
I should point out that if you had a definitive screening design, there are only three zeros in each column. Having an extra zero makes the power for detecting a quadratic effect a little higher than for the definitive screened design. That's the second construction method.
The third construction method is very similar to the second, except that you have two different ways of adding Hadamard matrices to the example. Here we have the DD part is the same as the first two construction methods. The HH part, there are two HH things. One with the vector 1, 1 -1 , -1 , and the other one with the vector 1, -1 , -1, 1 . The three vectors that are ones and negative ones are all orthogonal to each other. That yields an orthogonal main effects design. In this case, the third construction again has 4K rows and 3K columns, that is K factors at three levels and two K factors at two levels.
Those are the three methods. Here's an example of that construction with using a 4 by 4 conference matrix and a 4 by 4 Hadamard matrix. The result is a twelve- column design with 16 rows. Twelve factors in 16 rows. Very efficient design for looking at twelve factors. It's also orthogonal for the main effects. Main effects are orthogonal to two- factor interactions.
Now I want to show you three scripts for creating these designs using JSL. In the meantime, before we drop this methodology into JMP, you can create these designs with a very simple JSL script. The first command is creating a conference matrix, in this case conference matrix with six rows and six columns.
Then D is the direct product of the vector 1, -1 , 1, -1 and C. That gives you the matrix DD that we saw in our constructions. Then eight is a Hadamard matrix with twelve rows and twelve columns. Notice that twelve is two times six. We were requiring that for the first construction, the Hadamard matrix has to have twice as many rows and columns as the conference matrix.
We make HH by multiplying 1, -1 by using the direct product, the Carnegie Product Construction. That gives you a 24- run HH design. The H thing has 24 runs and twelve columns. Then the last step is to horizontally concatenate D and HH to produce ML which I just shortened, shortened OML to ML. Then the as table command makes a table out of that matrix.
The OML that we just created has 24 rows and 18 columns. Six of the columns are factors at three levels, and twelve of the factors are at two levels. Now, the six in the first line can be replaced by eight, 10, 12, 14 up to 30, except for 22. The twelve in the third line must be twice whatever number you put in the first line. You can use this construction to create all kinds of OML designs just by changing the numbers in the first and third columns.
Here's the second construction method script. I start again with the conference matrix. This time I'm doing a conference matrix of four rows and four columns. D is a direct product of 1, -1 , 1, -1 and C. That's the same as before.
This time I make H be a Hadamard of four instead of twice the number in the first line. I have to have a vector with four elements to direct product with H. I use 1, 1, -1, -1, use the chronicle product of that or the direct product in JMP and JSL speak. I get a design that has 16 rows and eight columns by horizontally concatenating that double vertical line thing horizontally concatenates two matrices and then the S table makes the table out of it. This second construction has 16 rows and eight columns. There are four factors at three levels and four factors at two levels. The four in the first and third lines can be replaced with any even number for which a conference matrix exists.
I need to correct myself. The conference matrix has to be a multiple of four in order for this to work, because the Hadamard matrix is a multiple of four.
Here's the last construction method. Again, we have a conference matrix of four, but it could be four, it could be eight, or twelve or 16. We make a direct product of this vector of ones and negative ones and C to get the replicated definitive screening design. Here we create a Hadamard matrix of four, but we have two different direct products. The first one where we're making a chronicle product of 1, 1, -1, -1 with H, and the second one we're making a chronicle product or direct product of 1, -1 , -1 , 1 and H.
Those are two different matrices and they happen to be orthogonal to each other. Then we horizontally concatenate all three of these matrices and make a table from that. This design now has 16 rows because it's four runs in the conference matrix times four. You have 16 rows and twelve columns. There are four factors at three levels and eight factors at two levels. Here are three very easy JSL scripts. To make these designs, I'll put JSL into the... When it goes into the JMP community, I'll add the JSL. I'll also add several examples of these OML designs that you can use.
Now I want to talk about a little bit about the properties of these designs. Here we see the design properties for method one and the colour map and the correlations shows that there are no correlations between any of the 12 factors in this or actually 18 factors in this design. The three- level factors are about 10% less efficiently estimating than the main effects than the two- level factors. That's because of the zeros in each of those columns. That doesn't help you, the zeros don't help you estimate main effects.
Now I want to show you the alias matrices for this design construction method. You can see that there are a lot of main effects that are uncorrelated with two- factor interactions, but there are also a lot of main effects that are correlated with two- factor interactions.
The three- level factors main effects are not alias with any of their two factor interaction. The same is also true of the two- level factors. Their main effects are not alias with their two- factor interactions because both sides of this design are constructed from fold- over designs. W e see that there's quite a bit of potential aliasing of main effects from active two-factor interactions. In some sense, method one is a little riskier to use than the other methods.
Here are the design properties for method two. You can see that here I'm making a design that has 16 factors and eight columns, I mean 16 rows f or eight factors. The three- level factors have 15% longer confidence intervals than the two- level factors. Again, that is because those four factors all have four zeros. Four of the 16 runs are zero instead of 1 or -1.
The cool thing about the second design construction is that none of the main effects is correlated with any two- factor interactions. That has many of the desirable effect or characteristics of a definitive screening design. There's a lot of orthogonality between pairs of two- factor interactions, but there are also some correlations. You can see here are some correlations, here are some, and so on.
Finally, the design properties for method three show that three- level factors are a little less efficiently estimated than two -level factors 15%, 15.5%. We can see that there's some aliasing between main effects and two- factor interactions, but not as much as for the design construction number one. In terms of risk, this accommodates more factors with less risk than the first construction method.
I'd like now to compare DSD to an orthogonal main effect or mixed- level design. You can make a DSD with eight factors, eight, three- level factors, and that would have 17 runs. I f you get rid of the centre run, you would have a run that's directly comparable with a multi- level 16- run design that you've seen in design construction too.
Now, if we compare the efficiency for estimating main effects, the definitive screening design is only 91% D efficient with respect to the mixed- level design, the G efficiency is 92%, the A efficiency is roughly 90% and the I efficiency is roughly 82%. Y ou can see that the fraction of the design space plot shows that the curve for the mixed- level design is below the curve for the definitive screen design pretty much everywhere. This design is clearly preferable to the defendant screen design for estimating main effects at least.
Now I want to talk about data analysis and use an example. I've created a design using the second construction and I created a Y vector by adding random normal errors to a specific function with both main effects and two- factor interactions and rounding it to two decimal places. The true equation, the true function is this one here. It has A, B, E and F main effects are all active and the AB, BE and EF two- factor interactions are all active.
That's a function without error. I added normal random errors with a standard deviation of one. What I used was since this design can be fit using the Fit Definitive Screening Design platform within JMP, that's what I used. Here you see that the Fit Definitive Screening Design finds all seven real effects and doesn't find any spurious effects. It gets the exact correct set of effects.
The deviation between the true parameter values and the estimated parameter values are pretty small. For example, the true parameter value for factor A is 2.03 and its estimate is 2.45 plus or -35. That's one standard deviation so it's just a little bit more than one standard deviation from its true value.
Here that the true value of the coefficient of B is 3.88 and I get 3.94, which is very close to its exact correct value. You can see for yourself the estimated value of the root mean squared error is 1.2 and the true amount of random error I added was exactly one. You can see again, this analysis procedure has really chosen the exact correct analysis.
Now I want to do a little JMP demo that shows basically the actual by predictive plot that you see there below. Then this plot shows that the residuals don't have any indication of any problem as well. I'm going to leave PowerPoint for a second and just go to JMP. Here's my data, here's the function with no error. I can show you that, that's just this formula that I showed you in the slide.
Then here's the data where I've added random error to each of these values with it. Then what delta is, is the difference between the prediction formula of Y and the Y with no error. These values are how far we missed the true value of Y for every point.
If I do the Fit Definitive Screening Design of Y, I get what I showed you before and I get the correct main effects and also the correct two factor interactions. Then when I combine them, I get the correct model with the correct art, very close to the true estimate of sigma.
If I run this model using Fit model, this is the actual by predicted plot I get. This is the residual plot I get. Here's the prediction profiler that shows the predictive value of Y. You can see that if you look at the slope of the line of B and see as I change the value of A, the slope of B changes. If I change this B, the slope of A changes, if I change E, the slope of F changes.
This indicates interactions happening. If I wanted to maximize this function, I would choose the high value for each of the factors. Then one of the things I did was I created a profiler that where I look at this is the true function, this is my predictive formula and this is the difference between those two functions.
This is the setting that leads to the largest difference between the predictive value and the true value of the function. That's what I wanted to show you in JMP and I'll move back to my slides now. Let me summarize, we've talked about definitive screen designs with their pros and cons.
I then introduced the idea of a Chronicler product and showed how to construct these orthogonal multilevel designs in three different ways. I showed you the JSL scripts, I shared the JSL scripts for constructing these designs.
You can use this script by changing the numbers in the first and third lines to make designs that with increasingly large numbers of runs. I talked about the statistical properties of these designs and particular showed that their orthogonality, but also in the case of design construction two, not only are they orthogonal for the main effects, but the main effects are orthogonal to the two factor interactions.
Design construction two only exists for designs that have a multiple of 16 rows and columns, which is a slight disadvantage compared, there's more flexibility with the other approaches. Then finally I showed how to analyze these designs. Let me make a couple of recommendations.
Design construction method two is the safest approach because of all of the orthogonality involved and the fact that the two factor interactions are uncorrelated with main effects. I pointed out already that you don't have to use all the columns.
You can create one of these designed and then throw away certain columns in order to accommodate the true number of factors that you have. The advantage of doing that is that you will also get better estimates of the error variance.
Then it's important to remember that the three level factors are for continuous factors only. It wouldn't make sense to have three level categorical factors for these columns because there are far fewer zero elements than ones and plus ones.
A couple of more things. Quadratic effects it turns out, are slightly better estimated by an orthogonal main mixed level design than a DSD. But if you wanted to improve the quadratic effect estimation, you could add two rows of zeros to the continuous factors. Those would be like center points.
Then for the categorical factor, you can choose any vector of plus ones that has plus ones and minus ones in it. Then the other, the second row would have just the fold over of the first row in the ones and minus ones. That is all I have for you today. Thank you very much for your attention.
