Starting with design of experiments without prior statistical information
Dec 13, 2011 9:56 AM
Recently, a number of customers have asked me how to start with design of experiments if they have technical knowledge about the product to build, or the process steps to follow, but no prior statistical information. They want the next experiment to be planned thoroughly and that data collected in a way that allows for a sound analysis. And they hope that this next experiment becomes the basis of more sophisticated development and results in a significant business improvement.
JMP makes it easy for anyone to design experiments that allow drawing reliable conclusions with the smallest possible number of runs. My favorite example is the construction of a paper helicopter. Here’s a simple way to make one.
We can use JMP to put in the framework of our experiment right away. To access the dialog, click the menu item DOE and select Custom Design. In the first design area, we specify the responses, that is, the variables that we observe to qualify the results of our experiments. In this case, we double-click in the name field of the first row and type in “Time” as it is (the only) variable of interest in this case. We want to maximize the flying time, so we can simply accept the default. For more complex situations, we could add any number of target variables and an arbitrary combination of Maximize, Minimize or Match Target goals. In addition, the responses can be weighted to reflect their relative importance.
As a second step, we list all those factors that we want to combine in different combinations to find the optimal result for our target variables. The experiment that will be designed by JMP is defined by different combinations of factor levels. The only thing we need to know before adding a factor is its data type — that is, is it a continuous variable or a categorical one? Now we enter all our factors one by one: starting with the paper category, whether it’s Regular or Bond (two levels); adding the continuous variable Wing Area, which may range from 12 to 13; and so on, until all driving factors are entered as seen in Figure 1.
Here again the number of factors is unlimited, and any combination of categorical and continuous variables is allowed in JMP.
When the factors are entered, we find the Model area populated with an effect called Intercept and a list of all the factors that we entered. At this point, we need to think about our goals and the technical information or assumption that we might have. Do we want fast decisions about which of the factors are influential at all, and can we ignore the others in improving the process? If so, then we accept the model as is.
On the other hand, do we know or expect that the relationship between a factor and the target variable is not linear? Do we know or expect that factors interact with each other, either in a synergistic or antagonistic way? If so, then we need to add corresponding terms to the model. In the example, Wing Area could have a nonlinear impact upon flight time. So we mark Wing Area in both the model and in the factor list (see Figure 2) and click on the Cross button.
Now JMP displays the recommended number of runs, and we can make the design (Figure 3). This process involves random starts, so building the design several times with the exact same settings does not necessarily lead to identical experiments. But all the designs fulfill the same quality criteria.
When the design is created, a new section called Design Evaluation appears. A useful subheading within Design Evaluation is the Relative Variance of Coefficients. It helps us to see, how likely it is that a factor that is really significant is discovered as significant by the experiment (that is, the Power). You may want to see values higher than 0.8 here, but it all depends upon the signal-to-noise ratio. The higher it is, the earlier you will detect factor influences. If you do have this information and you are not satisfied with the calculated Power, you may go back (using the Back button at the bottom of the window), increase the custom number of runs, generate the design and look at the Power calculation again. You can repeat these steps until you get the right trade-off between number of runs and Power of the experiment.
Next, select Make Table, and JMP generates a data table that holds all your factors with the levels at which you should run every single experiment plus an empty column for each response variable. The variables all have meta-information that saves your specifications from the trial design, and there is a script that you can run when the experiments are done and the data is collected to evaluate the experiments using the model that you specified.
This experimental design has the lowest number of runs for the given task. It required little technical information and no statistical expertise to be calculated. When I explained these steps to my customers, they felt confident that they could improve their way of drawing conclusions and making decisions. So now it’s your turn to apply the customer designer in JMP. How will you use it?