One concern I often hear about the use of software for optimal design of experiments is that the algorithm producing the design is a “black box.” To use the design, an investigator has to trust the black box. An optimal design is one created to maximize some scalar measure of information about some model(s). Generally, though, an investigator has interest in more than one scalar measure of design quality. For example, orthogonality or near-orthogonality is a desirable design characteristic. A measure of the orthogonality of two design columns is their correlation. If there are many factors in a design and if there is interest in two-factor interactions, the number of pairwise correlations to consider can be daunting. In this post, I introduce a diagnostic plot that incorporates all the pairwise correlations in a form that is easy to digest. Using this plot allows a quick and informative evaluation of how close a screening design is to being orthogonal.
Currently, there are several commercial software tools for generating optimal designs. Design quality is a valid concern of any user of such tools. Even if the supplied design is optimal according to the advertised optimality criterion, there are many quality characteristics of a design. Box and Draper in their book Empirical Model-Building and Response Surfaces provide a list of 14 such characteristics. In another, more recent text on response surface modeling, Myers, Montgomery and Anderson-Cook provide their own list of qualities of a good design. Two characteristics of interest here are the ability to precisely estimate model parameters and the ability to assess the adequacy of the model.
In screening experiments, the primary interest is in finding the few factors of many that have the most influence on the response of interest. Such experiments generally have few runs due to budgetary constraints and the preliminary nature of the investigation. In such cases, the investigator hopes that a first-order model including only the main effects of the factors will prove adequate to identify the vital few factors. However, the investigator may have a nagging concern that an active two-factor interaction may bias the analysis, resulting in the selection of an inactive main effect. A more unpleasant prospect is that an active main effect might be masked by an active two-factor interaction having a coefficient of the opposite sign.
A simple approach is to consider two models of interest – the model to fit (primary) and the model that may alias the fitted coefficients (potential). Typically, the number of terms in the combined models is larger than the number of budgeted runs. In a 1994 Technometricsarticle Bill DuMouchel and I considered this two-model framework and developed a Bayesian modification to D-optimal designs to allow some ability to estimate the potential model while guaranteeing that the primary model is estimable. Last year in an article in the Journal of Quality Technology, Chris Nachtsheim and I considered the same model setup. Our objective was to minimize the aliasing of the primary model due to active effects in the potential model while requiring that the statistical efficiency for the primary model not fall below some specified lower bound.
Considering the multiple criteria for design quality, it makes sense to have a diagnostic plot to allow direct evaluation of any design. In the next section, I describe how to make and use one such graph.
How do you make this graph?
For evaluating design orthogonality, I propose using a cell plot showing the pairwise correlation between two model terms as a colored square. Suppose the primary model has k two-level factor main-effects. One reasonable potential model would contain the k(k-1)/2 two-factor interactions. In this case, there are k(k+1)/2 terms of interest. The cell plot is a matrix of colored cells of dimension k(k+1)/2 by k(k+1)/2. For the design in the figure, k = 9 and there are 45x45 = 2025 cells in the plot, each depicting one pairwise correlation. A table containing these values has the same information but is impossible to scan quickly.
To construct this plot, start with the n by k matrix of factor settings where n is the number of runs in the experiment. Add to this matrix k(k-1)/2 columns obtained by the elementwise multiplication of every pair of factors. Now compute the correlation matrix of this matrix by finding all the pairwise correlations among the columns. The result of this computation is a square symmetric matrix of dimension k(k+1)/2 by k(k+1)/2.
The absolute values of the elements of this matrix range from 0 to 1. It is useful to create a color gradient that makes it easy to distinguish small absolute correlations from large ones. Two possibilities are to use a grey scale scheme where a zero correlation is white and a correlation of 1 is black. A more colorful alternative is to use blue for zero correlations with a color gradient to red for absolute correlations of 1. The diagnostic plot below shows an example using the blue to red color gradient.
So what does this tell me?
Note that the diagonal of the plot is a line of red squares. These squares show the correlation of each column with itself, which is, of course, 1.
It may be useful to divide the plot into four sections that divide the terms in the primary model from the potential terms. The figure has a vertical line separating the main-effect terms from the two-factor interaction terms. The horizontal line makes the same separation. These lines divide the plot into four meaningful sections.
The upper-left section of the plot shows the correlations of the primary terms with each other. The most desirable color of all the off-diagonal squares is blue. Colors ranging toward red in this area mean that the terms in the primary model are correlated. The result of this correlation is inflated variance of the coefficients in the model. The figure shows the correlation plot for an orthogonal design, so all the off-diagonal squares in this section are blue.
The upper-right and lower-left rectangles show the correlation between terms in the primary model and the potential terms. Again, the most desirable color for all these squares is blue. As the color of an individual square approaches red, the associated potential term, if it is active, will add increasing bias to the associated primary term. If a square in this region is red, then the potential term and the primary term are confounded, making it impossible to determine which effect is causal. The 16-run textbook design associated with the figure is resolution III, meaning that some two-factor interactions are confounded with main effects. For example, the main effect of factor A is confounded with the DG interaction. Ideally, we would like to avoid such confounding. In my next post, I will introduce two designs that do just what we want.
The lower-right square of the plot shows the correlations between terms in the potential model with other terms in the potential model. If there are fewer runs than terms in the combined models, all of the off-diagonal squares of this section of the plot cannot be blue. If a square in this section is red, then the corresponding potential terms are confounded. Note, for example, that the AB, CH, EI and FG interactions are confounded.
Happily, this plot is already one of the design evaluation tools in JMP so you do not have to follow the construction steps above by hand.