Choose Language Hide Translation Bar
Why design experiments? Reason 4: Clarity

I want to demystify design of experiments (DoE) with a series of posts with simple explanations of some of the terms and concepts that can be confusing for people when they are starting out. In the first post in this series, we looked at the challenge of the vast possibility space in multifactor systems and introduced the concepts of factors, responses and full factorial designs. In the second post,we looked at how we can understand the behaviours in processes, including factor interactions, with visual and statistical models. In the third post, we showed how we build models of the process using data from efficient experiments and explained what we mean by multiple linear regression, main effects and quadratic effects.

We have illustrated all of this with a real-life case study in which we need to understand how to set the Large Volume Injector (LVI) in a GC-MS analysis system. The objective is to optimise the detection of low levels of contaminants in water samples. It seems appropriate that in this post we will be talking about clarity.

Here, when we talk about clarity, we mean clarity about the effects that our different factors (injection volume, injection speed, etc.) have on our response (the sum of all peak heights). We will see how the 26-run experiment that we introduced is almost uniquely able to provide this clarity. As a contrast, we will also look at an example of data that does not bring clarity to our understanding. Along the way, we will explain orthogonality, confounding and aliasing. I want to bring clarity to your understanding of these concepts. We will therefore need to start simple.

### A Simple Example of Correlated Effects, Orthogonality, Aliasing and Confounding

Let’s begin by thinking about just the first two factors for our LVI set-up and these three different experimental designs: Each point on the plot represents a run in one of the three experiments.

• Design #1 is the full factorial for three levels (low, mid and high or L, M and H) of the two factors.
• Design #2 is the same but with two combinations removed – the LH and HL.
• Design #3 has only the LL, MM and HH combinations.

The coverage of the possibility space is clearly reduced from #1 through #2 to #3. Another thing that is changing is the correlation of our factors. The full factorial has 0 correlation between the factors. We say that the design is orthogonal for the main linear effect of these factors. The confidence ellipse for #2 is tighter because it has a moderate correlation, r, of 0.5. We say that there is partial aliasing between these effects in this design. The density ellipse for #3 has collapsed to nothing because there is perfect correlation (r = 1). We say that the main effects are completely aliased or confounded in this design.

What are the consequences of this aliasing? To demonstrate this, we have simulated some response data for each run of each design. The points in the plots are now labelled with the value of the response, SimResp, for the runs. Just by looking at the numbers from Design #1, you can get a sense of the effect of changing Vol(Inj) and Speed(Inj) on the response. You might also be able to get a sense of this from Design #2, but it not as clear.

There is an obvious problem with trying to understand the effect of the two factors from the data for Design #3: It is not possible to separate the effects of the factors. Each change we have made to one factor has been precisely matched with a change in the other factor. You can see why we call this confounding! We have a complete lack of clarity about the effect of our factors and therefore a poor understanding of our process.

As discussed in previous posts in this series, the best way to understand the effects of factors is by fitting a model. We will fit the main effects models to the data from each design or at least we will try to.

In the case of Design #3, it is not possible to fit a model with the main effect of both factors, and we get a singularity warning when we try to fit this model in JMP: This is telling us that we can fit one effect or the other, but not both.

We can fit the main effects model from the data from Designs #1 and #2. Here are the Profiler plots: Profiler plot of main effects model from Design #1. Profiler plot of main effects model from Design #2.

These plots include the confidence intervals that represent our uncertainty about the true effect of the factors. Our model is only an estimate after all. We expect that the line representing the effect is within the shaded bands of the confidence intervals. You can see that our uncertainty about the effects is greater in the case of the model from Design #2. This is partly because we had less data (seven runs versus nine runs) but also because of the partial aliasing of the effects. This is sometimes called variance inflation.

The take-home messages:

• We get the most clarity about the effects of factors with orthogonal experimental designs.
• We can cope with some aliasing – we can learn about the effects but with less precision.
• We can’t learn about the effects of factors that are confounded.

### Experimental Designs For Clarity

Now let’s consider our 26-run design. The next plot is a Scatterplot Matrix of our first three factors. A Scatterplot Matrix shows scatterplots for every pair of variables. This one shows 0 correlation between all pairs of these factors; the design is orthogonal for these main effects. Our understanding of each of these effects is clear of confusion from the other effects. So far this looks like a good design.

We can represent the correlation of the effects more efficiently with a correlation colour map: This map has the same nine squares as the Scatterplot Matrix. Now the correlation of a pair of effects is represented by the colour of the square. The lack of correlation between our main effects is seen here in the pale blue squares. The dark blue diagonal from top left to bottom right is the correlation of each effect with itself (1 of course!) and is there for completeness.
With this more efficient representation of the correlations, we can look at all the main effects. The continuous main effects are all orthogonal with respect to each other. The clarity that this brings is one of the main reasons why these 26 runs make for such a useful design. The slightly darker blue squares under Liner tell us that there is partial aliasing of the categorial Liner effect with the other main effects (r = 0.0836). This should not be a concern. In fact, it is possible to calculate how much larger this would make the confidence intervals versus an orthogonal design (not more than 2.5%).

We can also look at a correlation colour map for all main and quadratic effects. The colour maps below are the same except I have labelled two of the squares in the plot on the right to help with interpretation. The pale blue region in the upper right quadrant represents the correlations between the main effects and the quadratic effects. For example, we can see that there is 0 correlation between the main and quadratic effects of injection volume, Vol(Inj) and Vol(Inj)*Vol(Inj). The map shows that there is a correlation of 0 between every quadratic effect and every main effect: Quadratic and main effects are orthogonal with respect to each other. This is no accident. Again, this was one of the reasons for this choice of experimental design.

The stronger blue in the lower right represents the higher correlations within the quadratic effects. For example, we can see that there is a correlation of 0.4091 between the quadratic effects of injection volume and injection speed, Vol(Inj)*Vol(Inj) and Speed(Inj)*Speed(Inj). The map shows that there is the same correlation of 0.4091 between all quadratic effects: The quadratic effects are partially aliased with respect to each other. This is a compromise that we can live with in this situation. It means that we should be able to understand the important curvilinear behaviours of any of the factors with just a little less precision. The important thing is that there is no confounding in the effects that we are looking at.

We have seen how the data from the 26-run experimental design can bring clarity to our understanding of the important behaviours in our process because of the minimal correlations of effects. Without the methods in software like JMP, it would be challenging to find an experimental plan that has both relatively few runs and these useful correlation properties.

### As Clear As Mud

When data is collected without a plan there is inevitably more correlation, making it more difficult to understand what is driving the behaviours in a system. A classic example is this econometric data (the Longley data set available in the JMP sample data directory): y is a measure of employment and x1 to x6 are potential predictors (x6 appears to be calendar year). The correlation colour map for the main effects shows correlations between x1, x2, x5 and x6 that are very close to 1: The effect on our understanding of the effects of the factors can be seen from a Profiler plot of the full main effects model: The widest confidence intervals are those for the most correlated effects and demonstrate the difficulty in trying to understand a system from data with highly correlated factors. For example, the model estimates that y decreases as x2 increases, but the confidence intervals indicate that the effect of that factor could also be strongly positive. We know almost nothing about the effect of x2.

We have seen that if we want to understand our processes and systems with clarity, we need to generate data from experiments that has minimum correlation between the effects we care about. We have seen that the correlation colour map is a powerful visual tool for understanding how good a design is in this respect. We have only looked at main and quadratic effects so far. What about more complex behaviours like the interactions that we mentioned before? In the next post in this series, we will look at these effects and see why the 26 runs in our experiment are a good choice if we want to ensure that these complex behaviours do not cloud our understanding of the system.

### Notes

 There are a couple of steps if you want to manually determine the correlation of quadratic effects. For example, let's say you wanted to look at the correlation of the quadratic effects of two factors, X1 and X2. You first need to recode the factors so that that they are on the same scale. It is common to code the factors on the scale -1 to +1, with -1 being the lowest setting in the range, 0 being the mid-point and +1 being the highest value. You will often see this coding in books on DoE. With the factors coded in this way, you can then calculate the quadratic effects, X1*X1 and X2*X2, and look at the correlations. The same applies for looking at other higher order effects including interactions. DoE software tools, such as Design Evaluation in JMP, will enable you to evaluate the correlation of effects without having to go through these steps.

Did you miss any posts in this series on Why DoE? No problem. See the whole series here.

Article Labels

There are no labels assigned to this post.

Article Tags
Level II

Hello Phil,

My congratulations for these series of post about DOE. I think they are really good, clear and explanatory about DOE. I personally find that although the concept of DOE is not complicated, the mathematics behind and the REAL understanding of the concept is not that easy and specially the transition from book case studies to real life problems...

I have one question about the topic of correlation of factors or confounding. I understand the concept and i understand correctly, this correlation or confounding can be calculated BEFORE the runs are actually done (measurements), allowing you to know if the experiment that you have in front of you (set of runs) is a good one to understand well your problem.

A good example is the econometric data example where you can tell from the color map that there is a serious correlation between some of the factors (X1-X2 and X5-X6).

Could you please confirm if this color map can be determined before having the measured data (response Y)? So is the color map generated only from the factors information (and therefore, the response is not needed for this)?

And then, once you know this, what can you actually do?
I suppose that if you are actually designing the experiments, JMP can help you to give the settings of the factors that will minimise the correlations (is this right?)

but if we are using existing data (as the econometric data) to do our own study, is there anything else that we can do (more than knowing as you are noting that the model is "as clear as mud" with very little real information about the impact of factors X1, X2 and X5 (with possible negative or positive impact on Y1)?

Thank you again for these series, i think they are great and I would recommend it to anybody trying to do something in DOE

Asier Staff

Yes, looking at the correlation of effects is something that you can do before running the experiment and collecting the response data. We are not condisdering the response at all when we look at correlation of effects.

If you have strongly correlated effects you can augment the data with additional runs to reduce the correlation. The Augment Design platform in JMP's DOE menu provides tools for this.

If you are not able to collect additional runs there are some helpful approaches to modelling. Principal Components Analysis (PCA) can be used to determine a reduced number of orthogonal (zero correlation) variables. These principal components are a combination of the original variables. For example, x1, x2, x5 and x6 from the econometric data, above, would effectively be combined into a single principal compenent because they are all strongly correlated with each other. You could then use the principal components as the variables in your model. Often the principal components can be interpreted as "latent" variables that describe some underlying causal factor in your system.

However, there is no magic. The more correlated your effects are, the less you will be able to understand them.

I hope this helps.

Regards,

Phil

Level II

Hi Phil,

I was re-reading these great series about DoE again and another question came to my my mind.

I used the Longley data to recreate your analysis. I get the same correlations between factors that you mention here but I was surprised to see the graph (actual vs predicted) for the fitting. If i see the fitting below with such a good fit and with such narrow confidence intervals, i think i would assume without hesitation that the model is very good. However, from your analysis and the profiler we also see that we know very little about the impact of effects x1, x2 and x5...

What is the reason for this "contradiction"? i bet there is a good mathematical reason for this "apparent contradiction"

Thanks

asier  Staff

Hi @Asier ,

Yes, I can see how this seems like a contradiction: the correlation between predictors (factors) is bad but the model is able to make good prediction. In fact, there is no contradiction. There is no reason why a model can't be a good fit and also have horribly correlated predictors.

The problem with correlated factors is not that it means we are unable to make good predictions. The problem with correlated factors is that we can't get a good understanding of the effect of each factor.

In this case, if we know the Xs we can make a good prediction of Y (employment). But what we can't do is understand what the effect of each of the Xs is.

So the correlation structure of the factors tells us nothing about the ability to build a well-fitting model and make accurate predictions. It tells us how precisely we will be able to measure the individual effects of each factor.

I hope that helps.

Phil