Help: Best practices regarding DOE blocking, split/strip plots, covariates, nuisance factors, and ANCOVA

[#1]

Hi All,

  Hoping to get some feedback from experts and the collective mind regarding the above topics. I'll first describe the scenario and then go into the problem at hand.

SCENARIO: Our business is often in a situation where DOEs run by people in R&D suffer from an external nuisance factor -- and managing this in order to "save" the DOE is sometimes challenging. The R&D people will typically generate DOEs to test several factors at a time. In this process, there are normally multiples of 3 batches made, each batch has a different recipe in order to test the factors affecting the outcome(s). The factors going into the recipe are all continuous factors, and depending on the design are typically 2-level (low/high). Center points are often included to help estimate the variance. The R&D people typically make 3, 6, 9, etc. batches. The reason is because after the initial steps in the process, the batches need to be dried in an oven, and it's only possible to dry three batches at a time, hence batch quantities in multiples of 3. Batches are immediately taken to be dried after the first step. Directly after drying, the material is milled. Because of limited time within a day, there can only be a fixed number of runs performed each day, again this would be a multiple of 3, with no more than maybe 9 done in one day.

PROBLEM: Because of physical limitations of where the oven is located, the external ambient temperature and humidity (more often humidity) are nuisance factors. I call them nuisance in the sense that they are known, but uncontrollable. The ambient temp and humidity can be measured at the time a set of 3 batches are placed in the oven for drying, but there is no way to control these factors or to know what they will be ahead of time. It is known that the humidity can affect the material during (and/or shortly after) the drying step, and it's suspected the ambient temp might do so as well. 

DISCUSSION: What I'd like to get a better understanding of is how to manage these nuisance factors in a DOE. Certainly, we can block by day and have a well-distributed, random set of runs within each block, thereby obtaining an estimate of the noise introduced by which day the batches were made and dried. We could also introduce it as a fixed effect (even though it's not really fixed), which would help to estimate the mean day-to-day differences in each block. We could even treat "day" as a "very hard" to change effect, to obtain a split plot design, which would help to see how that factor interacts with the other factors -- BUT -- we can't control the humidity and don't know what it will be before the experimental run is started. We could treat it as a covariate, which would require doing an ANCOVA analysis to remove the effect of the nuisance factor from the response. Or, we could simply introduce the nuisance factor as an uncontrolled factor.

Some discussion points:

1. It would be nice, although not necessary to estimate the noise that is introduced because of the nuisance factors, which would suggest randomized blocking.
2. It would also be nice to estimate the day-to-day variation caused by the nuisance factors, which would suggest fixed blocking.
3. It would be nice too see if, or how, these nuisance factors are interacting with the other known main factors, suggesting including them as "very hard" to change factors, but indeed they are impossible to change factors as we have no control over them.
4. Performing an ANCOVA might help to remove the nuisance factors in order to compare batches made on different days. However, it's my understanding that to do this in JMP, one can only have a single categorical factor combined with a continuous covariate factor in order to obtain the right kind of analysis. We certainly could change some of our main factors to categorical data types, but the problem is that we have several main factors that must be included in any model. When doing so, JMP no longer splits the data into the different lines of fit as you would expect for an ANCOVA. Although maybe I don't fully understand how JMP is performing the ANCOVA under situations where you have multiple factors in the model. The sample data and online help on this topic is very limited.
5. Lastly, we could add it into a model as an uncontrolled variable, leaving the levels blank and fill in the data when the runs are executed. The analysis of the model later should show whether or not the uncontrolled variable has any impact on the response(s).

Some questions:

1. In this kind of scenario where we typically have a fixed number of batches dried at a time, and all of them could be affected by the nuisance factors, is there a best approach in incorporating the nuisance factors into a model in order to understand what effect, if any, it has on the response?
2. Would we use a singe approach or mixed approaches. For example, is it best to use only randomized (or fixed) blocking, or to use randomized blocking, but also include the nuisance factors as uncontrolled variables? Similarly, is it best to use Day as a very hard to change factor while also including humidity and air temp as uncontrolled factors?
3. Is ANCOVA really the right kind of approach, or could this lead to wrong conclusions?
4. Have you had to deal with a similar situation in your workplace, and if so, what methodology did you use to manage your nuisance factors?

Thanks for your help, and I look forward to reading the responses.

Thanks!,

DS

[#2]

DS,  First let me say your post is quite well written and detailed, though there are some critical missing points.  What is R&D designing, a recipe for a product that is made in batches?  Or is it a process for making batches?  Or something else? Do you need to consider mixture components of a formulation?  Are you trying to "pick a winner" or do you want to investigate the causal structure and learn how and why it works...which requires iteration? Secondly, there is no one correct way to plan experiments.  Do due diligence (develop hypotheses, identify all x's, link hypotheses to x's. determine what effects you need to estimate, predict/rank order of model effects (at least through 2nd order), etc.), design multiple plans, predict all possible outcomes (and what you will do in every outcome) and weigh the potential knowledge gained vs. cost to gain that knowledge.  

That being said, there are certainly a number of options as you indicate.  Here are my initial thoughts:

1. I'll call your nuisance factors noise.  I suggest there may be more than the ones you've specified in your post (e.g., measurement errors, lot-to-lot variation of ingredients, consistent of mix time, within batch temperature gradients, cleanliness of the vat, ability to adhere to recipe, within oven variation, etc.), but certainly I understand the ones you have specified are ones you have strong hypotheses about.  While I understand you are in R&D, has anyone studied the actual manufacturing processes (any process mapping completed?)?  Any component of variation studies (i.e., directed sampling) done to estimate measurement errors, within batch variation, batch-to-batch variation, lot-to-lot variation of ingredients, ambient conditions, oven-to-oven, etc.?  Having a good idea of how much the noise varies can help determine how to include that noise in your experiments.  Also, how you can manage noise over a short-term period has an effect on which options to use.

2. What are the response variables?  Are the measurement systems adequate?  Are you interested in factor effects on both mean and variation?  You may want repeats for estimating a variance response.  Particularly useful for short-term noise components like within batch variation.

3. In the R&D environment, typically you want to design that is robust to noise.  The noise for a designer is typically the manufacturing process, raw materials and use of the product in the hands of the customer.  I'll define robustness of the design as the absence of noise-by-factor interactions, meaning the designer is looking for a design that performs consistently over changing noise.  So the designer needs to expose potential noise-by-factor interactions as early in the design process as possible.  This will give the designer the most options for remedying the situation (vs. discovering the issue once you're in production). With this in mind, you could:

  a) run blocks, purposely holding noise as constant as possible within the block (you might need to use equipment to manage the noise for a short period of time) and then changing the noise, exaggerated to expand the inference space appropriately (see CoV studies).  Treat the blocks as fixed effects and get block-by-factor interactions to estimate robustness.

 b) run split-plots where noise could either be in the whole plot or subplot (cross-product arrays, similar to a Taguchi concept, only analyzed a split-plot).  These designs provide excellent resolution with increased precision for both the whole plot and subplot.

4. While you could perhaps measure the noise and account for that variation in the model (covariate), while this will increase the precision of the statistical tests, it may not provide the insight you need.  Again the question is still you want a winner or understanding?  Do you want a model that seems to work regardless of understanding?

[#3]

Hi @statman ,

  Thanks for your reply and thoughts/comments. Below are some replies to your questions.

1. Unfortunately, I can't share too much regarding the details, but I'll share what I can (in general terms): Usually, R&D develops a recipe for a product that will ultimately be produced in a continuous process. So, in a way, it is both a recipe and a process. The lab-scale material is made in batches, whereas the process is a continuous process. We do not need to consider mixtures because there's no fixed rule that the sum of all components has to equal a certain value. What matters more is the ratio of one component to the whole. For example, X1 might be 0.05% of the whole and X2 might be 0.1% of the whole, but there's no limitation that says X1+X2+X3+...Xn must equal 1. We know that going from the lab-batch process to the continuous process is not a direct 1:1 or a fixed ratio change in the ingredients, but once we have a robust lab product, it's pretty straight forward to transfer this to the continuous production process.
2. It is well understood what the different components of the recipe do to the measured outcomes. We know that changing different X's will result in a corresponding change in certain Y's. However, when customers ask for material with specific properties, we can change our X's in order to achieve the desired Y's that the customer wants. Following the example above, one customer might have X1 set to 0.025% and X2 at 0.075% while another customer might have X1 at 0.1% and X2 at 0.1%; it all depends on their needs. R&D does DOEs at this stage because there are several different approaches of changing the X's that can actually result in very similar Y's after the drying and milling stages. Some of those approaches are more robust than others and some of those (not necessarily the same) result in better materials after the second stage in the process -- however, that's a different matter and has it's own set of DOEs taking material from the first round as covariates to feed into the next round of DOEs for the second stage -- meaning: based on a certain outcome Y_target, batches within +/-w units of Y_target will be selected (using JMP) as covariate inputs to the new set of X's during the second stage. But again, as I said, this is a different stage in the process and is not impacted by ambient temp and humidity because that part of the lab process is performed in a controlled environment.
3. We have evaluated our raw material suppliers, age of raw materials, cleanliness of lab equipment, etc. and found that the noise introduced by these factors are much less than the noise of the measurement system. That being said, we have also tested whether our measurement system is capable, and it is for the responses that we measure -- given the standard deviation and difference to detect between experimental runs.
4. As for the actual manufacturing process, yes, we've mapped that out and know where the largest sources of noise are.
5. I would say that in general, from the DOEs, R&D would like to find the most robust (to noise) material that matches customer specifications. But, there are external noise factors (primarily temp & relative humidity) that can alter the outcome. We'd like to both understand how the humidity and temp are affecting the outcome as well as account for this variation within the DOE so that we can compare samples made on different days. I had mentioned that the noise of the measurement systems are larger than noise from batch-to-batch variation or raw material supplier or incoming lot material etc. But, the noise introduced by the ambient temp and RH can be larger than the measurement noise. For example, if 3 batches using all the same recipe are made individually and all dried and milled together, they will all show the same Y responses within the typical measurement error of our system. However, if the same recipes are repeated on a day where the RH is high (in practice this is often near the 75-80%+ RH level), measurements on the same Y responses will result in significantly different values. As mentioned in the first post, we can't control the RH or ambient temp, and we also can't tell the R&D folks to never do experiments on days with high RH, so having a DOE framework that can account for this day-to-day variation and hopefully even explain a little about HOW the noise is affecting our Y outcomes would be extremely helpful.

  I know there is no one way to design the experiments, but having some generalized framework would be helpful so that R&D can manage this noise and not get lost or waste their time trying to recover a DOE that has been affected by this noise. Even a kind of a flow-chart concept would be helpful to give them so they have some kind of structured approach. 

  What I find difficult, and is alluded to in the title of this post, is when and why should an experimenter use random blocking, fixed blocking, split/strip plots, covariates, uncontrolled variables, or ANCOVA to manage external noise like what we're experiencing. The goal is to understand how and why the noise is affecting our Y outcomes in order to develop a recipe that is robust against such noise (temp/RH). One of the difficulties is that this will change based on the relative amounts of X1, X2, X3, etc, which are different for each customer's different needs -- the result is that once we figure this out for one situation, we won't necessarily be able to extend it to all other situations. I know the method shouldn't be fixed in stone, but at least having a logical decision path is better than nothing. So far, I have not been able to find much within the JMP help or online webinars that does a good job of differentiating between the different approaches and when/why/where one should consider a certain approach over another.

  I hope this helps to better understand what we're dealing with. Again, appreciate your feedback and thoughts/comments. I hope to also hear from others on this matter, as I'm sure we are not the only people that have to deal with something like this.

Thanks!,

DS

[#4]

Unfortunately, I think this is more of a one-on-one discussion and difficult to do via this forum.  I have several flowcharts and templates for helping to plan experiments, but they are not for general consumption and are situation dependent. I will give you some general guidance which, of course, can be debated.  There are four "philosophies" for handling noise:

1. What I find most typical among engineers and scientists is the desire to hold noise constant during the experiments.  The thought is that will greatly reduce the error and make factors easy to identify.  Unfortunately, this reduces the inference space and therefore is a poor strategy for analytical work (so don't do this).

2. The typical approach is to have some sort of randomized replication.  This will increase inference space, may provide a better estimate of the random errors therefore providing a "better" statistical test (and somehow improve the likelihood of meeting the analysis assumptions).  This may negatively impact the precision of the design.

3. Another statistical approach is to assign the noise (or set of noise factors) so its effect can be removed from the error estimate (MSE) thus increasing the precision of the statistical test.  The use of random blocks as well as use of covariates can accomplish this .  This also has the added benefit of increasing the inference space. Emphasis is on statistical tests of significance.

4. A perhaps more analytical approach is to first identify the noise, develop hypotheses about the potential noise effects and then purposely manipulate those noise factors.  This can be accomplished using blocks as fixed effects (because you are actually combining multiple noise factors and setting them all at extreme levels in the experiment).  This approach increases inference space while not negatively affecting precision and also allows for estimating robustness.

"Block what you can, randomize what you cannot." G.E.P. Box

The split-plot experiment ideas are used when you have specific noise variables that can be manipulated and are factorialized in the experiment.  This provides increased inference space, increased precision, and excellent resolution of the design factor by noise interactions.

I highly recommend this paper:

Box, G.E.P., Stephen Jones (1992), “Split-plot designs for robust product experimentation”, Journal of Applied Statistics, Vol. 19, No. 1

[#5]

Hi @statman ,

  I really appreciate your thoughts and feedback on this discussion. Honestly, I'm a little surprised that no one from JMP, or other professionals in the field, have joined in on this discussion. I would have thought there would be several others in a similar situation and looking for ways to manage the nuisance/noise factors that are uncontrollable.

  Well, based on the discussion and what our group can really manipulate wrt the noise factors, it seems to me that option 3 above would be our best choice -- which would be a mixed approach where randomized blocks are generated and the noise is added as a covariate in the model. Although this might not be the best approach, like #4, I think it is a pragmatic approach that we can use to at least help with some of the issues we're facing.

Thanks again for your help and good discussion!,

DS