Custom Design Around Existing Data With Blocking Factor

[#1]

Hello,

I'm trying to optimize process parameters for a 3D printing task. I have 6 different "plants" that I'd like to treat as a blocking factor. In order to get an estimate of process variance, I collected 18 center-point samples for my factors, 3 samples per plant. I'd like to design an I-optimal experiment and I'd like to use these center-point samples in that study to reduce the number of new samples needed. Here's what I've tried:

1. Set all factor columns coding and design role. I have 4 numeric continuous factor, one numeric discrete factor, and one blocking factor.
2. Augment design
   1. I don't select the blocking factor, as I can't edit the "number of runs per block" in augment design 
   2. Set I-optimality criteria
   3. Add appropriate levels to factors (augment design treats my discrete numeric factor as continuous, but this seems ok because the levels are just 0, 1, 2 which works as a continuous variable with levels 0 and 2).
   4. Add 2FI and 2nd power effects to model where applicable
   5. Set number of runs to 78 (targeting 90 runs total with 12 repeats)
   6. Generate table
3. Custom design, here's where I'm getting stuck
   1. Add blocking factor to table
   2. Make sure blocking factor column design role is "blocking"
   3. Add 72 runs from table in last step as covariate factors including blocking factor
   4. Add same 2FI and 2nd power effects to model from last step
   5. Select "Include all selected covariate rows in the design" and "Allow covariate rows to be repeated"

A few problems I've ran into:

• It doesn't look like I can specify "12 repeats" when I am creating a custom design with covariate factors, I can only tell JMP the number of runs
• If I select the blocking factor as a covariate factor it isn't treated like a blocking factor, just a covariate factor. If I don't select the blocking factor when adding covariate factors and add it as a blocking factor in my custom design, the custom design has no way of knowing what blocks each of my 18 data points belongs to.
• All of the factors that I add as covariates appear in my factors table as "covariate" role. I'm not sure if this is a problem, but it means I can't distinguish blocking, numeric continuous, and numeric discrete
• I get the error "The selected terms in the model outline are linearly dependent on previous terms" whenever I have any effects other than my blocking factor and the interecept. When I tried just making a custom design with 90 total points, 18 center-points, and 12 repeats with all the same modeled effects that worked fine so I'm not sure why I'm getting this error here.

I feel like this workflow (get center-point data to estimate variance, then build a DOE around your already-collected center-point data) is pretty common so there must be a good way to do it. Any guidance would be greatly appreciated!

[#2 · Accepted solution]

Hi @TimCarrWPI,

Welcome in the Community !

From what I understand, you're able to augment your design, but you are facing problems to enforce the blocking factor correctly in the augmented design ? Concerning your factors, it seems the ironing pass factor is wrongly defined as continuous and not discrete numeric (you also have to change the modeling type to Ordinal).

What may be possible is to :

1. Augment your initial 18 center points into a RSM I-Optimal design, and specifying a total number of runs of 90. This option may however not provide exactly 12 replicate runs, so instead, you can use the Custom Design with the factors you have defined to create the design around the center points, and enforce the 12 replicate runs condition:
   
2. Once your design is created, you can concatenate your initial 18-runs dataset to this custom design to get to 90 runs.
3.  You finally have to sort the runs by the block number (Mandrel factor), and randomize the runs within each block, and you'll end up with the design you expected.

I recreated and added the script in the final design table to evaluate the design and launch the fit model platform. Please find attached the factors table I used to create the "Custom Design part", the Custom design, and the final design with your 18-runs dataset.

You might get interested in this closely related topic: Forcing center points to be equally distributed across blocks custom design 

I hope I have understood your situation and that this response will help you,

[#3]

Thank you for your help @Victor_G ! I believe you've presented two potential solutions, but I have a few questions/concerns about them:

Augment the design around my 18 center-points at 90 total samples. Here's what I am trying for this method: 

1. Start with "Augmented Design Input.jmp"
2. Select "Augment Design" an add all columns besides the response (adhesion strength) to X with adhesion strength in Y
3. Add levels to each non-blocking factor so they're not treated as constant

Here are the issue I'm having:

• I can't change "runs per block", so when I try to expand to 90 runs it adds a ton of levels to the blocking factor (mandrel)
• If I can't specify the number of repeats, does JMP intelligently choose repeats to ensure I have a good estimation of the error? Obviously I need some repeats to perform a lack of fit test.

Use a custom design at 72 runs and then concatenate my center-points. Here's my concerns with this method:

• JMP's custom design tool doesn't "know" that I've already collected 18 center-points, so I worry that the optimizer will choose a sub-optimal distribution of points as it tries to fill in near the center of the parameter space not knowing that I already have a bunch of points there.
• If I could force the custom design solver to put 3 center-points per block when it generates the design, then I could just fill in those points with my already-collected data, but based on the discussion you linked to it doesn't seem like that's possible.

TLDR:

• I need exactly 6 blocks and Augment design doesn't let me specify the runs per block
• If custom design doesn't "see" my previously-collected data then it won't generate an optimal design (I think)

[#4 · Accepted solution]

Hi @TimCarrWPI,

Yes, the direct design augmentation path may be difficult to handle regarding the blocking factor. One way to solve this is to augment your initial design without the blocking factor up to 90 runs, and then use the Custom Design platform to include all the runs of the augmented design as covariates, add a blocking factor with 15 runs per block : 

As you mention, during the augmentation you won't have full control over the number of replicate runs, but JMP will allocate replicate runs to reduce the prediction variance where it is the highest.

Regarding your concerns about the optimality of the design with the second option:

• Center points are primarily used for two reasons: estimate pure error for the lack-of-fit test and decrease variance prediction in the centre of the experimental space (see effect of centre points for more details). They are not helpful for model terms estimation, so I wouldn't worry much about any optimality loss of not considering them during design creation.
• By default, when you specify a RSM model, JMP is not proposing any center points, but instead some points where one factor is at the middle level, and the other factors levels are at min and max values. So even if you build the Custom design independantly before concatenating your center points dataset, you won't create new center points. One way to check this is to look at my final design file shared previously, you can select one row of your center points for the 5 factors columns and use Rows > Row Selection > Select Matching Cells. The center points highlighted in the table only comes from your initial dataset. 
  

I'm not aware of a way to enforce center points "homogeneously" across blocks, so that's why I linked the previous discussion relating the same issue.

Hope this answer will help you,

[#5]

Your explanation makes sense, thank you! I'll proceed using the custom design tool to create 72 runs and just concatenate the center-points in front of that data during analysis.

Cheers!

[#6]

I have a few comments that may help you think through your situation. Note, I am not trying to persuade you in what to do, just to be more thoughtful prior to executing an experiment.

1. Blocking is a strategy to handle sets of noise variables. You might say you are confounding "plants" with the block. Why do you need 6 Plants? You most certainly would not entertain a 5th order polynomial including plants as a factor in that model. A better question is why would Plants affect your response? What factors are confounded with Plants (what factors are different or at set at different levels)? Be fastidious in identifying the factors that are different. Do you have hypotheses (explanations) about why those factors that vary from plant-to-pant would have an effect? What will you do if block has a large effect? What if it has a small effect? You might consider doing this study more iteratively. Could you identify Plants that are the most different? Perhaps start with those (2-level) and estimate the size of the block effect. If its is large then replicate over other plants. This could also allow for estimate the consistency of factor effects across blocks.

Blocks are most certainly not a covariate which is a specific factor that is measurable on a continuous scale.

Another strategy would be to do a COV sampling plan where Plants is one of the components (another discussion). 

2. You suggest you are optimizing factors, and that will likely be an end goal, but how far are you from that "space"? You have 5 factors and a block (which is a chunk of X's). I think you are still doing some screening perhaps.

3. Center points can be a very useful strategy (the factors should be continuous to do this, but you may be able to cheat on this). You suggest you are using those to get an estimate of the process variance. You mean you don't know this? What would 3 data points per plant give you a reasonable estimate? Perhaps you should also get an estimate of stability before estimating variance and likely the 3 CPs per plant might give a clue, but it also might be better to run more center points within each block, reduce the number of blocks and get an estimate of stability in the design space (perhaps 8 per block for a total of 16). Also make sure the center points are run randomly throughout your experiment. This increases the confidence in treating those as independent events. When you analyze the data, first sort the center points in actual run order and get an individual moving range chart. The MR chart will provide data to estimate the variance and the individual chart will estimate shifts and trends in the design space. The next use of the center points is to assess the linear assumption within the design space.

4. Lastly, no one on this forum or any forum knows the best experiment for you to run. My advice is to create multiple options, consider what knowledge you could potentially gain from each option. Predict all possible outcomes from the options. How will they help you iterate (gain knowledge). Weigh this against resource requirements. Pick 1, run it and be prepared to iterate.

[#7]

Thank you for your thoughts @statman. I'll answer in order:

1. I am 3D printing on rods (called mandrels); each "plant" is a different rod. The rods are in theory all the exact same, but they have small deviations in surface geometry that present as systematic error between rods. Due to the mechanism of effect (bumps and curves basically), the mandrel effect is at least in part confounded with my "z height" effect but likely with no other main effects. I am including this as a blocking factor because I want to see the effects of each of my parameters independent of which mandrel they're printed on. I expect that trends will be similar across mandrels. I believe that this strategy is sound, but please correct me if I'm wrong!
2. I've done a manual search for near-optimal parameter combinations which is defining my search space for this study. I'm kind of trying to both do screening and optimization in one study: I want to identify the most significant effects so that the information I gather is useful to other people (I suspect that the specific optimal point will vary setup to setup), but I also want to find the optimal setpoints for my specific process on my specific setup.
3. Excuse my vague language in my initial post, I was trying to just explain my situation in stats language instead of explaining more of my experiment. I was collecting data at the center points (1 for my discrete numeric parameter with values 0, 1, 2) to get an estimate of the random error between each print that's not attributable to differences between mandrels. I think this is what you're calling stability and I was calling process variance. I'm not totally clear on the difference here. To do this, I split error like you would for ANOVA where SST = SSM + SSE and used that to get RMSE. I want to use 6 blocks because I'd like to use all 6 mandrels (plants) when I'm generating samples to save time. Doing 18 samples on one plant takes only a little less time than doing 108 samples across 6 plants. I'm not too worried about randomly dispersing my samples across time, as the 3D printer is extremely repeatable print to print, though I obviously will randomly disperse my samples within blocks to reduce any print to print error creating effects where there aren't any.
4. I wasn't initially really looking for advice on which experiment to run, just on how to get JMP to generate an RMS I-optimal design around fixed points with a blocking factor, but I appreciate the advice!

[#8]

Really no need to respond to my post. I obviously did not have enough information to provide specific advice. You can also ignore my follow-up comments.

1. I am confused by this: "Due to the mechanism of effect (bumps and curves basically), the mandrel effect is at least in part confounded with my "z height" effect" I believe you mean the variation in the mandrel effects the Z height response because small deviations in surface geometry that present as systematic error between rods. This is an excellent hypothesis. This is where you need data to support or reject.  Of course the mandrels vary, as do raw materials. There are different interpretation for the use of blocking in experimental design. The most typical use of blocking is to partition and "remove" (by assigning) the effect of noise variables from the design factors. This increases the precision of the experiment. The second use is to treat the block as a fixed effect and then, in addition to partitioning and assigning the noise, you can also ascertain block-by-factor interactions. "I am including this as a blocking factor because I want to see the effects of each of my parameters independent of which mandrel they're printed on." I think you might also want to know if the effects of your factors are consistent across mandrels. This is Block-by-factor interaction effect. If, for example, there is a factor you find significant for one mandrel but it is not for another, or if the setting for a significant factor changes due to mandrel, you want to know this.

2. Your strategy makes sense as long as the processing equipment, raw materials, ambient conditions such as humidity, nozzle wear, nozzle temperature, contamination, setup, cooling conditions, measurements systems stay constant.

3. Unfortunately, terminology is a challenge. I understand why you want to do center points - to get an estimate of the MSE for statistical tests. Always remember practical significance is more important than statistical significance. I did not make myself clear. My question was whether three center points would be sufficient to get the estimate you are looking for. What if one of those center points was "unusual"? this would inflate the MSE. This might lead to beta errors. What is they didn't vary much? This would under value the MSE and potentially lead to alpha error. This is why checking for stability can help prevent those errors and 3 points isn't enough to do that. I guess my question was, if you are optimizing, don't you already have an estimate of the random errors in the process? Directed sampling is a powerful tool to use in your 3D printing process.

4. I understood this and Victor gives excellent advice on how to handle this. My intent was not to provide the same advice. I apologize if you mistook my post as criticism of your approach, I was simply offering an alternative.