Svem analysis DOE with replicated runs

[#1]

In last DOE discussion forum Svem analysis was discussed for a DOE with relicated runs. Is was suggested to avarage the result of replicated runs. What if is there is significant variation between the replicates? Would it not be better to run Svem with extra "replicate run variable" to estimate the within replicates variation? If within replicte variation is important Svem will pick this up; in a next mixed model analysis the replicate variable can be assigned as random be separated from fixed effects that Svem assigned to be important. 

[#2]

I'm sorry I didn't attend the DOE club meeting and am not familiar with SVEM analysis (I did download the file and will review).  Analysis of experiments is a personal choice. There are many ways to do this.  Before I comment, I want to make sure you mean replicate and not repeat.  That is replicates are considered independent runs of the identical treatment combination, each run having an independent experimental unit as an output.

Here are my uninformed comments:

1. When you summarize data you invariably throw out information.  If you are going to average, you should also consider the variation of the data you averaged as a response.  IMHO, you throw out information (and DF's) if you average across the replicates.  Now if you are doing repeats, that is a great idea and also consider the response of variation to model.

2. If these are indeed replicates, it seems that you want to understand what changes between replicates.  If you can identify and assign what factors (noise) change between replicates, you can learn a great deal about the robustness of factor effects.  This can be an excellent strategy to simultaneously increase inference space and increase the precision of the design.

"Block what you can, randomize what you cannot". G.E.P. Box

Confound what you can identify and assign with the block.  For factors you have failed to identify, randomization is the strategy.

[#3 · Accepted solution]

Hi @frankderuyck,

SVEM analysis works great in two situations :

1. To build predictive model with very limited data, by stacking the different individual models that have been trained and validated on slightly different data (different weights for training and validation).
2. To assess practical significance and increase the precision of effects estimates for unique individual experiments (no replicates) in a Machine Learning way, thanks to models stacking.

No matter the situation, it's important to remember in which context SVEM technique has been introduced : to provide a validation technique/framework in the presence of small dataset without any other possible validation technique.

Talks and presentations : Re-Thinking the Design and Analysis of Experiments? (2021-EU-30MP-776) - JMP User Community

SVEM: A Paradigm Shift in Design and Analysis of Experiments (2021-EU-45MP-779) - JMP User Community

So just conceptually, I have mixed feelings about the idea of combining SVEM and replicates, as this looks like the merging of two different model building methodologies that rely on different concepts, objectives and validation methodologies : screening based on statistical significance and error/noise estimation for designs involving replicates and statistical modeling vs. predictive modeling based on predictive accuracy and bootstrapping/models stacking.

In presence of replicates, I fear that data leakage may occur when using SVEM for model building : as each experiment will have randomly (and in an anticorrelated way) training and validation weights, replicate runs could have different emphasis on training and validation, even if they represent the same treatment ! So the model building may be slightly biased (as it has information on the training and validation sets through replicates with different weights), and may result in "overconfidence" and a big decrease of standard deviation for terms estimates :

When increasing the variability between replicates on the same example (Reactor 20 Custom JMP dataset with added replicate runs up to 30 runs vs. original Reactor 20 Custom dataset), you can see that the two situations of using SVEM on design with replicates and using SVEM without replicates are fairly similar in terms of estimates and standard errors : 

So again, I don't see what could be the benefit of merging these two techniques, as you're merging two (very) different validation techniques, which is not why SVEM has been created for.

I'm also skeptical about using SVEM to estimate within replicate variation, as you can already have this information if you have replicates with a standard analysis, and using SVEM might decrease the estimation of standard deviation, like bootstrapping : see the great interactive visualization for bootstrapping : Seeing Theory - Frequentist Inference

For other interested in this discussion, here are the recordings of DoE Club Q4 : Recordings DOE Club Q4 2024

Hope the response make sense and interested in the inputs from other members,   

[#4]

Hi Victor, when variation between replicates is low one can expect not much influence when increasing #replicates. But what if there is qute high variability between the replicates, not interesting to introduce a blocking factor for each replication and include this blocking effect in Svem?

[#5]

Hi Frank,

I would still think this may not be a good idea, as SVEM method seems to be "design-agnostic" so it won't take into consideration any grouping, blocking structure or presence of replicates, possibly resulting in "overconfidence"/data leakage/decrease of standard deviation on the blocking effect term estimate.
Also as it is a specific type of bootstrapping, it may result in a decrease of the standard deviation estimate, as shown before with the interactive visualization.

EDIT: More info on SVEM :

Overview of Self-Validated Ensemble Models

Lemkus, T., Gotwalt, C., Ramsey, P., and Weese, M. (2021). “Self-Validated Ensemble Models for Design of Experiments.” Chemometrics and Intelligent Laboratory Systems 219:104439.