topic Re: How to treat replicates and failed runs in a definitive screening design in Discussions

How to treat replicates and failed runs in a definitive screening design

ajoneja — Thu, 08 Jun 2023 21:08:06 GMT

Hi community, I am having trouble modeling some data that I got from a DSD. This was a screen of 10 factors. We ran this with 3 blocks (operators) and I included the maximum number of extra runs (8). This gave me a table of 31 runs. We did some initial range-finding experiments to set reasonable bounds for all of the factors. Unfortunately we still got 4 runs where the reaction completely failed, so I have no value to put in as a response.

My first question is how these failed runs should be treated? Ideally I suppose they should be excluded, and I would still have 27 runs that did provide data. But if I exclude them from the table and try to run the Fit Definitive Screening script, nothing happens. There is no error message but the window just doesn't open. My response is "Time to Completion", with the objective of minimizing this response, and I could just put in a large number representing a time longer than the experiment duration but this seems arbitrary. The script will run though.

My second question is whether I can include replicate runs to have more confidence in the model? We ran 8 replicates of each run condition. However, if I copy and paste a run and try to run the script, again nothing happens. Right now I am just taking the average of the 8 runs and using that, but I'm wondering if it would be better to somehow include the individual values and if so, is there a way to do it with the Fit Definitive Screening script?

I can use the standard Fit Model module to solve both of these problems but it seems like I'd be missing the benefit of the DSD. Thanks for any responses! I'm on JMP 15.2. If posting the table would be helpful I'm happy to do it.

Re: How to treat replicates and failed runs in a definitive screening design

P_Bartell — Tue, 15 Feb 2022 13:45:18 GMT

First, I have not looked at your data so I can't replicate the failure modes you describe...but I think you are out of luck. To reap the features of a DSD, you really need to have all the responses for each treatment combination. It's a very economical and information packed design for the two stage analysis as prescribed by the Fit Definitive Screening script. Here's my general suggestion, again I haven't looked at your data. But I'd give this a try at least...you can maybe back your way into the same conclusions you'd reach from a DSD analysis workflow.

You'll need to use the Fit Model platform. Include all the replicates 'as is'. Might as well...there may be useful information in the variation among those points...either statistical or practical. Since a DSD analysis workflow assumes effect sparsity to realize the benefits of a DSD approach...I'd start by fitting a main effects only model and see which effects are active. Hopefully the missing treatment combinations don't hurt too much finding these active main effects. Once you got those active main effects, now fit a model with those active main effects AND all possible two factor interactions including those main effect terms. Fit that model...and see what it tells you.

This two step approach conceptually mimics what the Fit Definitive Screening analysis platform does...sort of. I think this approach might get you close?

I'd be interested to see what others such as @Mark_Bailey or @statman might think?

Re: How to treat replicates and failed runs in a definitive screening design

statman — Tue, 15 Feb 2022 15:28:20 GMT

My thoughts:

1. Response variables: Can you measure something other than "time to completion"? For example, % of completion or chemistry after 'x' amount of time....in other words other ways to measure the phenomena.

2. This is a concern for any DOE strategy with optimality in mind. They are more susceptible to lost/missing /atypical data.

3. How to replace the missing data (some ideas on this, but the more missing data the less useful)?

Use the mean of the data set
Use predicted data (hopefully these predictions were done á priori
Regress on the data that you do have and use the model to predict the missing data (using the fit model platform)
Do all of the above and compare the results....if they somewhat agree, you have greater confidence in the resultant analysis, if not then you have to decide What did you learn and what should be the next iteration.

4. I'm a bit confused by the statement "We ran 8 replicates of each run condition". Replicates or repeats? Were they independent treatments? If they truly are replicates and all 8 replicates did not react (the 4 times), what did you learn?

Re: How to treat replicates and failed runs in a definitive screening design

Mark_Bailey — Tue, 15 Feb 2022 15:36:34 GMT

I want to verify that Fit Definitive Screening does not work if you have missing values or deleted or excluded runs from the definitive screening design. The specialized analysis depends on the fold-over structure of the design matrix when forming the model matrices for analysis. (Note that you can use Fit Definitive Screening with any fold-over design!)

I wonder if it would be possible to replace the missing values for the runs without a reaction with a very large number for Time to Completion? This recode is a numeric trick. You simply want the analysis to proceed where some runs took essentially an infinite time, but represent this with a finite number. For example, if the reaction time ranged from 5 minutes to 2 hours, then you might use a value of 1000 to 100000 instead of missing values. Does that work? I am trying to reclaim as much of your experiment as possible.

Are the extra runs truly replicates? That is, they are not simply repeated measures of a single run. I suspect that Time to Completion requires that they are true replicates - just checking. If so, then I would definitely include the individual replicates and not the average of them. A counter argument, though, is that you could estimate the mean and the standard deviation and model these to find conditions that give you both the minimum mean time and the minimum variation in the time.

My final comment is that Fit Definitive Screening is a wonderful, specialized tool. It is really about model selection, and that is as far as it goes. You have many other tools for selecting the best linear model.

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Tue, 15 Feb 2022 19:08:22 GMT

Hi @Mark_Bailey , @statman , @P_Bartell ,

Thank you for the thoughts! I have attached my data table so you can replicate what I am seeing, if that is helpful. Right now, the cells without a value in the "Time" column are the failed runs. Without a value, the Fit Definitive Screen script doesn't run. If I exclude those rows/runs, the script still doesn't run. If I assign random values, it will run. I know it is best not to exclude any runs and this will harm the model, but since I added 8 extra runs when generating the table, I was hoping that losing 4 to failures might still work. I was just surprised that the script won't even try! I have started by trying the approach that @P_Bartell suggested and I have questions there too but that should probably be another thread and I want to spend some more time on it first. To answer some of the other questions that came up:

- Yes, I can try to use a different metric than "Time to Completion". The only one I could think of is "Time left in experiment", and try to maximize that value (it would be zero for the failed runs). This is a biochemical reaction that produces a signal that is supposed to increase with time until it reaches a maximum. In the failed runs, no signal was ever produced, so I don't have any other value to use.

- I have tried assigning values to the failed runs, such as numbers larger than the duration of the experiment, or just very large numbers (approximating a Time of infiniti because it never worked), these seem to really skew the model and make all the other data points less impactful.

- To clarify, my replicates truly are repeats of the run conditions, not just multiple measurements of the same run. So in total we ran 31*8 = 248 reactions. These reactions naturally have some variability so replicates are important. The values currently in the table are the mean of the 8 runs. I thought there would be some value in including the data from each run in the model, but if I try to add extra runs to the table, the Fit Definitive Screening script doesn't run.

- To address "What did I learn from the failures": If I look at the failed runs, 3 out of 4 have a combination of Low Mg and High Log2X1. This was expected to be detrimental to the reaction, and I had considered setting up a constraint so that this combination could not be used, but the DSD didn't allow it. If I had done a custom DOE I would have made this constraint. (although, there are cases where Low Mg and High Log2X1 did work, so this is not a guaranteed failure).

For now I will try modeling this data by excluding the failed runs, including the individual replicate data, and not using the Fit Definitive Screening script.

Re: How to treat replicates and failed runs in a definitive screening design

statman — Tue, 15 Feb 2022 20:57:29 GMT

Quick question: How much of a change in Time is of scientific significance (or practical significance)?

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Tue, 15 Feb 2022 21:46:22 GMT

Anything greater than 5 min would be of interest. For our "midpoint" assay for the 3 Blocks/Operators, it was completed in 16, 25, and 25 min (which is in itself a bit concerning that one operator was somehow so much faster).

Many of these conditions are much worse than that but we were hoping to identify factors that could help us improve on that timing.

Re: How to treat replicates and failed runs in a definitive screening design

statman — Tue, 15 Feb 2022 22:32:52 GMT

I ran a few analysis (fit models) by block and including block as a fixed effect. Attached data sheet. The analysis are using saturated models and have not been reduced. Yes, block effects can be quite challenging...leading to lots of questions.

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Wed, 16 Feb 2022 14:59:18 GMT

Thank you @statman ! I am fairly new to DOE modeling so trying to find my own way to the models that you scripted, but focusing on using the whole data set with block as fixed effect, and it seems like including the replicate data helps. The differences between the individual block models are certainly concerning. Thanks to everyone for the help.

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Fri, 25 Feb 2022 14:44:08 GMT

Hi @statman , the model you provided including all data and the block as a effect makes sense from what we know about the system. I have been trying to recreate your model but having difficulty, in particular I am not sure how you settled on looking at the secondary interactions and quadratic effects for a subset of the factors (pH, Mg, Log2X2, Log25X3), which are providing the nice curvature that we expected.

I have tried the strategy that @P_Bartell mentioned above - screening main effects, then crossing those with all possible combinations of factors. I have also tried using Stepwise Linear Regression in a few different ways. But nothing has lead me to anything as predictive as what you provided, and I'd just like to know how you got there. Any insight into your thought process would be appreciated! Or if there is a particular help page I should read up on, please let me know.

Re: How to treat replicates and failed runs in a definitive screening design

statman — Fri, 25 Feb 2022 18:31:39 GMT

My thought process when analyzing DOE data is to start with a saturated model. This is called the subtractive approach to model building (vs. the additive approach like stepwise does). A saturated model is when all degrees of freedom are assigned. In your case, I treated the block as a fixed effect and include block-by-factor interactions. When analyzing the results first think what practical effect does each factor (model term) have and then look to statistical significance. Remove the insignificant items, then re-run the reduced model and analyze the residuals. I don't know what practical significance is for your data set, so I could not reduce the model.

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Fri, 25 Feb 2022 23:27:31 GMT

Thanks statman, I think I understand the approach.

To generate my saturated model, I have been starting with all factors, all factors squared, and all interactions (using the Response Surface macro). Then running a Standard Least Squares. Is there an option for treating the block as a fixed effect? Currently the result lists a couple singularities that seem to be preventing me from getting p values for any of the interactions. Getting stuck there. In your model, you have all of the factors and a few interactions but not all of them, does this mean that you have reduced the model somehow already? How did you know which ones to remove?

Re: How to treat replicates and failed runs in a definitive screening design

statman — Sun, 27 Feb 2022 18:35:55 GMT

In the future, I would recommend knowing what model effects you want to separate, which model effects you are willing to confound and which model effects you don't want to include in the study BEFORE selecting the design to run!

You can't use the response surface macro and you must ensure the block is not a random effect. To create the model, you will need to understand the total degrees of freedom available and the aliasing in your experiment. (Never put aliased terms in the model, you will get singularity errors). You might start here:

https://www.jmp.com/support/help/en/16.2/?os=mac&source=application&utm_source=helpmenu&utm_medium=application#page/jmp/effective-model-selection-for-dsds.shtml#ww292566

When you create the design in JMP, you should get an arrow that is a script to Evaluate Design, this will provide the aliasing structure. If you have run a complete block, your model will contain the definitive screening design model, block and all block-by-definitive screening model effects.

Re: How to treat replicates and failed runs in a definitive screening design

ajoneja — Mon, 28 Feb 2022 20:48:56 GMT

The more I learn, the more I realize how much I don't know. Thank you for the continued guidance!

Re: How to treat replicates and failed runs in a definitive screening design

statman — Mon, 28 Feb 2022 21:14:25 GMT

My pleasure. Experimental design is a huge topic with many "facets". What further complicates the subject, is there is no one "right way". Some will be more efficient, some will be more effective. I have found that I have to first understand the situation, what questions I'm trying to answer, what hypotheses I'm trying to get insight into, what model effects do I need to estimate. I design multiple options/plans, think through what knowledge could be potentially gained (I actually predict every possible outcome) for each option and weigh that against the resource requirements. Recognizing that I will be iterating, so what do I need to continue the investigation.

I have been extremely fortunate to learn from some of the best minds (Box, Taguchi, McLean, Sanders, Diamond, Bisgaard, et.al.) and had the luxury of being involved in designing and analyzing 1000's of experiments. Many of witch in retrospective could have been designed better...LOL

Re: How to treat replicates and failed runs in a definitive screening design

SaraA — Sat, 09 Mar 2024 18:12:39 GMT

Hi @Mark_Bailey

Following up on that, how do you input the individual replicates in the DSD? When designing the table of the DOE, there is no box to indicate your number of replicates (while this box is present when designing a classical screening design for example).

Thank you

Sara

Re: How to treat replicates and failed runs in a definitive screening design

Mark_Bailey — Mon, 11 Mar 2024 20:12:27 GMT

You do not have the same option with a DSD. You cannot replicate some runs if you want to use Fit Definitive Screening. You can copy all the runs after you click Make Table, and paste a copy below the original rows to replicate the design.

Re: How to treat replicates and failed runs in a definitive screening design

Victor_G — Tue, 12 Mar 2024 07:51:48 GMT

Hi @SaraA,

As @Mark_Bailey has mentioned, you can replicate manually the full design by copy-pasting all the rows belonging to the DSD.
You can also use the platform Augment Design to replicate your DSD : Replicate a Design