topic Replicating DOE analysis from a paper in Discussions

Replicating DOE analysis from a paper

PolygonBison420 — Thu, 08 Jun 2023 16:33:05 GMT

Hey guys,

very very new to the JMP/DOE/stat world and I have some questions.

I'd like to replicate ANOVA and RSM analysis I've found in a journal.

What I know:
CCD design
20 treatments including 5 center points
3 factors and 3 responses

Table of factors and their responses is attached.

What I learned from the table:

- rotatable CCD (value of 1.682)
- it's actually 6 center points, not 5 (am I correct with this one? Value combination of 60/1/3.5 appears 6 times)

I am not sure how to incorporate axial values to the analysis. Do I just go Fit Model and then add factors via Macros/Response Surface?

Thanks!

Re: Replicating DOE analysis from a paper

Victor_G — Sun, 26 Feb 2023 15:06:51 GMT

Hi @PolygonBison420,

Welcome in the Community !

In order to create a similar design, you can go in "DoE" menu, then "Classical" and "Response Surface Design".

You can then specify the 3 continuous factor and 3 responses.

Then you have the choice between different designs :

The CCD with Uniform Precision does contain 6 centre points, like in your example.

Then, select the axial value corresponding to rotatable design (1,682 for 3 continuous factors) :

And you'll end up with a similar design to the one you have shown. Here is the script to generate the design :

DOE(
	Response Surface Design,
	{Add Response( Maximize, "Capacity factor (k)", ., ., . ),
	Add Response( Minimize, "Retention time (Rt2)", ., ., . ),
	Add Response( Maximize, "Resolution (RS12)", ., ., . ),
	Change Factor Settings( 1, 55, 65, "A: Organic Solvent" ),
	Change Factor Settings( 2, 0.8, 1.2, "B: Flow rate" ),
	Add Factor( Continuous, 3, 4, "C: pH", 0 ), Set Random Seed( 31406552 ),
	Make Design( 2 ), Set Axial Choice( 1 ), Center Points( 6 ),
	Simulate Responses( 0 ), Save X Matrix( 0 ), Set Run Order( Randomize ),
	Make Table}
)

For more infos about Response Surface Designs, you can look here : Build a Response Surface Design (jmp.com)

In order to analyze each model linked to the responses, you can have a look here : Standard Least Squares Models (jmp.com)

I attach the design datatable with a script "Fit Least Squares" that provides the analysis of the responses (please take note that I am not sure that the target for each response is correct, i.e Capacity factor should be maximized, Retention time should be minimized, and Resolution should be maximized ?).

You can try it yourself by launching the script "Model", selecting the responses (model is already defined, but if you have to do it manually you can select the factors and use the macro "Response Surface") and check "Fit Separately". You can then compare the models in the publication with yours.

Hope this answer will help you,

Re: Replicating DOE analysis from a paper

Phil_Kay — Mon, 27 Feb 2023 09:32:23 GMT

Hi,

In addition to what @Victor_G has shown you about designing the experiment, I think you are interested in how to repeat the analysis in JMP.

The starting point is obviously to have the data in JMP. If this comes from a pdf, you should be able to import the data using the option in JMP to import from pdf.

Or you can manually input the data into a JMP table.

Or you might have the data as .csv or .xlsx, which can also be imported directly into JMP.

I suggest that you share the data as a JMP file first. Then the community will be able to help you to replicate the analysis.

I hope this helps,

Phil

Re: Replicating DOE analysis from a paper

PolygonBison420 — Wed, 01 Mar 2023 08:23:23 GMT

Victor, that works brilliantly. I was just confused since authors talked about 5 center points which didn't make sense when looking at the options suggested by JMP. Center points are always treatment of all variables at the middle level/0, right?
It seems it's also harder to recreate RSMs from the papers since almost no authors list response limits.

Since you've been so kind to recreate the design in JMP I also want to ask you what's the most effective way to do so.

For instance, I recreate the design via DOE/Clasical/RSD. My result is a data table with the variables however what's the best way to add response data from the paper?

Is there a quick way for me to match the pattern order of the runs of the paper (via random seed perhaps)? Or do I just have to manually seek for it ("aha --- treatment is in line four, let's copy the response to the first response column of my table..." ).

Hope the question makes sense.

Phil, thanks for the suggesting that.

Re: Replicating DOE analysis from a paper

Victor_G — Wed, 01 Mar 2023 09:03:20 GMT

Hi @PolygonBison420,

Is the paper accessible online ? Do you have a link (it could help to better know how the centre points were defined, what were their expected goals regarding the different responses, and their conclusions/interpretation/optimum found).

Centre points are points that have all factors at their middle level 0, right.

Maybe the authors meant that they had replicated centre point 5 times (or did 5 replicate runs of the centre point) ?

In my case, I generated a design through JMP, and had to "manually" match the responses values from your screenshot to the correct rows of the design. Not an optimal solution, I know, there might be a possibility to have the same order of rows, but as you mention, that would mean knowing the random seed (from the publication) and having access and use it in JMP during design creation : Output Options (jmp.com) (option in the red triangle of the DoE window creation "Set Random Seed").

As @Phil_Kay suggested, you might also directly import the table from the pdf or .csv/.xslx into JMP (this way you don't have to manually copy-paste some values), but before running the analysis, you may have to add some column properties, so that JMP better understand some properties linked to your factors and responses :

For factors, column properties "Coding" (specifying values for low (-1) and high (+1) levels), "Design Role" (specifying the type of factor in your design, here "Continuous" for the 3 factors) and "Factor Changes" (by default on "Easy", but for specific design like split-plot design, it is changed to hard or very hard in order to constrain the randomization of runs),
For Responses, column property "Response Limits" is necessary so that the optimization goal is set (maximize, minimize, match target or None), and the values and associated desirabilities are known for the Prediction Profiler.

I hope this answer will help you,

Re: Replicating DOE analysis from a paper

PolygonBison420 — Wed, 01 Mar 2023 09:50:26 GMT

Victor,

thanks again.

The paper (open access):
https://aip.scitation.org/doi/pdf/10.1063/1.5112268

I did look more carefully in the center points but all I can find is the following quotes:
"Twenty experiments along with 5 center points were studies with three factors"
"A three-factorial, CCD was taken with fifteen experimental runs and 5 center points"

Your process of matching responses was similar to mine then.

Re: Replicating DOE analysis from a paper

Victor_G — Thu, 02 Mar 2023 08:16:37 GMT

Thanks a lot for the link.

I'm checking the analysis results from the script "Fit Least Squares" I added in the datatable, you'll have very similar results (if not identical) to what is described and analyzed in the publication.

Concerning the optimization goals (maximize/minimize), it seems my initial guess was right, as when I am trying to optimize all responses with the Prediction Profiler from the "Fit Group", I have an optimum at 56,09 for factor A, 0,8 for factor B (flow rate) and 4 for factor C (pH) :

Compared to 56,1 for A, 0,8 for B and 3,7 for C in the publication :

However, I am surprised by the results of the predictions of the validation point from the publication, as Retention Time and Resolution have different prediction results when predicting in JMP, compared to the values in the publication, despite having similar models and similar prediction equations.

Here are my predicted values for the optimum found in the publication and with the same terms in the responses models :

I don't understand how predicted values can be so different (and not in the confidence intervals) between the publication and JMP (more noticeable for Resolution and Retention Time) ? I also tried relaunching a model with 3-factors interaction and cubic power terms (even if in the publication they mentioned only quadratic model), but that didn't create predicted values closer to what is obtained in terms of measured values and predicted values in the paper.

As the content related to analysis is quite light, I don't know exactly how they are obtaining these predicted values for the optimum.

If anyone has an idea, feel free to solve this mystery :)

Re: Replicating DOE analysis from a paper

PolygonBison420 — Thu, 02 Mar 2023 20:02:57 GMT

Hey Victor,

thank you for still trying with this one.

I also got the pH value of 4 (compared to 3.8) and I still don't know why those values differ. To be fair, I kind of gave up after that so I didn't even notice the mismatch with the validation and such.

Re: Replicating DOE analysis from a paper

Victor_G — Thu, 02 Mar 2023 20:29:37 GMT

Hi @PolygonBison420,

Since we don't know exactly the desirability targets/profiles for each response (and relative importance/weight of each response), I'm not surprised to not have found exactly the optimum found in the publication (even if the one found with JMP is very close from the one in the publication, which is a good sign).

What troubles me the most are really these predicted values for responses for the validation point. As we are able to have very similar (if not exactly the same) models for each response, predicted values for the factors levels set on the optimum from the publication should give us very close values. Since it's not the case, I'm really wondering if/what I may have missed, or if/what the authors have missed.

Re: Replicating DOE analysis from a paper

statman — Thu, 02 Mar 2023 22:01:37 GMT

Just a comment...I have found errors in many analyses of published papers in well respected journals. I don't think there is much rigor challenging analysis to get a paper published. Without knowing how the analysis was performed, subtle differences are not unusual and in this case might be rounding protocol.

Re: Replicating DOE analysis from a paper

Victor_G — Fri, 03 Mar 2023 08:57:52 GMT

Hi @statman,

Interesting, thanks for sharing your comment. I wouldn't have been so "perplexed" if the predicted values between JMP models and the publication were close, but here, for at least 2 out of 3 responses, the differences are quite big.
I just retried this morning to change the formula from the models and "round up" the parameters from the equation based on the equation models from the publication (just to be sure I didn't miss something), but I'm not getting the same results as they have in the publication...
Sure, they have nice predicted values for their responses (almost too good to be true...), but I really don't know how they could get these based on the modeling and equations they have.

The datatable is attached if you want to take a look (I added three formula columns for the equations from the publication, and the validation point to compare actual vs. predicted values from the models).

Re: Replicating DOE analysis from a paper

statman — Fri, 03 Mar 2023 17:18:53 GMT

Victor, I have the following thoughts/comments/questions (Note: I did not thoroughly read the paper):

1. The first issue I have with the analysis is I don't know what practical significance is for the 3 response variables? How much of a change in Capacity, Retention or Resolution is of scientific or engineering interest? The range for each is 1.2, 6.7 and 10.4 (in order). Is this interesting? Need an SME to help.

2. I don't see run order in the data table?

3. It appears the terms removed from the model and replication of the 6 center points is what is being used for the estimate of the MSE for statistical tests (again, without run order are these replicates or repeats?). This variation (the removed terms, which can bias the MSE) is very small in comparison to the total variation, hence why so many terms in the model look statistically significant (especially Capacity and Retention). There is a mention of within day and between day estimates of precision. I do not see day accounted for? So the question is, is the variation of the removed terms and the replicated counterpoints representative of the true variance (random errors) in the process? If I look at a Normal plot and use Lenth's PSE, it appears the RMSE may be under-estimated (note: JMP labels estimates according to RMSE, not PSE)

4. There're 2 outliers from a multivariate analysis (-,-,+ where all responses are particularly high, and a,0,0 where Retention is high) what happened there?

5. Is there any estimate of measurement error or was the measurement system studied in the paper?

6. I didn't take a detailed look at residuals yet, but residuals or Resolution look unusual.

7. The correlation of the prediction formulas from your analysis and the paper analysis are excellent (.9999, 1.0, .9954).

Re: Replicating DOE analysis from a paper

PolygonBison420 — Mon, 06 Mar 2023 18:13:27 GMT

statman, thank you for giving this analysis a go too. I might be able to help with question 1 however can you clarify a bit what is it you want to know about the three responses? I don't know how to explain "interesting" but I am aware of the function and "good" values for those.

Re: Replicating DOE analysis from a paper

statman — Mon, 06 Mar 2023 19:46:59 GMT

Sure, before any data analysis, it must be determined whether the variation in the response(s) is sufficient enough to be worthwhile to do any statistical analysis. I have no context for the responses in the data table, so I don't know if they varied sufficiently. The question I always ask is what is the smallest increment of change in the response that would be of interest or would you care about? This I call Practical Significance.

Re: Replicating DOE analysis from a paper

PolygonBison420 — Tue, 07 Mar 2023 08:57:30 GMT

statman, let me try this.

capacity factor: here we want (almost a necessity) a value of at least 1. It doesn't matter if it's 2.5 or 3 but it matters a lot if it's 0.5 and 1.0

retention time: the idea behind the range here is just faster is better (with others responses being optimal). A runtime of 4 would be preferable to runtime of 11.

Resolution: here we want (almost a necessity) a value of at least 2. It's basically a measure of spaking between the locations of the two molecules. Values such as 15 are excessive.

In my opinion, response values vary quite a lot, and they are above the threshold of minimum increment of change in the response. If ranges were 10x less (0.12, 0.67, 1.04) then my answer would be no (generally).

Re: Replicating DOE analysis from a paper

PolygonBison420 — Wed, 08 Mar 2023 21:43:26 GMT

Another thing I've noticed here - could it be an issue that data for capacity factor response doesn't seem to be normally distributed?

Re: Replicating DOE analysis from a paper

statman — Thu, 09 Mar 2023 00:00:34 GMT

Not IMHO. Remember, you are manipulating factors and that is the response variable. Essentially you are trying to create variation in that response, hopefully assignable, therefore not randomly distributed. What you want normally distributed is the residuals (NID(0, variance)).