topic Re: Is it possible to make DOE generation reproducible? in Discussions

Is it possible to make DOE generation reproducible?

frankderuyck — Fri, 03 Nov 2023 12:04:43 GMT

When setting up a Custom DOE in a training often the trainees get slightly different Experiments fequently resulting in different models after analysis. For non experienced users this is confusing. I would like that everyone in the class gets the same Experiment after specification of the targe model, is this possible?

Analysis results can made reproducible when using a Random reset function in the response outcome formula.

Re: Is it possible to make DOE generation reproducible?

Victor_G — Fri, 03 Nov 2023 12:16:28 GMT

Hi @frankderuyck,

There is an option in the Custom design platform that may help you getting reproducible design for your class.

Click on the red triangle next to "Custom Design" and click on "Set Random Seed". If all students have the same value for the seed, they should obtain the same design :

Hope that this option will help you, happy teaching :)

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Fri, 03 Nov 2023 12:31:50 GMT

Works fine, will be a great help to avoid discussions & time loss.. again, thanks a lot Victor!

Re: Is it possible to make DOE generation reproducible?

Victor_G — Fri, 03 Nov 2023 13:12:47 GMT

You're welcome @frankderuyck !

I do understand the need for students to have the same design when you are using already collected measurements or publicly available data to create a design and analyze the results.

I don't know which level/class your students are, but since Custom Design is based on a Coordinate-Exchange algorithm to create an optimal design for the assumed model (compared to Classical designs relying on "fixed" textbook designs), it can also be interesting to show them how optimal designs are created, and that for a specific situation, several optimal designs may be created. Even if the experiments may not be the same between several optimal designs, their comparative evaluation can show how similar they are.

I attached a JSL script I downloaded once on this Community (don't remember where) about this exchange algorithm, very useful for teaching.

Select all the JSL code, click on "Run". A dialog box will open with the number of runs, time between each step (in sec) and optimality criterion to specify, and once these info included, the script will show you in 2D (X1 and X2) how the algorithm works and moves random points in the experimental space to improve the design based on optimality criterion specified.

I found this script very excellent to convoy a good understanding about how this coordinate-exchange algorithm works.

Happy teaching and all the best,

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Fri, 03 Nov 2023 14:34:28 GMT

Hi Victor, this indeed can be quite interesting for an advanced DOE course with more seasoned participants. So far my training is at basic/intermediate level so they need first to learn well to drive the car and then in a next step something about the engine.

Thanks for input! Frank

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Wed, 08 Nov 2023 13:43:26 GMT

Where can one find the Random Seed code for a given DOE so it can be reproduced? Attached I added a DOE, what is the Random Seed code?

Re: Is it possible to make DOE generation reproducible?

Victor_G — Wed, 08 Nov 2023 14:01:12 GMT

Hi @frankderuyck,

In your datatable, right-click on the script "DOE Dialog" and then "Edit" to read the script :

DOE(
	Custom Design,
	{Add Response( Maximize, "Selectivity", ., ., ., ., ., "%" ),
	Add Factor( Continuous, 200, 250, "Temp", 0, "°C" ),
	Add Factor( Continuous, 9, 25, "O2", 0, "vol%" ),
	Add Factor( Continuous, 0.1, 0.5, "CAT", 0, "m%" ),
	Set Random Seed( 2057741289 ), Number of Starts( 29219 ), Add Term( {1, 0} ),
	Add Term( {1, 1} ), Add Term( {2, 1} ), Add Term( {3, 1} ), Add Term( {1, 2} ),
	Add Term( {1, 1}, {2, 1} ), Add Term( {2, 2} ), Add Term( {1, 1}, {3, 1} ),
	Add Term( {2, 1}, {3, 1} ), Add Term( {3, 2} ), Set Sample Size( 16 ),
	Optimality Criterion( "Make I-Optimal Design" ), Simulate Responses( 0 ),
	Save X Matrix( 0 ), Make Design}
)

You'll see in the script that a random seed has been defined by default so that you can regenerate the same design. In your example, the seed is 2057741289.

Hope this answer will help you,

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Wed, 08 Nov 2023 15:16:31 GMT

One more time Thanks Victor!

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Wed, 08 Nov 2023 19:46:14 GMT

Attached a DOE I can't reproduce despite I follow all instructions: when I open the script of this DOE (DOE dialog/Edit) and run it again it gives another Custom DOE?? The new Custom DOE script contains some instructions on aliasing which are not present in the script of the original, attached DOE; what went wrong?

Re: Is it possible to make DOE generation reproducible?

Victor_G — Thu, 09 Nov 2023 08:21:49 GMT

Hi @frankderuyck,

You're generating the same design as the original one, and an easy way to check this is to click on "Set Random Seed" (in the red triangle next to "Custom Design"), and see that the seed you have fixed earlier is kept on the Custom Design platform: "123".

However the order of the runs is set on "Randomize" :

So when you click on "Make Table", you'll get the same 24 experiments but ordered in a random manner. For teaching, it might be easier to create a design and fix the run order to "Keep The Same" or other non-randomized settings. This way, you'll get a design that will have the same runs and same runs order when generating it.

I checked the alias terms you mentioned, they are the same in the original (using the script "Evaluate Design") and the newly created custom design (on the left original design, on the right the new one) :

I hope this answer will help you and reassure you,

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Thu, 09 Nov 2023 08:56:11 GMT

Hi Victor, clicking on Edit here below I get the script of the DOE in attachment

DOE(
Custom Design,
{Add Response( Maximize, "Conversion", ., ., ., ., ., "%" ),
Add Factor( Continuous, 230, 250, "Flowrate", 0, "lmin-1" ),
Add Factor( Continuous, 15, 25, "O2", 0, "vol%" ),
Add Factor( Continuous, 0.3, 0.5, "CAT", 0, "m%" ),
Add Factor( Continuous, 15, 20, "Pressure", 0, "10² kPa" ),
Add Factor( Categorical, {"LINE A", "LINE B", "LINE C"}, "Supply Line", 0 ),
Set Random Seed( 123 ), Number of Starts( 100 ), Add Term( {1, 0} ),
Add Term( {1, 1} ), Add Term( {2, 1} ), Add Term( {3, 1} ), Add Term( {4, 1} ),
Add Term( {5, 1} ), Add Term( {1, 1}, {2, 1} ), Add Term( {1, 1}, {5, 1} ),
Add Term( {2, 1}, {3, 1} ), Add Term( {2, 1}, {4, 1} ),
Add Term( {2, 1}, {5, 1} ), Add Term( {3, 1}, {5, 1} ),
Add Term( {4, 1}, {5, 1} ), Add Term( {1, 2} ), Add Term( {2, 2} ),
Add Term( {3, 2} ), Add Term( {4, 2} ), Set Sample Size( 24 ),
Optimality Criterion( "Make A-Optimal Design" ),
"A-Optimality Parameter Weights"n(
[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
), Simulate Responses( 0 ), Save X Matrix( 0 ), Make Design}
)

When I click on run the run order is different from attached DOE? Hard to understand why the script of this DOE does not reproduce the DOE??

Re: Is it possible to make DOE generation reproducible?

Victor_G — Thu, 09 Nov 2023 10:29:03 GMT

Yes @frankderuyck, as the original DoE seems to have been created with a randomized run order, when you're clicking on "Make Table" after having launched the script "DOE Dialog", you'll end up with a different run order than your original design (or the design presented in the Custom Design platform for creating it), even if you select "Keep the Same" : it will keep the same ordering as the one seen in the Custom Design platform, not the one from the datatable.

One option would be during the design generation (and before clicking on "Make Table") to check the option "Include Run order column", so you'll be able to sort in acending order with this column the runs for any design created with the same settings :

Unfortunately this option doesn't seem to be written somewhere in the script from DOE Dialog, so it doesn't seem possible to add it "a posteriori" to your original datatable to have an information about run order.

Re: Is it possible to make DOE generation reproducible?

frankderuyck — Thu, 09 Nov 2023 11:01:37 GMT

OK thanks Victor,

thought that fixing Random Seed was meant to assure constant run order randomization when selection this option before DOE generation. So guess now I'll have to regenerate my DOE's with Run Order Keep the same.