Verifying Random Run Order and Design Validity in JMP DOE

[#1]

Hi everyone,

I’m currently working with a Design of Experiments (DOE) created in JMP, and I’m interested in evaluating the quality of the randomized run order. Although JMP automatically randomizes the run sequence during design generation, I would like to objectively assess how effective that randomization is.

In particular, I’d like to know whether JMP provides built‑in functionality or established workflows to:

• detect trends, systematic patterns, or drift across the run sequence,
• identify clustering or non‑uniform distribution of factor levels,
• perform formal statistical tests of randomness (e.g., runs tests, autocorrelation or independence tests),
• or otherwise quantify how closely the run order approximates true randomization.

If JMP does not offer a direct method for this type of assessment, are there recommended JSL scripts, add‑ins, or best‑practice approaches for evaluating randomness in DOE execution order?

Additionally, I am interested in whether JMP provides tools to verify that the integrity and statistical properties of the experimental design are preserved after manually reordering the runs.

Thank you in advance for your insights.

BayesKnight

[#2]

Hi @BayesKnight : I've been following this with some interest...and, if I'm honest, some confusion. It seems to me that what you are asking is; is the "random" run order (generated by JMP) truly random? As @Dan_Obermiller said, "As long as a reputable random number generator is used (which JMP does have reputable random number generators), any random pattern is typically appropriate." Of course, even "reputable" random number generators are largely pseudorandom number generators. And, as such, they typically use an algorithm to generate a sequence of random numbers that have the properties of a random sequence. So, the answer is "yes".

One simple way to randomize the run order would be to generate a Uniform(0,1) number for each run. Then reorder the runs by that U(0,1).

My question to you though is this; what prompted your question?

[#3]

Hi @BayesKnight ,

  JMP will generate a robust randomized run order for your DOE, you don't have to worry (or even shouldn't worry) about that. JMP has been an industry leader in this field for more than 35 years. They certainly wouldn't still be around if their software was not reliable.

  Even if you did create some way to quantify and evaluate the run order BEFORE you did the DOE, all you'd really end up doing is testing JMP's randomization algorithm to determine if it's "truly" random. As @MRB3855 mentioned, no (digital) random number generator is truly random. Take your favorite music app and test out their "random" play function. In reality, their "random" play is restricted because they prioritize playing each song in the playlist once before "re-randomizing" the list. That's not a truly random order.

   I also agree with @MRB3855 , what has prompted you to bring up this question of evaluating the randomness BEFORE even running the DOE in the first place? Why do you think that JMP doesn't provide a robust random run order? 

If you have any historical data that suggests this is the case, then I'd first spend more time on planning the DOE and use blocking or some other form to account for it. I would not double-guess JMP and make the assumption something is wrong on their end -- it's much more likely the DOE wasn't planned out right in the first place.

DS

[#4]

Hi @BayesKnight. I agree with @MRB3855 that this is quite an interesting thread to follow. There are a ton of good well thought out answers here that offer different perspectives. It does seem that in response to every answer you are still not satisfied and as  pointed out it seems that you just want to know if there is a way to evaluate the "quality of the randomness." I do believe that random is random is random, and sometimes to the naked eye if we see patterns we might not think something is random even when it is.

I am not aware of any accepted statistical technique that exists to evaluate the quality of randomness aside from just looking at the order of your factor levels and seeing if they are "jumbled up enough." Maybe the answer is to do just that and to keep randomizing until you get an order that looks most jumbled? One could argue (I probably would as a statistician) that this is biasing the run order and potentially removing randomness.

One silly solution could be to plot your factors on the y-axis, your run order on the x-axis and use a heatmap with factor level (-1, 0, +1) as color. I guess this could help you visually assess if there are any patterns.