WHAT ARE THE PARTICULAR BENEFITS OF OMARS DESIGNS?

[#1]

There are many powerful JMP DOE tools with proven succes over a long time on the other hand recently a lot is written and published on OMARS Designs.

I would like to ask experienced DOE users what are the particular benefits of these new kind of designs in addition to what is available on the JMP DOE menu? On what kind of particular cases would you advice to use OMARS? Are there cases that can't be treated with OMARS? from what I know I assume mixtures, split plot and random effects are not within the OMARS scope, correct?

[#2 · Accepted solution]

Hi @frankderuyck,

OMARS designs were recently introduced (around 2020) as screening and optimization designs by Peter Goos and Jose Nunez.

The goal behind the use of these designs was to bridge the gap between Definitive Screening Designs and Response Surface Designs. You can find their presentation quite easily where they demonstrate some of the nice properties of OMARS designs :

• Vidéo : http://www.experimental-design.eu/videos/omars-designs/
• Slides https://falltechnicalconference.org/wp-content/uploads/2017/11/webinar-FTC-Peter-Goos-OMARS-designs....  

In their example and demonstration, for an equivalent number of runs than DSDs (22), OMARS may have better projection properties and larger power for quadratic effects, and able to fit all 2-factors interactions. You can reproduce their comparison using the OMARS Designs Add-In  and comparing an OMARS design with a DSD design using the plateform Compare Designs. 
I attached the designs table to make the comparison easier if you want to have a look.

• You can check that OMARS may be able to fit more terms (all interactions and more quadratic effects than DSD), and power for interactions and quadratic effects are equivalent or higher (for the cost of a small power decrease for main effects) :

• They also have lower maximum prediction variance and lower Fraction of Design Space Plot curve (so lower prediction variance over the entire experimental space) :
  
• The Correlation Map is also interesting to compare, with lower correlations between terms :
  

Maria Lanzerath also has a use case where she has shown the use and benefit of OMARS on her use case.

• Video available here : https://www.youtube.com/watch?v=MUv6-oQWnSg
• Slides available here Slides or on ISEA website : https://isea-change.org/event-5399089  

Intellectually, this design is very interesting as it provides a family that unify and bridge the gap between DSD and RSD. On some use cases, it could be interesting to try them and compare them with other designs type. You'll have similar constraints as DSDs for their use: no mixture factor, no constraints, no hard-to-change factors/Split-plot situation, no random effects (even if you can use block in DSDs and OMARS), ...

But I don't think developpers will push forward the implementation of OMARS design in JMP, as Peter and Jose have created their own company EFFEX distributing OMARS designs in a web-based interface. You can still experiment with OMARS designs with the add-in and see how it compare to other designs.

I think there is already a sufficient diversity of designs to choose from (DSDs, OMLs, RSM, Optimal, ...), so I didn't had the chance to try using OMARS designs on real use cases.

I hope this discussion starter may help you,

[#3]

OK Victor, that's clear to me. Personally I am not a big fan of solving everything with one big experiment; building knowledge by a sequential approach is my favorite way to go and so far screening strong/active effects with DSD and in a 2nd step augmenting the DOE to RSM with active effects nice results are achieved; in many cases, with only < 4 significant effects, augmenting was not even necessary so with only few runs there's a solution. I still am not conviced about OMARS; to my opinion with no or only few knowlege push is too hard to get everything done with one experiment, I like more the sequential quality PDCA approach. I agree that when there is sufficient knowledge OMARS could be beneficial (?)

[#4]

Frank, I agree, a sequential approach is always better.

IMHO, there will always be those that look to optimize the design structure for a given situation (e.g., optimality criteria, et. al.), BUT the most important aspect of experimentation is how you handle the noise (e.g., short-term noise like measurement error, within batch, within part and long-term noise like ambient conditions, lot-to-lot raw materials, human technique).  If the noise is constant during the experiment, then you have a narrow inference space where results will likely not extrapolate.  If the noise "randomly" varies during the experiment, then you compromise precision.  The strategies to handle noise are not well described by the software (any software).  There is always too much focus on the design structure (and how you can economize runs to learn more).

 “Unfortunately, future experiments (future trials, tomorrow’s production) will be affected by environmental conditions (temperature, materials, people) different from those that affect this experiment…It is only by knowledge of the subject matter, possibly aided by further experiments  (italics added) to cover a wider range of conditions, that one may decide, with a risk of being wrong, whether the environmental conditions of the future will be near enough the same as those of today to permit use of results in hand.”

Dr. Deming

"Before you make general rule of this case, test it two or three times and observe whether the tests produce the same effects"

Leonardo da Vinci

"BLOCK WHAT YOU CAN, RANDOMIZE WHAT YOU CANNOT"  

 G.E.P. Box

[#5]

Strong arguments in favor of sequential experimentation! Build models with factors that are strong enough to overcome noise. The lattercan be detected with limited run screening experiments, eventually with replication. DSD still is a great tool!

[#6]

Is OMARS possible with > 2 level categorical effects? Also with many or only categorical effects?

[#7]

@frankderuyck I do agree about your sequential experimentation discussion with @statman and how to handle noise.

However, let's remind that DoEs are often applied in a context of real physical/chemical experiments, so you also have to consider cost and time regarding the choice of a design (and associated runs number). If a response needs a high duration/time length to be measured/recorded (for example stability studies for several months/years), it may be more reasonable to do a bigger "one-shot" DoE and have all the needed results at the same time, instead of several sequential ones where you will spend a lot of time at each iteration. Maria Lanzerath also emphasizes in her use case on the benefits of OMARS designs regarding model selection, in context with model complexity related to the available number of possible terms.

Concerning your questions :

• OMARS can be done with any number of 2-levels categorical factors or continuous factors,
• I don't know how relevant OMARS may be for studies involving only categorical factors (there might be better options).
  But they can include any number of 2-levels categorical factors without loosing orthogonality properties. This was a drawback of DSDs when you introduced several categorical 2-levels factors. This situation was solved by the introduction of Orthogonally Mixed Levels designs presented by Bradley Jones last year, and available in JMP 18.
  Information about OML designs : 
  Developer Tutorial: Using JMP to Create Orthogonal Mixed-level Screening Designs 
    
  Comparison of correlation map between OML and DSD created for 2 two-levels categorical factors and 4 continuous factors :
  

  You can see that in the case of DSD, you have small correlations between main effects of categorical factors and the other main effects. With OML, you have a structure that enable to have no correlation between main effects and between main effects and 2nd order terms. OML also enable to have other/different design size than DSDs.

   

Hope this answer will be helpful and fruitful to the discussion,

[#8]

Thanks Victor, OML is interesting. Also 3-level categorical effects are possible, also with OMARS?

By the way I had a look at the case study of Maria Lanzerath where quite a few knowlege is available and assumptions are made on interaction effects; in this case a equally good DOE can be constructed with Custom Design skipping the unnecessary interactions and if doubt switch to "if possible".

[#9]

No, 3-levels categorical factors are not possible with DSDs, OMARS or OML designs, only 2-levels categorical factors.

[#10]

Hi Victor, comparing OML & DSD what about the correlations of the 4 quadratic effects? From the correlation tables above I would prefer the DSD: correlation between some main efects is véry low and negigible and each interaction effect is correlated with only 1 other interaction effect. Correlation structure of OML interactions is quite complex; think in OML algorithm there is too hard push to get zero correlation between main efffects resulting in more comlex correlation between interactions (?) (there is no free lunch, a price has to be paid..) Achieve zero correlation between main effects too my opnion is not necessary, in lot of unbalanced custom designs and when incuding covariate factors there's always a little correlation among main factors; R < 0,2 is acceptable.