Hello,  everyone, and  welcome  to  JMP  Discovery.

Andrew  Karl  and  Heath  Rushing.

I  have  a  presentation  today that's  going  to  highlight

some  of  the  features  in  JMP  17   Pro that  will  help  you  out

in  the  world  of  mixture  models.

Many  of  you   are  involved  in  formulations,

and  to  be  honest,   that's  what  we've  been  doing  a  lot  lately.

We're  a  lot  like  ambulance  chasers, where  we'll  just  go  after

the  latest  thing    that customers  are  interested  in.

But  that's  what  we're  seeing  a  lot  lately, is  folks  that  are  doing  mixture  models

that  are  actually  quite  complex and  more  so  than  we'd  ever  know.

We  just  decided   we  would  do  some  deeper  investigation

with  some  of   the  new  techniques that are out,

and  that  JMP  17   performs  for  us.

Andrew,  would  you  like  to  get  started and  maybe  give  a  little  bit  of  background

on  the  whole  idea   of  what  a  mixture  model  is,

and  some  of   the  other  techniques?

Okay,  so let's  start  out with  a  nice,  easy  graph.

Let's  take  a  look  over  here at  the  plot  on  the  left.

We're  in  an  experimental  setting,

so now  I  suppose   we've  got  two  factors,

Ingredient  A  and  Ingredient  B, and  they  range  from   0-1.

If  there's  no  mixture  constraints, then  everything  in  the  black  square

is  a  feasible  point in  this  factor  space,

and so  our  design  routine  is  going to  give  us  points  somewhere  in  this  space.

However, if  there's  a  mixture  constraint

where  these  have  to  add  up  to  one, then  only  the  red  line  is  feasible.

We  want  to  get   a  conditional  optimum

given  that  constraint, and  we  want  to  end  up

somewhere  in  that  line   for  both  our  design  and  our  analysis.

If  we  move  up   to  three  mixture  ingredients,

A,  B  and  C,   all  able  to  vary  from   0-1,

then  we  get  a  cube  for  that   0-1  constraint  for  each  of  them.

But  with  the  mixture  constraints, that  takes  the  form  of  a  plane

intersecting  a  cube, and  that  gives  us  this  triangle,

so  only  this  red  triangle is  relevant  out  of  that  entire  space.

If  we  have  four  dimensions, if  we  have  four  mixture  factors,

then  that  allowable  factor  space is  actually  a  three- dimensional  subset,

a  pyramid  within  there.

Looking  back  to   the  three- mixture  setting.

See  this  triangle? That's  the  allowed  region.

Well,  that's  why JMP  gives  us  these  ternary  plots.

For these ternary plots,

what  JMP  will  do  is,  if  you  have   more  than  three  mixture  factors,

is  you'll  have  two  factors   shown  at  a  time,

and  the  third  axis  will  be the  sum  of  all  the  other  mixture  factors.

We  can  look  at  these  ternary plots,  rather  than  having  to  have  a  pyramid

that  we're  looking  throughout.

We  have  to  decide, do we  want  a   Space Filling Design

or  an  optimal  design?

Now,  normally  in  a  non-mixture  setting, we'd  normally  use  an  optimal  design,

and  for  the  most  part,  we  wouldn't consider  a   Space Filling Design.

There's  a  few  reasons that  we  want  to  consider

a   Space Filling Design in  mixture  settings.

Often  in  the  formulations  world, if  you  go  too  far,

there's  a  little  bit  more  sensitivity to  going  too  far  in  your  factor  space,

making  it  too  wide, then  your  entire  process  fails.

S uppose  that  happens over  here  where   X2  is.

Suppose  it  fails   everything  below   0.1.

You're  going  to  lose  a  good  chunk   of your runs  because  the  optimal  design

tends to  put  most  of  your  runs on  the  boundary  of  the  factor  space,

so  you're  going  to  lose  these.

You're  not  going  to  be  able  to run   your  full  model

with  the  remaining  points, and  you're  not  going  to  have

any  good  information  about   where  that  failure  boundary  is.

For  the   Space Filling Design, if  you  have  some  kind  of  failure

below 0.1,  you're  losing a  smaller  number  of  points.

Your  remaining  points   still  give  you  a   Space Filling Design

in  the  existing  space   that  you  can  use  to  fit  the  model  effects,

and now  we're  going  to  be  able to  model  that  failure  boundary.

Also  in  the  mixture  world, we  often  see  higher  order  effects  active:

interactions   or  curvatures,  polynomials,

than  we  might  see in  the  non- mixture  setting.

If  we  don't  specify  those, because  any  of  these  models  are  optimal,

conditional  on  the  target  model  we  give, so  if  we  don't  specify  an  effect  operator

for  the  optimal  model, we  might  not  be  able  to  fit  it

after  the  fact, because  they  might  be  aliased

with  other  effects  that  we  have.

These  space filling  runs  act  as something of  a  catch- all  of  possible  runs,

so  there's  a  couple  of  reasons

that  we  might  want  to  consider space  filling  runs,

but  we  want  to   take  a  look  analytically.

What's  the  difference in  performance  between  these,

after  we  run  through  model  reduction, not  just  at  the  full  model,

because  we're  not   just going to be  using  the  full  model.

That's  the  design  phase  question.

When  we  get to  the  analysis,

whenever  you're  looking  at these  mixture  experiments  in  JMP,

JMP  automatically turns  off  the  intercept  for  you,

because  if  you  want  to fit the  full  model

with  all  of  your  mixed  remain  effects, you  can't  include  the  intercept.

You'll  get  this  warning

because  the  mixture- made effects  are  constrained  to  one,

and  they  add  up  to what the  intercept  is,  so  they're  aliased.

Also,  if  we  want to  look  at  higher  order  effects,

we  can't  be  like  a  response  surface where  we  have  pure  quadratic  terms.

We  have  to  look  at these  Scheffé  Cubic  terms

because  if  we  try to  look  at  the  interactions

plus  the  pure  quadratic  terms, then  we  get  other  singularities.

Those  are  a  couple of  wrinkles  in  the  analysis.

However,  going  forward with  the  model  selection  methods,

Forward  Selection  or   Lasso, which  are  the  base  methods

of  the  SVEM  methods that we're  going  to  be  looking  at,

we  want  to  consider  sometimes  turning  off this  default  'no  intercept'  option.

What  we  find  is  for  the   Lasso method  we  actually  have  to  do that

in  order  to  get reasonable  results.

After  we  fit  our  model, now  we  want  to  do  model  reduction

to  kick  out irrelevant  factors.

We've  got  a  couple  of   different  ways  of  doing  that  in  base  JMP .

Probably  what  people  do  most  frequently is  they  use  the  effect  summary

to  go  backwards  on  the  p- values, and  kick  out  the  small  p- values.

But  this  is  pretty  unstable because  of  the  multicollinearity

from  the  mixture  constraint, where  kicking  out  one  effect

can  drastically  change  the  p-values of  the  remaining  effects  in  the  design.

What  this  plot  shows  is if  we  go  backwards  on  p- values,

what  is  the  largest p-value  that's  kicked  out

and  we  see  some jumping around   here  and  that's  from  that  effect.

Given  that  kind  of volatility  in  the  fitting  process,

you  can  imagine if  you  have  small  changes

in  your  observed  responses, maybe  even  from  assay  variability,

or  any  other  variability, just  small  changes  can  lead

to  large  changes in  the  reduced  model

that  is  presented  as a  result  of  this  process.

That  high  model  variability is  something  that'd  be  nice

if  we  could  average  over  in  some  way,   in  the  same  way  that  maybe

with  Bootstrap  for est we  average  over  the  variability

of  the  CART  methods or  the  partition  methods

and  that's  what  the  SVEM  methods would  be  looking  at  doing.

In  the  loose  sense, they're  kind  of  the  analog for that,

for   the  linear  methods  we're  looking  at.

Also  we  can   go to  Step wise and  look  at  min AICc,

which  is  maybe  the  preferred  method.

In our  last  slide  of the show  today  we'll  be  taking  a  look  at,

for  the  base  JMP  users, AICc  versus  B IC  versus  p-value  selection

with  our  simulation  diagnostics.

Give  credit  to  a  lot  of  the  existing work  and  leading  up  to  the  SVEM  approach.

These  are  some  great  references that  we've  used  over  the  years.

Also  want  to  thank  Chris  Gottwalt  for  his patience  and  answering  questions

and  sharing  information  as  they've discovered  things  along  the  way.

That's really  helped  set  us  up  to  be  able to  put  this  to  good  effect  in  practice.

Speaking  of  in  practice, where  have  we  been  using  this  quite  a  bit

over  the  years  is  the  setting  of  liquid nanoparticle  formulation  optimization.

What  is  lipid  nanoparticle  formulation? Well, a lipid nanoparticle,

if  you've  gotten  any  mRNA  COVID  vaccines Pfizer,  Moderna,  then  you've  gotten  these.

What  these  do  is  you  take  a  mixture of  four  different  lipids,  summarized  here,

and  they  form  these   little  bitty  bubbles.

Then  those  are  electrically  charged,

then  they  carry  along  with  them a  genetic  payload,

mRNA,  DNA,  or  other  payloads that  either   act as  vaccines

or  can  target  cancer  cells.

The  electric  charge between  the  genetic  payload,

and then  the  opposite  charge  in  the nanoparticle  is  what  binds together.

Then  we  want it  to  get  through  the  cell and  then  release  the  payload  inside.

The  formulation  changes  depending on  what  the  payload  is.

A lso  sometimes  we  might  change the  type  of  ionizable  lipid,

or  the  type  of  he lper  lipid  to  see which  one  does  better,

so  we  have  to  redo  this process  over  and  over.

For  the  most  part, the  scientists  have  found  that  these

ranges  of  maybe 10-60% for  the  lipid  settings

and then  a  narrower  range  of   1-5%.

That's  for  the  most  part the  feasible  range  for  this  process.

That's  been  explored  out, and  that's  what  the  geometry

we  want  to  match here  in  our  application  is.

We  want  to  say, given that  structure that  we're  doing  over and over,

do we  have  an  ideal analysis  and  design  method  for  that?

A lso  we  want  to  set  up  a  simulation that  if  we're  looking  at  other  structures,

other  geometries  for  the  factor  space, maybe  we  can  generalize  to  that,

but  that's  going  to  be our  focus  for  right  now.

Given  that  background,

I'm  going  to  let  Jim  now  summarize the  SVEM  approach  and  talk  about  that.

Yes,  thank  you.

This  particular   discovery  presentation,

what  we're  going  to  do is  a  little  bit more centered  on   PowerPoint,

unfortunately,  because  the  results  are really  what  this  is  all  about

for  the  simulations  that  we've  done.

But  this  particular  session, I  will  show  you  some  of  the  JMP  17

that  we  have  new  capability.

If  I  go  and  I  want  to set  up  a   Space Filling Design...

Now,  previously  we  weren't  able to do  mixture  models  with   Space Filling Designs

right from  out  of  the  box,  if  you  will.

We  certainly  could  put constraints  in  there,

but  now  what  we  want  to  do is  show  you  how  you  can  do

a  Space F illing  Design with  these  mixture  factors.

This  is  new  that  I  now  have these  come  in  as  mixture,

which  is  good because  it  now  carries

all  of  those  column  properties with  it  as  well.

One  thing  worth  mentioning   right  now is

the  default  is  you're  going to  get  10  runs  per  factor,

so  40  runs  in  a  DoE   typically  is  good

and  we  are  happy because  our  power is  well  above  90%

or  whatever  our  criteria  is.

But  that's  not  the  case [inaudible 00:09:54]  these  mixture  models

because  there's  so many  constraints  inherent  in  it.

What  that  is  telling  me,  unfortunately, is  even  if  I  were  to  have  40  runs,

I'd  still  only  have  5%  power from  doing  Scheffé  Cubic

and  even  if  it's  main effects  only, there's  only  20%  power.

Power  now  is  not  really  a  design  criteria that  we're  going  to  look  to

when  we  do  these  mixture  models.

Now,  typically  in  our  applications,

unfortunately,  we  don't  have the  luxury  of  having  40  runs.

In  this  case  we'll  do 12  runs  and  see  how  that  comes  out.

We'll  go  ahead  and  make   that Space  Filling  Design,

and  you can see that it's  maybe evenly  spread  throughout  the  surface.

Of  course,  we  do  know that  we're   bounded  with  some  of  these  guys  here

that  we  can only go from  1 -5%  on  the  polyethylene  glycol.

What  I  want  to  do now is  just  go  to  fast  forward.

Now  let's  say  I've  run  this  design and  I'm  ready  to  do  the  analysis.

This  is  where  SVEM  is really  made  huge  headway

and  if  you  listen  to some  of  Chris  and  Phil  Ramsey's  work

out  there  on   JMP  community, you'll  see  this  is  a  step  change.

This  is  a  game  change in  terms  of  your  analytical  capability.

How  would  we  do  this  in  16?

In  JMP  16  what  we'd  have  to  do is  we'd  have  to  come  through,

and actually  it's  worth  going   through the  step  just  because  it  gives  you

a  little  bit  of  insight, though  the  primary  mission

of  this  briefing or  this  talk  is  not  SVEM,

it  will  give  us an  idea  of  what's  going  on.

What  we  can  do  here  is  we  can  go  ahead and  we'll  make  the  Auto validation  Table.

This  is  the  JMP  16  methodology.

What  you'll  note  here  is  we've  gone  from  12  runs  to  24.

We  just  doubled  them and  you  see  the  Validation  Set.

The  training  may  be    the first 12, validation  the  next  12.

That's  what's  added and  then  we  have  this  weight.

This  is  the  Bootstrap  weight and  it's   fractionally  weighted.

What  happens  is we  will  go  ahead  and  run  this  model

and  come  up  with  a  predicted  value,

but  then  we  need  to  change  these  weights and  then  keep  doing  this  over  and  over

for  our Bootstrap, much like a  random  forest,  idea  for  the  SVEM.

Now  what  is  useful is  to  kind  of  take  a  quick  look.

What  is  the  geometry   of  these  weights?

We  can   see  they're  anti- correlated,

meaning if in   the  training  set  that I'm  low,

I'm  probably  going to  be  high  in  the  validation  set.

This  is  kind  of  a  quick little  visual  of  that  relationship.

Now  I'm  ready  to  go   do  my  analysis  in  JMP  16.

It  would  be  analyzed  and  we'd  just  do  our  fit  model.

Of  course,  we  want a  generalized  regression

and  we'll  go  through and  do  a   Scheffé  Cubic  here,

because  it's  a  mixture.

But  here's  where we  have  to  add  in  the  step,

we  put  the  validation set as  your  validation  column

and  then  this  validation  weight is  going  to  be  that  frequency.

Now  I  can  run  this.

By  the  way  we  could,

in  many  of  our  instances   we're  not  normal,  we're  log  normal,

we  could  put  that  in  right  there.

Here  we  have  our generalized  regression  ability

to  go  ahead  and  run  this  model and   voila,  there  are  the  estimates.

What  we  would  do  then is we come here  under  the  Save  the  Prediction  Formula.

Then  here  is  one  run.

Okay,  so  we  got  one  run.

You  can  see  that  the  top  is  15.17,  and  we  actually  saw  15.23,

so  not  bad  in  this  model,

but  we  would  do  this  over  and  over.

We  used  to  do  it  about  50  times  or  so.

But  with  JMP  17, now  this  whole  process  is  automated  to  us.

We  don't  have  to  do  this 50  times  and  then  take  the  average

of  these  prediction  formulas.

We're  able  to  go  directly  at  it.

If  I  come  back  to  my original  design  here  with  the  response,

I  can  get  right  at  it.

By  the  way,  this  is  showing that I  have  another  constraint  put  in  here.

A  lot  of  times  we  have  the  chemists and  biochemists  like  to  see  that

to  make  sure  that  the  ratios based on  molecular  weights  are  within  reason.

Not  only  do  we  have the  mixture  constraints,

we  also  have a  lot  of  other  constraints.

I'm  working  with  a  group where  we  have  maybe  15  different  ingredients

and  probably  30  constraints in  addition  to  the  mixture  constraints,

so  these  methods   work  and scale up,

probably  is  the  best  way  to  say  it, pretty  well.

Now  this  is  17, so  17  I  can  get  right  at  it.

I'm  going  to  go  ahead  into  Fit  Model, and  I'll  go  ahead  and  do  a   Scheffé Cubic.

From  here,  what  we're  able  to  do is come  into  a  generalized  regression.

In  this  case,  we  don't  need to  worry  about  these  guys  in  here.

We  can  change  it  to log normal  if  we  so  desire.

One  of  my  choices  in  the  estimation

instead  of  Forward  is in  fact  SVEM  Forward,

so  I  do  SVEM  Forward and  I'm  going  to  go  do  200.

You'll  see how quickly  they  have  this  tuned.

Really the  only  thing  you  can  do in  advanced  controls

is  check  whether  or  not you  want  to  force  a  term  in.

I  hit  Go  and  instantaneously  I've  done 200  Bootstrap  samples  for  this  problem.

Of  course, I  now  can  look  at  the  profiler

and  that  is  the  average  of  the  200  runs.

That's  kind  of  my  end  model,  if  you  will.

Of  course,   with  Prediction  Profiler,

there  are hundreds  of  things  you  can  do from here

and  Andrew  will touch on  a  couple  more  of  those.

But  two  other  things worth  noting  here,

I'll  save  the  Prediction  Formula as  well  and   take  a  look  at  that  guy.

When  I  look  at  the  Prediction  Formula,

I'll  note  that  it  is  in  fact already  averaged  out  for  me  here.

This  is  the average  of the 200   different  samples  that  are  out  there.

With  that,  that  is  the  demo,

and  we'll  go  back  to looking  at  the  charts  there  to   say,

"Well, What  is  it  that  we're  seeing in  terms  of  the  results  of  SVEM?"

Andrew,  if  you  want  to  pull  up  that  slide.

This  is  maybe  a  quick  visual.

You  can  see  that if  I  look  at  those  first  three,

in  this  case,  red  is  bad.

What  we're  looking  at  here is  the  nominal  coverage.

This  is  a  mixture  model

at  the  predicted optimum  spot  of  performance.

We  can  see  that  the  standard Step wise  guys  are  not  doing  too  well.

That's  the  backward  and  forward  AIC.

This  is  the  coverage  rates.

We'd  like  it  to  be  a  nominal 5%  error  that  we  don't  actually  see

the  true  response  in  the  prediction or  actually  the  confidence  interval.

In  other  words, when  we  looked  at  the  profile

that  we  just  saw, it gives us  a  prediction  or  confidence  interval

was  the  true  value,

which  we  know  because we're  playing  the  simulation  game,  right?

We  know  what  the  true  value  is, what  percentage  of  time  was  that  in  there?

We  can  see  that  we  don't  do as  well  with  our  classical  methods.

The  full  model, putting  all  the  terms in,

Lasso  does  pretty  well at  a  10%  rate  or  so,

but  it's  not  until  we  get   the SVEM methods  here  that  we  start  seeing

that  we're  truly  capturing and  getting  good  coverage.

A  good  maybe  picture to  keep  in  your  mind,

that  we  are  way  outperforming  some  of  the  other  methods  out  there

when  it  comes  to  the  capability.

Now,  in  terms  of  how  this  simulation,

what  we're  focusing  on  here  is  a  little bit  different  than  what  you  may  think of,

in  terms  of  a  simulation  where   we're looking  at  a  model  and  saying,

"How  well  did  we  do  with  this particular  method?"

We  could  measure  that by  how  the  actual  versus  the  predicted  is

and  then  we'd  get   some sort of  a  mean  squared  error.

We  do  track  that  value,

but  we  find  in  our  work, we're  much  more  concerned  about  finding

that  optimal  mixture,  if  you  will, with  the  optimal  settings  that  achieve  us

a  maximum  potency or  minimizes some side-effects

or  helps  us  with this   [inaudible 00:18:58]

That's  going  to  be  called the "percent  of  max"  that  we're  looking  for.

We're  going  to  use  that as  our  primary  response  here

in  terms  of  being  able  to evaluate  which  methods  outperform  others.

It's  not  really  going  to  be,

how  far  away  am  I from  the  location  of  the  optimal  value?

It's  how  far  is that  response that  I  predicted  as optimum

how  far  is  that  from  the  actual  optimum?

That's  going  to  be our  measure  of  success.

The  way  this  will  work  is I'll  be   asking  Andrew  a  few  ideas  here

in  terms  of  what typically  comes  up  in  practice.

I  saw  the  geometry   he showed  me  early  on

and  the  optimal  design was  always  hit  the  boundaries.

What  if  I  like  things  that  we call  more  mix,  right?

You  have  more  mixed  stuff in  the  middle,

space fill,  which  is  better.

If  I  do  use  an  optimal  design,

it  defaults  to  D  maybe, but  what  about  I  and  A?

Then  how  about the  age- old  DOE  adding  center points?

Is  that  smart? Or  is  one  center point?

Or  how  about  replicates?

We've  already  discussed how  we're  not  being  helpful,

so  what  is  a  helpful  measure of  a  good  design?

That's  the  design  piece, but  also  the  analysis  piece is,

is  there  a  particular  method that  outperforms  everyone,

or  are  there  certain  areas that  we  should  focus  on

using  Lasso  and  others,

that  we  should just  use  SVEMs  for  selection?

These  are  practical  questions that  come  up  from  all  of  our  customers,

and  we'd  like  to  share  with  you

some  of  the  results that  we  get  from  the  simulation.

Andrew,  you  want  to  give  us

a  little  bit  more  insight into  our  simulation  process?

Yeah,  thanks,  Jim.

Before  I  do  that,

I  just  want  to  point  out  one  tool

that  we've  made  heavy  use  of in  the  analysis  of  our  results,

and  unfortunately,  we  don't  have  time to  delve  into  the  demo,

but  it  has  been  so  useful is, within  the  profiler

to  look  at  the  output  random  table for  these  mixture  designs

and  to  look  at  the  responses  especially,

we  frequently  have  potencies by  side  effects.

We  have  multiple  responses,

that we want to   balance  that  out with  the  desirability  function,

and  then  we're  going  to  look at  the  individual  responses  themselves.

When  we  output  a  random  table, we  get  a  Space  Filling  Design,

basically  not  a  design, but  we  fill  up  the  entire  factor  space,

and  we're  able  to  look at  the  marginal  impact

of  each  of  the  factors over  the  entire  factor  space.

For  example, for  the  ionized  lipid  type,

what  we'll  frequently  see  is, we  can  see  that  maybe  one  has

a  lower  marginal  behavior over  the  entire  space.

But  since  we're  wanting  to  optimize,

we  care  about  what the  max  of  each  of  these  is,

and  one  of  these will  clearly  be  better  or  worse.

We're  looking  at  the  reduced  model.

After  he  fits  them, we'll  go  to  the  profiler  and  do  this.

We  can  still  get the  analytic  optimum  from  profiler,

but  in  addition  to  that,

this  gives  us  more  information outside  of  just  that  optimum.

What  we  might  do  here  is for  candidate  runs,

because  we  always  running our  confirmation  runs

after  these  formulation and  optimization  problems

is  we  might  run the  global  optimum  here  for  H 102,

we  might  pick  out the  conditional  optimum  for  H 101

and  see  which  one does  better  in  practice.

Also,  looking  at  the  ternary  plots,

if  we  color  those  by  desirability or  by  the  response,

we  can  see  the  more or  less  desirable  regions

of  that  mixture  space,

so  that  can  help  us as  we  either  augment  the  design

or  either  include  additional  areas in  the  factor  space,

or  to  exclude  areas.

I  can't  do  much  more with  that  right  now,

but  I  wanted  to  point  that  out

because  that's  a  very  important  part of  our  analysis  process.

How  do  we  evaluate some  of  these  options

within  this  type  of  geometry of  a  factor  space?

We  built  a  simulation  script that  we  have  shared  on  the  JMP  website,

and  it  allows  us  to   plug  and  play for  different  sample  sizes  in  total,

how  many  runs are  in  the  design?

We  have  a  true  form  choice

that  gives  us  the  true  generating  function behind  the  process,

a  design  type, either  space  filling  or  optimal.

The  optimal  design  now  is  going  to  be of  a  certain  minimum  size

based  on  the  number  of  effects that  we're  targeting.

Do  we  have  a  second -order  model, a  third -order  model,

a   Scheffé Cubic  model?

What  do  we  have?

Normally,  whenever  you  build  a  model and  custom  design  in  JMP,

it  writes  a  model  script  out  to  your  table and  then  you  use  that  to  analyze  it.

Well,  something  we've  explored is  allowing  a  richer  model

than  what  we  get, what  we  target,

and  are  we  able  to  use these  methods  with  SVEM

and  get  additional  improved  results,

even  though  we  didn't  originally  target those  effects  in  the  design?

The  short  answer  there  is  yes.

That's  something  else we  want  to  consider,

so  we  allow  ourselves with  the  effective  choice

to  include  additional  effects.

We  can  look  at  the  impact of  adding  replicates  or  center points

and  that  custom  DWI  dialogue to  enforce  those.

How  does  that  affect  our  response?

Because  any  of  the  summaries that  you  get  out  of  the  design

and  out  the design  diagnostics are  beginning  targeting  the  full  model,

either  with  respect to  prediction  variants,

D-optimal, you're looking at standard  errors  for  the  parameters.

But  what  we  really  care  about is  how  good  is  the  optimum

that  we're  getting  out  of  this,

so  that's  what  we're  going  to  take a  look  at  with  these  simulations.

For  the  most  part in  these  LMP  optimization  scenarios,

a  lot  of  times, we'll  come  across  two  situations.

The  scientists  will  say, "I've  got  about  12  runs  available,

and  maybe  it's  not that  important  of  a  process,

or  the  material  is  very  expensive,

and  I  just  need  to  do  the  best  I  can with  12  runs.  That's  what  I've  got."

Or  it  might  be  something  where  they've  got a  budget  for  40  runs,

and  they  can  fit a  full  second -order  model

plus  third  order  mixture  effects,

and  we  want  to  try to  characterize  this  entire  factor  space

and  see  what  the  response  surface   looks  like  over  the  whole  thing.

Those  are  the  two  scenarios  we're  going   to  be  targeting  in  our  simulation.

Jim,  I  think  you  had  some  questions

about  performance under  different  scenarios.

What  was  your  first  question  there?

I  did.

I  guess  when  I  think  about a  12- run  scenario  here,

and  if  I  just  go  with  the  default, I'd  get  a  D -optimal

and  it  would  be main  effects  only.

I  recognize  I  could  do the  space  filling  like  I  just  did,

but  my  question  is, if  I  do  the  default,

which  one  of  the  analysis  methods would  be  preferred?

Or  is  there  one?

Okay,  so  taking  a  look  at  that.

For  the  D- optimal  design, as  a  general  rule,

it's  going  to  put  almost  all  of  its  runs along  the  boundary  of  the  factor  space

and  it's  not  going to  have  any  interior  runs

unless  you  have  quadratic  terms or  something  that  requires  that.

With  a  12 -run  design,

there's  90  degrees  of  freedom  required to  fit  all  the  main  effects  here.

We've  got  a  few  degrees of  freedom  for  error,

but  mostly  we're  only  targeting the  main  effects  here.

How  do  the  analysis  methods  do?

Is  there  any  difference in  the  analysis  methods?

What  we  do,  and  all  of  these we're  going  to  summarize,

we  show  the  percent  of  max for  all  of  our  simulations  that  we  do,

and  so  we  can  see  that  distribution for  each  of  the  analysis  methods,

all  for  this  12- run, D-optimal  design  target  effects.

Then  we  also  show any  significant  differences  between  these,

and  we're  just  using  students' T.

We're  not  making a  two  keys  adjustments, so  keep  that  in  mind

whenever  you're  looking at these  significant  values.

The  winner  here  is  our homemade  SVEM  neural  approach

because  it's  not  restricted to  only  looking  at  the  main  effects,

they  can  allow  some   additional  complexity  in  the  model,

and  so  it  wins  here.

Now,  don't  get  too  excited  about  that because  this  is  about  the  best

that  we've  seen  SVEM neural  do is in  these  small  settings.

But  if  we  are  running  more  than  one candidate  optimum  going  forward,

then  maybe  we  can  include  a  SVEM  neural, but  in  general,

we  wouldn't  recommend only  sticking  with a  SVEM  neural

just  because  it  tends  to  be  more  variable, have  heavier  low  tails.

What  are  the  other  results?

We  see  the  losers  in  this  application

or  anything  that's  doing the  single  shot  model  reduction

because  all  these  effects are  significant  in  the  model,

and  any  time  we  pull  one  of  them  out, we  are  going  to  get

a  suboptimal  representation of  our  process.

That's  why  in  this  case the  full  model  does  better  than  those.

But  what's  interesting  is the  SVEM  linear  approaches

are  able  to  at  least  match that  full  model  performance.

We're  not  losing  anything by  using  SVEM  in  this  application,

so  that's  a  nice  aspect where  we  don't  have  to  worry

about  the  smaller  setting.

Are  we  hurting  ourselves  at  all by  using  AICC?

Now,  something  else  we  tried  here is  given  the  same,

you've  only  got  12  runs.

You're  only  targeting  the  manufacture

and  the  D-optimal  criteria in  the  custom  DOE.

What  if  we  allow  the  fit  model to  consider  second -order  effects

plus  there are  mixture  effects,

so  which our  model  then was  targeted  to  do?

What  happens,  and  we  see  this  JMP  here,

this  SVEM  linear  methods  are  able to  utilize  that  information

and  give  us  better  percent  of  max, get  optimal  candidates,

and  those  are  our  winners  here  now is  these  SVEM  linear  methods.

What  we  see  is  that  interestingly,

the  base  methods  for  these   SVEM  approaches,  Ford  method,

or  Ford  selection  or  the  Lasso are  not  able  to  make  use  of  that,

only  the  SVEM  is, so  that's  a  nice  property.

They  actually  beat  out  Neural,

which  is  nice  because  now  these are  native  to  JMP  17

and  they  don't  require as  much  computation  time

or  manual  set  up  as  in  their  own.

What  we  start  to  see  here is  the  theme  that  we're  going  to  see

throughout  the  rest  of  the  day is  that  any  of  these  Lasso  approaches

with  no  intercept are  going  to  give  us  sub -optimal  results

because  without  the  intercept

and  the  penalization doesn't  work  right  in  Lasso,

so you  actually  want  to  turn  off the  default  option  of  no  intercept

if  you're  going  to  try to  use  SVEM  Lasso

or  even  just  Lasso without  an  intercept.

Okay,  so  I  guess  it  looks  like SVEM  Neural  did  well  there.

But  again,  that  is  not  native.

We  can't  do  that  with  JMP  17  Pro, that's  not  in there .

We  can,  we  have  to  have  a  manual   [inaudible 00:29:01]  scripted.

Yeah, it's  not  a  manual  option.

Okay,  this  is  good,

but  I'm  also  a  fan of  the  Space  Filling  Design,

so  how  does  that  play  out in  terms  of  the  analysis  methods?

For  the  Space Filling  Design, you  can  see  rather  than  having

all  the  points  along  the  exterior, along  the  boundary,

now  we  fill  up  the  interior  space

for  both  the  mixture  factors and  the  process  factors,

which  sometimes  in  practice what  we'll  do  is,

we'll  take  the  process  factors

and  round  these to  the  nearest   0.25  or  0.5

or  whatever  granularity  works  best  for  us, but  this  is  what  it  looks  like.

In  terms  of  the  results, how  do  they  perform?

Now  what  we're  going  to  do is  compare

the  concatenation  of  the  design  approach

along  with  the  analysis  method and  see  which  these  do  best.

Looking  at  now  still  allowing the  richer  second  and  third -order  model

for  the  selection  methods and  see  which  one  does  best.

When  we  look  at  the  comparison,

the  winners  are the  SVEM Linear  approaches,

Lasso  only  with  the  intercept, not  without  the  intercept,

and  the  D- optimal.

Again,  behind  the  scenes, you  have  to  remember,

now  you're  assuming for  this   D-optimal  approach

that  your  positive  model  is  true over  the  entire  factor  space

and  you've  got  constant  various over  that  factor  space.

If  you're  worried  about  failures along  the  boundary,

then  that's  something  else  to  take  into account,  and  it's  not  built  into  this.

You  have  to  consider  that.

But  if  you  are  confident,

maybe  you've  run  this  before and  you're  only  making  minor  changes,

then  the  way  to  go  is  the  D- optimal with  the  SVEM  approaches.

Down  here,  the  losers  are  the  Lasso, with  no  intercept.

We 're going to  avoid  those,

and  you  can  see those  heavy  tails  down  here.

Not  the  SVEM  Lasso, just  the  Lasso.

Actually  here's  the  SVEM  Lasso with  no  intercept  down here.

Yeah.

They  all  get  these  Fs, so  they  all  fail.

-Conveniently,  [crosstalk 00:30:48] . -Yeah

Okay.

What  often  will  come  up,

whether  it's  designed  up  front where  we've  done  our  12  runs,

and  the  boss,

she  has  some  more  questions and  we  have  more  runs.

If  we're  going  to  do  five  more  runs,

how  does  that  impact some  of  these  results?

When  you  say  five  runs, not  a  follow -up  study,

but  your  build  is  study either  12  or  17  runs

in  a  single  shot  right  now is  what  you're  considering,  right?

Yeah,  exactly.

Okay,  so  yeah, we  can  look  at  the  marginal  impact

because  there's  a  cost  to  you for  those  extra  five  runs.

What's  the  benefit of  those  five  extra  runs?

Using  the  design  analysis you  could  use,

look  at the  FDS  plot and  your  FDS  plus  means  lower,

reflecting  smaller  prediction  variants.

Power  is  not  that  useful for  these  mixture  effects  designs.

We  don't  care  about  the  parameters.

We  want  to  know,  how  well  would  we  do with  optimization?

That's  where  the  simulation's  handy, we  can  take  a  look  at  that.

How  does  your  distribution of  your  percent  of  max  change

as  you  go  from  12  to  17  runs?

Interestingly,  there's  no  benefit for  the  single  shot  40  ICC

to  having  17  versus  12  runs.

Now,  again,  right  now  we're  looking at  the  percentage  of  max.

If  you  look  at  your  error variance,

your prediction variance is  going  to  be  smaller,

and  there  might  be some  other   [inaudible 00:32:09] ,

but  mainly  your  prediction  variance

is  going  to  be  smaller if  you  look  at  that.

But  really,  we  don't  care  that  much about  prediction  variance.

We  want  to  know, where  is  that  optimum  point?

Because  after  this,  we're  going  to  be  running  confirmation  runs

and  maybe  in  replicas  at  that  point

to  get  an  idea  of  the  process and  assay  variance  then.

But  right  now,

we  are  just  trying  to  scout  out the  response  surface

to  find  our  optimal  formulation,

so  with  that  goal  in  mind, there's  no  benefit  for  a four-day  ICC.

Now  for  the  SVEM  methods,

we  do  see there  is  a  significant  difference

and  we  do  get  a  significant  improvement

in  terms  of  our  average percent  of  max  we  obtain,

and  maybe  not  as  heavy  tails  down  here.

But  now  you  need  to  know is  that  you  need  to  decide,

is  that  practically  significant?

Do you  want  to  move from  90%  to  92%  mean  percent  of  max

in  this  first  shot  with  five  extra  runs?

You  have  to  do  your  marginal  cost

original  benefit  analysis  there as  a  scientist

and  decide  if  that's  worth  it.

Just  looking  at  it  here, what  I  think  might  be  useful

because  you  have  to  run confirmation  runs  anyway

is  if  we  run  the  12 -run  design,

you  can  then  run a  candidate  optima  or  two

based  on  the  results  we  get,

and  then  plus a  couple  of  additional  runs  maybe

in  a  high -density  region for  what  that  looks  good,

or  even  augment  out your  factor  space  a  little  bit,

and  then  you're  still  running a  total  of  17  runs,

but  now  we're  going  to  have  even a  better  sense  of  the  good  region  here,

so  that's  something  to  consider.

Something  else  we  can  see

from  running  the  simulation with  17  runs  is,

let's  look  at  the  performance

of  each  of  the  fitting  methods within  each  iteration,

and  there's  actually a  surprisingly  low  correlation

between  the  performance

of  these  different  methods within  each  iteration.

We  can  use  that  to  our  benefit

because  we're  going  to  be  running confirmation  runs  after  this,

so  rather  than  just  having  to  take one  method  and  one  confirmation  point,

one  candidate  optimal  point,

if  we  were  to,  for  example, look  at  these  four  methods

and  then  take  the  candidate  optimum from  each  of  them,

then  we're  going  to  be  able  to  go  forward with  which  one  everyone  does  best.

We're  looking  at  the  maximum  of  these.

Rather  than  looking at  a  mean  of  92%  to  94%,

now  we're  looking  at  a  mean of  about  97%  with  a  smaller  tail

if  we  consider multiple  of  these  methods  at  once.

Okay,  very  useful.

Let's  now  put  our  eyes toward  the  40 -run  designs.

Very  good  information  in  terms of  my  smaller  run  designs.

Now  with  40,  how  does  it  play  out in  terms  of  these  analysis  methods?

Are  we  going  to  see  consistent  behavior with  what  we  saw  in  the  12 -run  design?

Then  how  about  the  Space  Filling  versus the  optimal  design,  D-optimal?

-I'd  be  interested  in  that. -Okay.

Well,  first  take  a  look at  the  D-optimal  design,   40  runs,

and  now  we're  targeting all  of  the  second -order  effects,

the  third -order  effects,  mixture  effects,

and  we're  targeting  all  the  effects

that  are  truly  present in  our  generating  function,

and  we  still  see  that  we're  loaded  up on  the  boundary  of  the  factor  space

with  the  optimal  design,

and  then  if  we  were  going to  see  now  with  the  space  filling  design,

we're  going  to  see  now we're  filling  up  the  interior

of  the  factor  space  for  the  mixtures

and  for  the  other continuous  process  factors.

Let's  see  what  the   performance  difference  is.

First  of  all, focus  on  the  space filling  design,

which  analysis  methods  do  best?

And  same  as  we  saw in  the  12 -run  example  of  the  SVEM  linear,

Ford  selection  with  that  intercept, Lasso with  the  intercept  does  the  best.

The  worst  case  you  can  do is  keeping  the  full  model,

or  then  trying  SVEM or  single  shot  Lasso   with  no  intercept

and  the   D-optimal  setting, same  winners,  which  is  reassuring

because  now  we  don't  have to  be  worried  about,

"Well,  we're changing  our  design  type. Now  we  got  to  change  our  analysis  type."

It's  good  to  see  this  consistency across  the  winners  of  the  analysis  type.

The  full  model  doesn't  do  as  poorly  here with  the  optimal  design,  I  think,

because  the  optimal  design is  targeting  that  model

and  the  losers  here  are  still the  Lasso  with  no  intercept.

Then  Neural  is  really  falling  behind  here, behind  the  other  methods.

Now  let's  compare  the  space  filling to  the  D-optimal  designs,

and  we  can  really  see

the  biggest  difference  here is  within  the  full  model,

the  space  filling  designs  are  much  worse than   the   D-optimal  design.

Anytime  you're  doing design  diagnostics,

that's  all  within  the  context of  the  full  model.

For  your  D -optimality  criteria,

your  average  prediction  variance, that's  all  there.

A  lot  of  times when  you  run  those  comparisons,

you're  going  to  see a  stark  difference  between  those

and  that's  what  you're  seeing  here.

However,  in  real  life,

we're  going  to  be  running a  model  reduction  technique.

With  SVEM,  even  the  single shot  methods  improve  it.

But  especially  with  SVEM  here,

it  really  closes  the  difference  between the  space  filling  and  the  optimal  design,

and  we  see  pretty  close  to  medium, and  slightly  heavier  tail  here.

But  now  you  can  look  at  this  and  say.

"Okay.  I  lose  a  little  bit with  space  f illing  design.

But  if  I  have  any  concerns  at all about  the  boundary  of  the  factor  space,

or  if  I'm  somewhat  limited in  how  many  confirmation  points  I  can  run

and  I  want  to  have  something that's  going  to  be  not  too  far  away

from  the  candidate  optimum that  I'm  going  to  carry  forward,

then  those  are  the  benefits of  the  space  filling  design."

Now  we  can  weigh  those  out. We're  not  stuck

with  this  drastic  difference between  the  two.

Again,  that's  based  only  versus the   D-optimal  design .

I  guess  a  lot  of  times  in  our  DOE  work,

we  like  to  maybe  look at  the  I-optimality  criteria

and  even  the  A has  done  really  well  for  us.

In  particular, it  spreads  it. It's c ertainly  not  space  filling,

but  at  least  it  spreads  it  out a  bit  more  than  the  D -optimal.

Do  we  have  any  ideas how  those  I  and  A  optimal  work?

Yeah,  we  can  swap  those  out into  simulations.

One  thing  we've  always  noticed,

I  love  the  A -optimal  designs in  the  non- mixture  setting.

It's  almost  my  default  now.

I  really  like  them.

But  in  the  mixture  setting, whenever  we  try  them,

even  before  the  simulations, if  we  look  at  the  design  diagnostics,

the  A -optimal  never  does  as  well as  the  D  or  the  I -optimal,

and  that  bears  out  here in  the  simulations,

that's  the  blue  here  for  the  optimal, gives  us  inferior  results.

Rule  of  thumb  here  is,

don't  bother  with  the  A-optimal  designs for  mixture  designs.

Now  for  D  versus  I -optimal, we  don't  see  any...

In  this  application for  this  generating  function,

we  don't  see  any  difference  between  them.

However,  a  reason  to  slightly  prefer the   D-optimal  is,

there  tends  to  be  some  convergence  issues for  these  LNP  settings

where  you've  got  to  peg  over  the  one  5%

and  you're  trying  to  target a   Scheffé Cubic  model  in  JMP,

so we've  noticed  sometimes some  convergence  problems

for  the   I-optimal  designs and  it  takes  longer.

The  D -optimal, if  there's  not  much  of  a  benefit,

then  it  seems  to  be  the  safer  bet to  stick  with  the   D-optimal.

Now  we  weren't  able to  test  that  with  the  simulations

because  right  now  in  JMP,

you  can't  script  in  Scheffé Cubic  terms  into  the  DOE

to  build  an  optimal  design.

You  have  to  do  that  through  the  GUI.

We  weren't  able  to  test  that, see  how  often  that  happens,

but  that's  why  we've  carried  forward

D-optimal  in  these  simulations and  we  stick  with  those.

If  you  want  to  in  your  applications, you  can  try  both  D  and  I

and  see  what  they  look  like

both  graphically and  with  the  diagnostics,

but  the   D-optimal  seems to  be  performing  well.

Okay,  I  guess  just  keep  pulling  the  thread a  little  bit  further  is,

a  lot  of  times we'll  try  some  type  of  a  hybrid  design .

Why  don't  we  start  out  with, say,  25  space  filling  runs,

and  then  augment  that with  some   D-optimal  criterion

to   make  sure  that  we  can  target the  specific  parameters  of interest?

Does  that  work  out  pretty  well?

Yeah,  we  can  simulate  that and  we  take  a  look.

Either  we've  got...

This  is  the  same  simulated  function,

generating  function  we've  been  looking  at for  you  to run  the  D-optimal,

for you to run the  space  filling,

or  a  hybrid,  where  we  start  out with  25  space  filling  runs

and  then  we  go  to  augment  and  load in  building  15  additional  runs  targeting

the  third  order  model, and  what   we  see  is  that  now,

we  have  no  significant  difference in  terms  of  the  optimization

between  the  40-run D-optimal and  the  hybrid  design,

But  in  the  hybrid  design,

we  get  the  benefit   of  those  25  space filling runs.

We  get  some  interior  runs  protection  to  fit  additional  effects

and  protection  against  failures   along  the  boundary.

It's  a  little  bit  more  work   to  set  this  up.

We'll  do  this  for  high  priority  projects

because  only  for  those because  of that extra  cost  and  time.

But  it  does  appear  to  be   a  promising  method.

Right.

Practically  you  think  about   where  your  optimal  is  going  to  be,

there's  a  good  chance   it  could  be  in  that  interior  space

that's  not  filled  in  the  D-optimal   along  the  boundaries.