Hello. My  name  is  Mark  Bailey.

I'm  a  senior  Analytics  Software  Tester at  JMP.

My colleague  today  is Jianf eng  Ding,

a  Senior  Research Statistician  Developer.

I'm  going  to  start  the  presentation

about  some  new  approaches  to  comparisons that  will  be  available  in  JMP  17.

I'm  going  to  start  over. I  don't  know  why.

Hello. My  name  is  Mark  Bailey.

I'm  a  Senior  Analytics  Software  Tester at  JMP.

My  co-presenter today  is  Jianfeng  Ding,

a  Senior  Research Statistician  and  developer.

I'm  going  to  begin  with an  introduction  to  our  topic.

Before  we  talk  about  specific  comparisons,

we'd  like  to  introduce some  fundamental  concepts.

All  of  this  has  to  do  when we're  comparing  populations.

Comparing  populations is  a  very  common  task.

The  comparison,  we  hope,  will  lead to  a  decision  between  two  hypotheses.

Samples  from  these  populations  are often  collected  for  the  comparison.

S tatistical  inference  can  provide  some valuable  information  about  our  samples.

In  particular,

is  there  sufficient  evidence  to  reject one  hypothesis  about  these  populations.

A clear  statement  of  the  hypothesis  is

really  essential  to  making  the  correct choice  of  a  test  for  your  comparison.

These  hypotheses represent  two  mutually  exclusive  ideas

that  together  include the  only  possibilities.

They're  called  the  alternative and  null  hypotheses.

The  alternative  hypothesis

is  really  a  statement  about the  conclusion  that  we  want  to  claim.

It  serves  to  represent  the  populations and  it  will  require  sufficient  evidence

to  overthrow  the  other  hypothesis, which  is  the  null  hypothesis.

It  states  the  opposing  conclusion  that must  be  overcome  with  strong  evidence.

It  serves  as  a  reference  for  comparison and  it's  assumed  to  be  true.

This  is  important  that  we sort  this  out  today  because  historically,

statistical  training  has  presented  only one  way  of  using  these  hypotheses.

The  most  often  taught  statistical  tests

are  used  to  demonstrate  a  difference between  the  populations.

But  that's  not  the  only  possibility.

The  lack  of  understanding about  this  distinction

can  lead  to  misusing  these  tests.

The  choice  of  a  test  is  not  a  matter of  the  data  that's  collected

or  how  the  data  is  collected.

It's  strictly  a  matter  of the  stated  hypotheses

for  the  purpose of  your  comparison.

Let's  look  at  two  similar  examples  that are  actually  fundamentally  different.

But  let's  start  out  where  we  have a  purpose  of  demonstrating  a  difference.

In  this  example,  let's  say  I  would  like  to demonstrate  that  a  change  in  temperature

will  cause  a  new  outcome, an  improvement  perhaps.

We  want  to  claim  that  a  new  level

of  our  response  will  result from  changing  the  process  temperature.

We'll  use  a  designed  experiment to  randomly  sample  from  a  population

for  the  low  temperature  condition and  the  high  temperature  condition.

The  two  hypotheses  are  the  null  states

that  the  temperature  does not  affect  the  outcome.

This  will  be  our  reference.

The  alternative  states  our  claim,

which  is  the  temperature  affects the  outcome,

but  only  if  the  evidence  is  strong enough  to  reject  the  null  hypothesis.

All  right,  this  is  going  to  sound  very similar,  but  it's  exactly  the  opposite.

In  this  case,  an  example  two, we  need  to  demonstrate  equivalents.

Here  we  want  to  demonstrate

that  a  temperature  change doe  not  cause  a  new  outcome.

That  is,  after  the  change, we  have  the  same  outcome.

For  example,  this  might  be  the  case  where

we  are  planning  to  change the  process  temperature

to  improve  the  yield,

but  we  want  to  make  sure that  it  doesn't  change

the  level  of  an impurity  in  our  product.

We  design  the  same  experiment to  collect  the  same  data

and  we  have  the  same  two  hypotheses, but  now  they're  reversed.

It's  the  null  that  states that  the  temperature  affects  the  outcome,

that  is,  there's  a  difference,

while  the  alternative  states

that  our  change  in  temperature will  not  affect  the  outcome.

Are  we  testing  for  a  difference or  for  equivalents?

Really  we  see  that  from  these  examples

that  it's  not  the  data,  the  data identical,  but  the  tests  are  different.

the  choice  is  not  about  the  data,

it's  about  our  claim,  or  in  other  words, how  we  state  our  hypotheses.

Also  remember  that  hypothesis tests  are  unidirectional.

They  serve  only to  reject  a  null  hypothesis

of  a  high probability  when  it's  false.

in  our  presentation  today,

we'd  like  to  introduce  some  new equivalents  tests  as  well  as  some

additional  methods  that  are  used  when comparing  two  measurement  systems.

I'm  now  going  to  hand  it  over  to  Jianfe ng to  talk  about  equivalence  test.

Thanks  Mark.

Hello.

I'm  Jianfeng Ding .

I'm  a  Research  Statistician Developer  at  JMP.

In  this  video I'm  going  to  talk  about  the  equivalence,

non-infererority and  superiority  test  in  JMP  17.

The  classical  hypothesis  test  on  the  left

is  the  test  that  most  quality professionals  are  familiar  with.

It  is  often  used  to  compare two  or  more  groups  of  data

to  determine  whether they  are  statistically  different.

The  parameters  data  can  be  a  mean  response

for  continuous  outcome  and  a  proportion when  the  outcome  variable  is  binary.

Theta  t  represents  the  response from  treatment  group

and  theta  zero  represents response  from  a  control  group.

There  are  three  types of  the  classic  hypothesis  test.

The  first  one  is  the  two  sided  test and  the  rest  are  one  sided  tests.

If  you  are  looking  at  this  two  side  test on  the  left,

the  no  hypothesis  is  that the  treatment  means  are  same

and  the  alternative  hypothesis  is  that the  treatment  means  are  different.

Sometimes  we  really  need  to  establish that  things  are  substantially  the  same

and  the  machinery  to  do  that  is called  an  Equivalence  Test.

An  equivalent  test  is  to  show the  difference  in  theta  t  and  theta  zero

is  within  a  prespecified  margin  delta

and  allow  us  to  conclude  the  equivalence with  a  specified  confidence  level.

If  you  look  at  the  equivalence  test, the  no  hypothesis  is  that

the  treat  statement  means  are  different and  the  alternative  hypothesis  is  that

the  treatment  means  are  within a  fixed  delta  of  one  another.

This  is  different  from  the  two sided  hypothesis  test  on  the  left.

Another  alternative  testing  scenario  is the  Non-inferiority  Test,

which  aims  to  demonstrate  that  results are  not  substantially  worse.

There  is  also  a  testing  scenario  called superiority  testing,

that  is  similar  to   non-inferiority  testing,

except  that  the  goal  is  to  demonstrate that  results  are  substantially  better.

There  are  five  different  types of  equivalent  type  test

depend  on  the  situation.

When  should  we  use  this  test will  be  discussed  next.

These  tests are  very  important  in  industry,

especially  in  the  biotech and  pharma  industry.

Here  are  some  examples,

if  the  goal  is  to  show  that  the  new  treatment

does  not  differ  significantly from  the  standard  one

by  more  than  some  small  margin, then  equivalent  test  should  be  used.

For  example,  a  generic  drug that  is  less  expensive

and  cause  few  side  effects than  a  popular  name  branded  drug.

You  would  like  to  prove  it  has  same efficacy  as  the  name  brand  one.

The  typical  goal in  non-inferiority  testing  is  to  conclude

that  a  new  treatment  or  process

is  not  significantly worse  than  the  standard  one.

For  example,  a  new  manufacturing process  is  faster.

You  would  make  sure  it  creates  no  more product  defects  than  the  standard  process.

A  superiority  test  tries  to  prove

that  the  new  treatment  is  substantially better  than  the  standard  one.

For  example,  a  new  fertilizer  has  been developed  with  several  improvements.

The  research  wants  to  show

that  the  new  fertilizer  is  better than  the  current  fertilizer.

How  to  set  up  the  Hypothesis.

The  graph  on  the  left,

summarize  these  five  different type  of  equivalent  type  tests  very  nicely.

This  graph  is  created  by  SAS  STAT  College, john  Castelloe  and  Donna  Watts.

You  can  find  their white  paper  easily  on  the  web.

Choosing  which  test depend  on  the  situation.

For  each  of  the  situation,

the  blue  region  is  the  region  that  you are  trying  to  establish  with  the  test.

For  equivalent  analysis, you  can  construct  an  equivalence  region

with  upper  bound  theta  zero  plus  delta and  lower  bound  theta  zero  minus  delta.

You  can  conduct an  equivalence  test  by  checking

whether  the  confidence  interval  of  theta

lies  entirely in  the  blue  equivalence  region.

Likewise,  you  can  conduct a  non-inferiority  test  by  checking

whether  the  confidence  interval  of  theta

lies  entirely  above the  lower  bound  if  large  theta  is  better,

or  below  the  upper  bound  if smaller  theta  is  better.

These  tests  are  available  in  one  way for  comparing  normal  means

and  in contingency for  comparing  response  rates.

The  graphical  user  interface of  equivalence  test  launch  dialog

makes  it  easy  for  you to  find  the  type  of  test

that  corresponds to  what  you  are  trying  to  establish.

A   [inaudible 00:12:00]  in  the  report summarize  the  comparison  very  nicely

and  makes  it  easy  for  you to  interpret  the  results.

Next,  I'm  going  to  demonstrate equivalence  tests.

I'm  going  to  use  the  data  set called  Drug  measurements.

That  is  in JMP  sample data  as  my  first  example.

Twelve  different  subjects  were  given three  different  drugs  A,  B  and  C.

And  32 continuous  measurements  are  collected.

We  go  to  theta  YX,

and  we  load  the  response and  the treatment.

This  will  bring  the  one  way  analysis.

Under  the  red  triangle find  the  equivalence  test.

There  are  two  options  means and  standard  deviations.

We  are  going  to  focus on  means  in  this  talk.

we  bring  the  dialogue  and  you  can  select a  test  that  you  would  like  to  conduct

and  the  graph  will represent  the  selected  test.

For  the  superiority  or   non-infererority test  there  are  two  scenarios.

Large  difference  is  better  or smaller  difference  is  better.

Choose  option  depend  on  the  situation.

You  also  need  to  specify the  margin  here  for  the  delta.

You  need  to  specify the  significance  level  alpha  as  well.

You  can  choose  use  pooled  variance  or unequal  variance  to  run  the  test.

You  can  do  all  pair  wise  comparison

or  you  can  do  a  comparison with  the  control  group.

We're  going  to  run an  equivalence  test  first

and  we  will  specify  the  three  as margin  for  the  difference.

We  click  the  Okay  button.

Here  is  the  result of  the  equivalence  test.

From  this  forest plot you  can  see  that

the  confidence  interval  for  the  main difference  between  drug  A  and  drug  C

is  completely  contained in  this  blue  equivalence  region.

The  max  P-value  is  zero  less  than  .05.

We  can  conclude to  the  .o5  significance  level.

Drug  A  and  drug  C  are  equivalent.

But  if  we  look  at  drug  A  and  B, drug  B  and  C

we  can  see  their  confidence  interval of  the  main  difference

is  both  beyond  this  blue  region.

At  the  .05  significance  level

we  cannot  conclude  that  drug  A  and  B or  drug  B  and  C  are  equivalent.

Assume  drug  C  is  our  standard  drug and  we  would  like  to  find  out

if  the  measurements  of  drug  A  or B  are  much  better  than  drug  C.

We  can  run  a  superiority  test  to  prove.

Let  me  close  this  outline  note  first and  we  bring  the  launch  dialogue  again.

This  time  we're  going to  do  a  superiority  test.

For  this  test  we  believe large  difference  is  better.

Here   we  keep  this  selection.

A lso  for  this  study  we  want  to  set drug  C  as  our  control  group.

We  plug  in  the  delta,  the  margin .04  for  this  case  click  Okay  button.

Here  is  the  result  for  superiority  test.

From  the  forest  plot  you  can  easily see  that  the  confidence  interval

of  mean  difference  between  drug  B  and  C

is  completely  contained in  this  superior  region

and  the  P-value is  less  than  .05 .

We  conclude  that  drug  B is  superior  to  drug  C.

The  confidence  interval  of  mean  difference

between  drug  A  and  C  is beyond  this  blue  region.

The  P-value  here is  much  bigger  than  .05 .

we  conclude  at  the  .05  significance  level

we  cannot  conclude  that drug  A  is  superior  to  drug  C.

This  concludes  my  first  example.

Now  I'm  going  to  use  a  second  example

to  use  the  relative  risk between  two  proportions

to  show  you  how to  conduct  a  non-inferiority  test.

Bring  the  data  table.

The  trial  is  try  to  compare  a  drug

called  FIDAX  as  alternative  to  drug  VANCO for  the  treatment  of  colon  infections.

Both  drugs  have  similar efficiency  and  safety.

221  out  of  224  patients  treat  with  FIDAX

achieved  clinical cure by  the  end  of  study.

Compare  to  223  out  of  257 patients  treated  with  VANCO.

We're  going  to  launch  Fe Y  by  X  again.

And  put  our  response  and  a  treatment variable  and  account  will  be  Freg.

Since  the  response variable  is  categorical.

Contingency  analysis  is  produced

and  all  the  test  here  is  based on  classical  hypothesis  test.

The  P-value  suggests  us that  we  cannot  conclude

that  clinical  Q  is different  across  the  drug.

But  for  this  study we  really  want  to  find  out

if  drug  FIDAX  is  not inferior  to  drug  VANCO.

We  go  to  red  triangle  menu, find  equivalent  test.

There  are  risk  difference and  relative  risk.

We  are  going  to  choose relative  risk  to  do  this  case.

In  the  launch  dialogue we  choose  non-inferiority  test

and  the  large  ratio is  preferred  by  us  for  this  study.

We  also  need  to  find the  category  interest.

For  this  study  we  select  yes as  a  category  of  interest

and  we  also need  to  plug  in  our  ratio  margin  here.

We  specify  zero  .09.

We  click  Okay  button  and  here  is the  result  of  non-inferiority  test.

From  the  forest  plot you  can  easily  see  that

the  confidence  interval  for  the  relative risk  between  drug  FIDAX and  drug  VANCO

is  completed contained  in  this  non-inferior  region.

We  conclude at  the  .05  significance  level,

drug  FIDAX  is  not inferior  to  drug  VANCO.

This  concludes  my  talk  and  I will  give  it  back  to  Mark.

Thank  you  JainFeng.

I'm  going  to  now  talk  about  a  very common  procedure  called  method  comparison.

It's  a  standard  practice  whenever  new measurements  are  being  developed.

We  have  to  assume  that  there  is a  standard  method  that  exists  already

to  measure  the  level  of  some  quantity.

Perhaps  it's  the  temperature or  the  potency  of  a  drug.

A new  method  has  been  developed for  some  reason.

We  want  to  make  sure  that  its  performance is  comparable  to  the  standard  method.

Today  there  are many  standards  that  have  been  developed

over  many  years  by  various  organizations to  make  sure  that  this  is  done  properly.

What  we  would  hope  is that  the  new  test  method

ideally  returns  the  same value  as  the  standard  method.

A scatter  plot  of  the  test  method versus  the  standard  method

would  show  that  the  data  agree with  the  identity  line  Y=X .

But  of  course  the  data  points won't  perfectly  agree

because  of  measurement  error

in  both  the  standard method  and  the  new  test  method.

Regression  analysis  can  determine the  best  fit  line  for  this  data

and  the  estimated  model  parameters  can be  compared  to  that  identity  line.

This  ends  up  being  stated  in  the  two hypotheses  as  follows.

The  null  hypothesis

says  that  they're  not  comparable  and  so another  way  of  saying  that  is

the  intercept  is  not  zero and  the  slope  is  not  one.

The  alternative

represents  our  claim  that the  new  method  is  comparable

and  so  we  would  expect  the  intercept to  be  zero  and  the  slope  to  be  one.

We'll  compare  by  using  regression and  ordinary  least squares  regression

assumes  a  few  different  things.

It  assumes  that  Y  and  X are  linearly  related.

It  assumes  that  there  are  statistical errors  in  Y  but  not  in  X.

These  statistical  errors  are  independent of  Y,  that  is,  they're  constant  for  all  Y.

There's  no  data  that  exert excessive  influence  on  the  estimates.

But  in  the  case  of  a  method  comparison,

the  data  often  violate  one  or more  of  these  assumptions.

There  are  measurement  errors in  the  standard  method  as  well.

Also,  the  errors  are  not  always  constant,

in  which  case  we  might  observe  that  the coefficient  of  variation  is  constant.

That  is,  the  errors  are  proportional,  but the  standard  deviation  is  not  constant.

Finally,  there  are  often  outliers  present

that  can  strongly  influence the  estimation  of  these  parameters.

other  regression  methods  can  help.

Deming  regression will  simultaneously  minimize

the  least  squared  error  in  both  Y  and  X

and  Passing-B ablok  regression is  a  non-parametric  method.

It's  based  on  the  median of  all  possible  pair-wise  slopes

and  because  of  that  it's  resistant to  outliers  and  non- constant  errors.

The  Deming  regression  is  available  in  JMP

through  the  Bivariate  platform using  the  Fit  Orthogonal  command.

Deming  regression  can  estimate the  regression  several  ways.

It  can  estimate  the  error  in  both  Y  and  X,

or  it  can  assume  that the  error  in  Y  and  X  are  equal,

or  it  can  use  a  given ratio  of  error  of  Y  to  X.

Passing  Bablo k  is  now  available  in  JMP  17,

again  through  the  Bivariate  platform using  the  Fit  passing  Bablok  command.

It  also  includes  checks for  the  assumptions  that

the  measurements are  highly  positively  correlated

and  exhibit  a  linear  relationship.

There's  also  a  comparison  by  difference.

The  Bland- Altman  analysis  compares

the  pair-wise  differences to  the  pair-wise  means

to  assess  the  bias between  these  two  measurements.

The  results  are  presented  in  a  scatter plot  of  Y  versus  X  for  your  examination

and  also  to  see  if  there  are  any anomalies  in  the  differences.

This  is  all  provided  through the  Match  Pairs  platform  along  with

several  hypothesis  tests.

I'll  now  demonstrate  these  features.

I'm  going  to  show  you Deming  regression  for  completeness

that's  actually  been available  in  JMP  for  many  years.

I'm  going  to  use  a  data  table  which  has measurements  for  20  samples

by  the  standard  method and  then  four  different  test  methods.

I'm  just  going  to  use  method  1.

I  start  this  by  selecting the  analyze  menu  set  Y  by  X.

The  standard  goes  in  the  X  roll, while  the  method  one  goes  in  the  Y  roll.

Here  we  have  the  scatter plot  to  begin  with.

I'll  click  the  red  triangle and  select  Fit  Orthogonal

and  you  can  see  the  different  choices  I mentioned  just  a  moment  ago.

I'm  going  to  have  JMP  estimate the  errors  in  Y  and  X.

There's  a  best  fit  line  using  Deming regression  along  with

information  about  that.

we  can  see  that  our  intercept for  the  estimated  line  is  close  to  zero,

our  slope  is  close  to  one,  and  in  fact our  confidence  interval  includes  one.

Now  I'm  going  to show  you  Passing  Bad blok.

I  return  to  the  same  red  triangle,

select  Fit  passing  Ba blok,

and  a  new  fitted  line is  added  to  my  scatter  plot.

It  looks  very  much  like  the  result from  the  Deming  regression.

But  remember  that  Passing  Ba blok

is  resistant  to  outliers or  non-constant  variance.

First  we  have  Kendall's  test that  is  telling  us  about  the  correlation.

Positive  correlation is  statistically  significant.

We  then  have  a  test, a  check  for  linearity,

and  we  have  a  high  P- value  here indicating  we  cannot  reject  linearity.

Finally  we  have the  regression  results.

I  see  that  I  have  an  intercept  close to  one,  but  the  interval  includes  zero,

so  I  can't  reject  zero.

The  slope  is  close  to  one.

My  interval  includes  one, so  I  can't  reject  that  the  slope  is  one.

Finally, using  Passing  Ba blok  this  curve  menu,

I'll  click  the  red  triangle and  select  Bland- Altman  analysis.

This  launches  the  Match  Pairs  platform, so  it's  a  separate  window.

Here  we  are  looking at  the  pair-wise  differences

between  method  one  and  the  standard

versus  the  mean  of  those  two  values.

We're  using  this  to  assess  a  bias.

The  Bland- Altman a nalysis is reported  at  the  bottom.

The  bias  is  the  average  difference.

We  hope  that  it's  zero.

The  estimate  is  not  exactly  zero,

but  we  can  see  that the  confidence  interval  includes  zero,

so  we  would  not  reject  zero.

We  also  then  have  lower  limits of  agreement,

and  we  see  that  they also  include  zero  as  well.

The  standard  methods  that  are  used

when  comparing  two  measurement  methods are  now  available  in  JMP  17.

That  concludes  our  presentation.

Thank  you  for  watching.