Hello. My  name  is  Bryan  Fricke.

I'm  a  product  manager  at  JMP  focused  on  the  JMP  user  experience.

Previously,  I  was  a  software  developer  working  on  exporting  reports

to  standalone  HTML  fire  files.

JMP  Live  and  JMP  Public.

In  this  presentation,

I'm  going  to  talk  about  using   Confidence Curves

as  an  alternative  to  null  hypothesis  significance  testing

in  the  context  of  predictive  model  screening.

Additional  material  on  this  subject  can  be  found  on  the  JMP  Community  website

in  the  paper  associated  with  this  presentation.

Dr.  Russ  Wolfinger  is a  Distinguished  Research  Fellow  at  JMP

and  a  co- author,  and  I  would  like  to  thank  him  for  his  contributions.

The  Model  Screening  Platform, introduced  in  JMP  Pro  16

allows  you  to  evaluate  the  performance

of  multiple  predictive  models using  cross  validation.

To  show  you  how the  Model  Screening  platform  works,

I'm  going  to  use  the  Diabetes  Data  table,

which  is  available  in  the  JMP sample  data  library.

I'll  choose  model  screening  from  the analyzed  predictive  modeling  menu.

JMP  responds  by  displaying the  Model  Screening  dialogue.

The  first  three  columns  in  the  data  table

represent  disease  progression in  continuous  binary  and  ordinal  forms.

I'll  use  the  continuous  column named  Y  as  the  response  variable.

I'll  use  the  columns  from  age  to  glucose in  the  X  factor  role.

I'll  type  1234  in  the  set  random  seed input  box  for  reproducibility,

I'll  select  the  check  box  next  to  K-Fold cross  validation  and  leave  K  set  to  five.

I'll   type 3  into  the  input box  next  to  repeated  K-F old.

In  the  method  list,  I'll  unselect  neural.

Now  I'll  click  Okay.

JMP  responds  by  training  and  validating models  for  each  of  the  selected  methods

using  their  default  parameter settings  and  cross  validation.

After  completing  the  training and  validating  process,

JMP  displays  the  results  in  a  new  window.

For  each  modeling  method.

The  Model  Screening  platform provides  performance  measures

in  the  form  of  point  estimates for  the  coefficient  of  determination,

also  known  as  R  squared, the  root  average  squared  error,

and  the  standard  deviation for  the  root  average  squared  error.

Now  I'll  click  select  dominant.

JMP  responds  by  highlighting  the  method

that  performs  best  across the  performance  measures.

What's  missing  here  is  a  graphic  to  show the  size  of  the  differences

between  the  dominant  method and  the  other  methods,

along  with  the  visualization of  the  uncertainty

associated  with  the  differences.

But  why  not  just  show  P- values  indicating whether  the  differences  are  significant?

Shouldn't  a  decision  about  whether one  model  is  superior  to  another

be  based  on  significance?

First,  since  the  P- value provides  a  probability

based  on  a  standardized  difference,

a  P-value  by  itself  loses information  about  the  raw  difference.

A a  significant  difference  doesn't imply  a  meaningful  difference.

Is  that  really  a  problem?

I  mean,  isn't  it  pointless

to  be  concerned  with  the  size  of the  difference  between  two  models

before  using  significance  testing

to  determine  whether the  difference  is  real?

The  problem  with  that  line  of  thinking  is that  it's  power  or  one  minus  beta

that  determines  our  ability to  correctly  reject  a  null  hypothesis.

Authors  such  as  Jacob  Cohen and  Frank  Smith

have  suggested  that  typical  studies have  the  power  to  detect  differences

in  the  range  of  .4 to .6 .

So  let's  suppose  we  have  a  difference.

Where  the  power  to  detect a  true  difference?

Zero  five  at  an  alpha  level  zero  five.

That  suggests  we  would  detect  the  true difference,  on  average  50%  of  the  time.

So  in  that  case,  significance testing  would  identify  real  differences

no  better  than flipping  an  unbiased  coin.

If  all  other  things  are  equal,   type 1 and   type 2  errors  are  equivalent.

But  significance  tests that  use  an  alpha  value  of  .05.

Often  implicitly  assume   type 2  errors are  preferable  to   type 1  errors,

particularly  if the  power  is  as  low  as  .5 .

A  common  suggestion  to  address  these and  other  issues  with  significance  testing

is  to  show  the  point  estimate  along with  the  confidence  intervals.

One  objection  to  doing  so  is  that

a  point  estimate along  with  a  95%  confidence  interval

is  effectively  the  same thing  as  significance  testing.

Even  if  we  assume  that  is  true, a  point  estimate  and  confidence  interval

still  puts  the  magnitude  of  the  difference

and  the  range  of  the  uncertainty front  and  center,

whereas  a  loan P-value  conceals  them  both.

So  various  authors, including  Cohen  and  Smith,

have  recommended  replacing significance  testing

with  point  estimates and  confidence  intervals.

Even  so,  the  recommendation  to  use confidence  intervals  begs  the  question,

which  ones  do  we  show?

Showing  only  the  95%  confidence  interval

would  likely  encourage  you  to  interpret  it as  another  form  of  significance  testing.

The  solution  provided  by   Confidence Curves

is  to  literally  show all  confidence  intervals

up  to  an  arbitrarily high  confidence  level.

How  do  I  show   Confidence Curves  and  JMP?

To  conveniently  create Confidence Curves  and  JMP,

install  the   Confidence Curves   add-in  by visiting  the  JMP  Community  Homepage.

Type   Confidence Curves into  the  search  input  field.

Click  the   Confidence Curves  result.

Now  click  the  download  icon  next to  confidencecurves. JMPa dd- in.

Now  click  the  downloaded  file.

JMP  responds  by  asking  if  I want  to  install  the  add- in.

You  would  click  Install.

However,  I'll  click  cancel  as I've  already  installed  the  add-in.

So  how  do  you  use  the  add-in?

First.

To  generate   Confidence Curves  for  this report,  select  Save  Results  table

from  the  top  red  triangle  menu  located on  the  Model  Screening  Report  window.

JMP  responds  by  creating a  new  table  containing  among  others.

The  following  columns  trial.

Which  contains  the  identifiers for  the  three  sets

of  cross validation  results.

Fold.

Which  contains  the  identifiers

for  the  five  distinct  sets  of  subsamples used  for  validation  in  each  trial.

Method.

Which  contains  the  methods  used  to  create models  from  the  test  data.

And  in.

Which  contains  the  number  of  data points  used  in  the  validation  folds.

Note  that  the  trial  column  will  be  missing if  the  number  of  repeats  is  exactly  one,

in  which  case  the  trial  column is  neither  created  nor  needed.

Save  for  that  exception, these  columns  are  essential

for  the   Confidence Curves add-in  to  function  properly.

In  addition  to  these  columns, you  need  one  column

that  provides the  metric  to  compare  between  methods.

I'll  be  using  R  squared  as  the  metric of  interest  in  this  presentation.

Once  you  have the  model  screening  results  table,

click   add-ins  from  JMPs  main  menu bar  and  then  select   Confidence Curves.

The  logic  that  follows  would  be  better placed  in  a  wizard,

and  I  hope  to  add  that  functionality in  a  future  release  of  this   add-in.

As  it  is,  the  first  dialog  that  appears

requests  you  to  select the  name  of  the  table

that  was  generated  when  you  chose 'save  results  table',

from  the Model  Screen  Reports  Red  Triangle  menu.

The  name  of  the  table  in  this  case  is Model  Screen  Statistics  Validation  Set.

Next,  a  dialogue  is  displayed

that  requests  the  name  of  the  method that  will  serve  as  the  baseline

from  which  all  the  other performance  metrics  are  measured.

I  suggest  starting  with  the  method that  was  selected  when  you  click

the  Select  Dominant  option

in  the  Model  Screening  Report  window, which  in  this  case  is  Fit  Stepwise.

Finally,  a  dialog  is  displayed

that  requests  you  to  select  the  metric  to be  compared  between  the  various  methods.

As  mentioned  earlier,  I'll  use R  squared  as  the  metric  for  comparison.

JMP  responds  by  creating  a  Confidence Curve  table  that  contains  P-values

and  corresponding  confidence  levels for  the  mean  difference

between  the  chosen  baseline  method and  each  of  the  other  methods.

More  specifically,  the  generated  table  has columns  for  the  following.

Model.

In  which  each  row  contains  the  name of  the  modeling  method

whose  performance  is  evaluated relative  to  the  baseline  method.

P- value  in  which  each  row contains  the  probability

associated with  the  performance  difference

at  least  as  extreme  as  the  value  shown in  the  difference  in  our  square  column.

Confidence  interval  in  which  each row  contains  the  confidence  level.

We  have  that  the  true  mean  is  contained in  the  associated  interval

and  finally,  difference  in  our  square,

in  which  each  row is  the  maximum  or  minimum

of  the  expected  difference  in  R  squared

associated  with  the  confidence  level shown  in  the  confidence  interval  column.

From  this  table, Confidence Curves  are  created

and  shown in  the  Graph  Builder  graph.

So  what  are   Confidence Curves?

To  clarify  the  key  attributes of  a   Confidence Curve,

I'll  hide  all  but  the  Support  Vector machine's   Confidence Curve

using  the  local  data  filter  by  clicking on  Support  Vector  Machines.

By  default,  a   Confidence Curve  only  shows

the  lines  that  connect  the  extremes of  each  confidence  interval.

To  see  the  points,  select  Show Control  Panel  from  the  red  triangle  menu

located  next  to  the  text that  reads  Graph  Builder  in  the  Title  bar.

Now  I'll  shift  click  the  points  icon.

JMP  responds  by  displaying  the  endpoints of  the  confidence  intervals

that  make  up  the   Confidence Curve.

Now  I  will  zoom  in  and  examine  a  point.

If  you  hover  the  mouse  pointer  over any  of  these  points.

A  hover  label  shows  the P - value confidence  interval,

difference  in  the  size of  the  metric

and  the  method  used to  generate  the  model

being  compared  to  the  reference  model.

Now  we'll  turn  off  the  points by  shift  clicking  the  points  icon

and  clicking  the  Done  button.

Even  though  the  individual  points are  no  longer  shown,

you  can  still  view the  associated  hover  label

by  placing  the  mouse  pointer over  the   Confidence Curve.

Point  estimate  for  the  main  difference

in  performance  between  the  sport  vector machines  and   Fit Step Wise  models

is  shown  at  the  0%  confidence  level,

which  is  the  mean  value  of  the  differences computed  using  cross  validation.

A   Confidence Curve  plots  the  extent

of  each  confidence  interval from  the  generated  table  between  zero

and  the  99.99%  confidence  level along  with  the  left  Y  axis.

P  values  associated  with the  confidence  intervals  are  shown

along  the  right  y  axis.

The  confidence  level  associated with  each  confidence  interval  shown.

The  Y  axis  uses  a  log  scale

so  that  more  resolution  is  shown at  higher  confidence  levels.

By  default,  two  reference  lines  are plotted  alongside  a   Confidence Curve.

The  vertical  line  represents

the  traditional  null  hypothesis of  no  difference  in  effect.

Note  you  can  change the  vertical  line  position

and  thereby the  implicit  null  hypothesis.

In  the  X axis  settings.

The  horizontal  line  passes  through the  conventional  95%  confidence  interval.

As  with  the  vertical  reference  line,

you  can  change the  horizontal  line  position

and  thereby the  implicit  level  of  significance

by  changing the  Y  axis  settings.

If  a   Confidence Curve  crosses  the  vertical line  above  the  horizontal  line,

you  cannot  reject  the  null  hypothesis using  significance  testing.

For  example,  we  cannot  reject  the  null hypothesis  for  support  vector  machines.

On  the  other  hand,  if  a   Confidence Curve

crosses  the  vertical  line  below the  horizontal  line,

you  can  reject  the  null  hypothesis using  significance  testing.

For  example,  we  can  reject  the  null hypothesis  for  boosted  tree.

How  are   Confidence Curves  computed?

The  current  implementation of  confidence  curves  assumes

the  differences  are  computed

using  R times   repeated K-fold  cross  validation.

The  extent  of  each confidence  interval  is  computed

using  what  is  known  as a  variance  corrected  resampled  T-test.

Note  that  authors  Claude  Nadeau and  Yoshua Bengio,

note  that  a  corrected resampled  T-test

is  typically  used  in  cases where  training  sets

are  five  or  ten times  larger  than  validation  sets.

For  more  details,  please  see  the  paper associated  with  this  presentation.

So  how  are   Confidence Curves  interpreted?

First,  the   Confidence Curve graphically  depicts

the  main  difference  in  the  metric of  interest  between  a  given  method

and  a  reference  method at  the  0%  confidence  level.

So  we  can  evaluate  whether  the  mean difference  between  the  methods

is  meaningful.

If  the  main  difference  isn't  meaningful, there's  little  point  in  further  analysis

of  a  given  method  versus  the  reference method  with  respect  to  the  chosen  metric.

What  constitutes  a  meaningful  difference depends  on  the  metric  of  interest

as  well  as  the  intended  scientific or  engineering  application.

For  example,

you  can  see  the  model  developed with  a  decision  tree  method

is  on  average  about 14%  worse  than   Fit Step Wise,

which  arguably  is  a  meaningful  difference.

If  the  difference  is  meaningful,

we  can  evaluate  how  precisely the  difference  has  been  measured

by  evaluating  how  much the  Confidence C urve  width  changes

across  the  confidence  levels.

For  any  confidence  interval  not  crossing the  default  vertical  axis,

we  have  at  least  that  level  of  confidence that  the  main  difference  is  nonzero.

For  example,

the  decision  tree  confidence curve  doesn't  cross  the  Y  axis

until  about the  99.98%  confidence  level.

We  are  nearly  99.98%  confident the  mean  difference  isn't  equal  to  zero.

In  fact,  with  this  data,

it  turns  out  that we  can  be  about  81%  confident

that   Fit Step Wise  is  at  least  as  good,

if  not  better,  than  every  method  other than  generalized  regression  Lasso.

Now  let's  consider  the  relationship between   Confidence Curves.

If  two  or  more   Confidence Curves significantly  overlap

and  the  mean  difference  of  each  is  not meaningfully  different  from  the  other.

The  data  suggest  each  method  performs about  the  same  as  the  other

with  respect  to  the  reference  model.

So  for  example,

we  see  that  on  average, the  Support  vector  Machines  model

performs  less  than  . 5% than  Bootstrap  Forest,

which  is  arguably not  a  meaningful  difference.

The  confidence  intervals  do  not  overlap until  about  the  4%  confidence  level,

which  suggests these  values  would  be  expected

if  both  methods  really  do  have about  the  same  difference  in  performance

with  respect  to  the  reference.

If  the  average  difference  in  performance

is  about  the  same for  two  confidence  curves,

but  the  confidence  intervals don't  overlap  too  much,

the  data  suggests

the  models  perform  about  the  same  as  each other  with  respect  to  the  reference  model.

However,  we  are  confident of  a  non- meaningful  difference.

This  particular  case is  rare  than  the  others

and  I  don't  have  an  example to  show  with  this  data  set.

On  the  other  hand, if  the  average  difference  in  performance

between  a  pair  of   Confidence Curves is  meaningfully  different

and  the  confidence  curves have  little  overlap,

the  data  suggests  the  models  perform

differently  from  one  another with  respect  to  the  reference.

For  example,  the  generalized  regression Lasso  model  predicts  about  13.8%

more  of  the  variation  in  the  response than  does  the  decision  tree  model.

Moreover,  the   Confidence Curves don't  overlap

until  about  the  99.9%  confidence  level,

which  suggests these  results  are  quite  unusual

if  the  methods  actually  perform  about the  same  with  respect  to  the  reference.

Finally,  if  the  average  difference in  performance  between  a  pair

of   Confidence Curves  is  meaningfully different  from  one  another

and  the  curves  have  considerable  overlap,

the  data  suggests  that

while  the  methods  perform  differently  from one  another  with  respect  to  the  reference,

it  wouldn't  be  surprising  if the  difference  is  furious.

For  example,  we  can  see  that  on  average

support  vector  machines  predicted  about 1.4%  more  of  the  variance  in  the  response

than  did  [inaudible 00:19:18]   nearest  neighbors.

However,  the  confidence  intervals  begin  to overlap  at  about  the  17%  confidence  level,

which  suggests  it  wouldn't  be  surprising

if  the  difference in  performance  between  each  method

and  the  reference  is  actually  smaller than  suggested  by  the  point  estimates.

Simultaneously,

it  wouldn't  be  surprising  if  the  actual difference  is  larger  than  measured,

or  if  the  direction of  the  difference  is  actually  reversed.

In  other  words,  the  difference in  performance  is  uncertain.

Note  that  it  isn't  possible  to  assess the  variability  and  performance

between  two  models relative  to  one  another

when  the  differences  are relative  to  a  third  model.

To  compare  the  variability  and  performance

between  two  methods relative  to  one  another,

one  of  the  two  methods must  be  the  reference  method

from  which the  differences  are  measured.

But  what  about  multiple  comparisons?

Don't  we  need  to  adjust  the  P-values

to  control  the  family wise   type 1  error  rate?

In  this  paper  about   Confidence Curves,

Daniel  Burrough  suggests  that  adjustments are  needed  in  confirmatory  studies

where  a  goal  is  prespecified, but  not  in  exploratory  studies.

This  idea  suggests  using  unadjusted P-values  for  multiple   Confidence Curves

in  an  exploratory  fashion

and  only  a  single   Confidence Curve generated  from  different  data

to  confirm  your findings  of  a  significant  difference

between  two  methods  when using  significance  testing.

That  said,  please  keep  in  mind  the  dangers of  cherry  picking,  p-hacking

when  conducting  exploratory  studies.

In  summary,  the  model  screening  platform introduced  in  JMP  Pro  16

provides  a  means to  simultaneously  compare  the  performance

of  predictive  models  created using  different  methodologies.

JMP  has  a  long  standing  goal to  provide  a  graph  with  every  statistic,

and  Confidence Curves help  to  fill  that  gap.

For  the  model  screening  platform.

You  might  naturally  expect  to  use significance  testing  to  differentiate

between  the  performance  of  various methods  being  compared.

However,  P-values  have  come under  increased  scrutiny  in  recent  years

for  obscuring the  size  of  performance  differences.

In  addition,

P-values  are  often  misinterpreted  as  the probability  the  null  hypothesis  is  true.

Instead  of  P-value  is  the  probability of  observing  a  difference

as  or  more  extreme, assuming  the  null  hypothesis  is  true.

The  probability  of  correctly  rejecting the  null  hypothesis  when  it  is  false

is  determined  by  a  power or  one  minus  beta.

I  have  argued  that  it  is  not  uncommon to  only  have  a  50%  chance

of  correctly  rejecting  the  null  hypothesis with  an  alpha  value  of  .05 .

As  an  alternative,  a  confidence  interval could  be  shown  instead  of  a  loan  P- value.

However,  the  question  would  be  left  open as  to  which  confidence  level  to  show.

Confidence Curves  address  these  concerns by  showing  all  confidence  intervals

up  to  an  arbitrarily high- level  of  confidence.

The  mean  difference  in  the  performance

is  clearly  visible  at  the  0%  confidence level,  and  that  acts  as  a  point  estimate.

All  of  the  things  being  equal, type 1  and  type  2  errors  are  equivalent.

Confidence Curves  don't  embed  a  bias towards  trading  type 1  errors  for   type 2.

Even  so,  by  default,

a  vertical  line  is  shown in  the  confidence  curve  graph

for  the  standard  null hypothesis  of  no  difference.

In  addition,

a  horizontal  line  is  shown  that delineates  the  95%  confidence  interval,

which  readily  affords  a  typical significance  testing  analysis  if  desired.

The  defaults  for  these  lines are  easily  modified,

but  different  null  hypothesis and  confidence  levels  desired.

Even  so,  given  the  rather  broad and  sometimes  emphatic  suggestion

to  replace  significance  testing  with  point estimates  and  confidence  intervals,

it  may  be  best  to  view a   Confidence Curve  as  a  point  estimate

along  with  a  nearly  comprehensive  view of  its  associated  uncertainty.

If  you  have  feedback about  the  confidence  curve's  add-in,

please  leave  a  comment on  the  JMP  community  site.

And  don't  forget to  vote  for  this  presentation

if  you  found it  interesting  or  useful.

Thank  you  for  watching  this  presentation and  I  hope  you  have  a  great  day.