Hi,  my  name  is  Ryan  Parker, and  I'm  excited  to  be  here  today

with  Clay  Barker  to  share  with  you  some new  tools  that  we've  added  to  help  you

analyze  spectral  data  with  the Functional  Data  Explorer  in  JMP Pro  17.

So  what  do  we  mean  by  spectral  data?

We  have  a  lot  of  applications

from  chemometrics  that  have motivated  a  lot  of  this  work.

But  I  would  just  start  off  by  saying

we're  really  worried  about  data that  have  sharp  peaks.

This  may  not  necessarily  be  spectral  data, but  these  are  the  type  of  data

that we've had  a  hard  time  modeling in  FE  up  to  this  point.

And  so  we  really  wanted  to  focus  on  trying

to  open  up  these  applications  and  make  it a  lot  easier  to  handle  these  sharp  peaks.

Maybe  potential  discontinuities.

Just   these  large, wavy  features  of  data.

Where  in  this  specific  example, with  spectroscopy  data,

we're  thinking  about composition  of  materials,

and  these  peaks  can  represent these  compositions,

and  we  want  to  be  able to  try  to  quantify  those.

So  another  application  is from mass spectrometry,

and  here  you  can  see these  very  sharp  peaks.

They're  all  over  the  place in  these  different  functions.

But  these  peaks  are  representing  proteins from  these  spectrums,

and  they  can  help  you,  for  example,

compare  differences  between  things  from  a  patient  with  cancer

and  patients  without  cancer to   understand  differences.

I mean, again,

it's  really  important  that  we  try to  model  these  peaks  well

so  that  we can  quantify  these  differences.

An  example  that  Clay  is  going to  show  comes  from  chromatography.

This  is  where  we're  trying  to ...

In  this  case,

we  want  to  look  at  quantifying  the  difference  between  an  olive  oil

versus  other  vegetable  oils.

And  so  the  components  of  these  things  represented  by  all  of  these  peaks,

we  need  to,  again,  try  to  model  these  well.

The  first  thing  I  want  to  cover  are four  new  tools  to  preprocess  your  data,

spectral  data  before  you get  to  the  modeling  stage.

The  first  one  is the  Standard  Normal  Variate.

So  with  the  Standard  Normal  Variate, we're  thinking  about

standardizing  each  function   by  their  individual  mean  and  variance.

So  we're  going  to  take  every  function   one  at  a  time,

subtract  off  the  mean,  divide  by  the  variance,

so  that  they  all  have   mean  zero  and  variance  one.

This  is  an  alternative  to   the  existing  tool  we  have  standardized,

which  is  just looking  at  a  global  mean  in  variance

so  that  the  data themselves  have  been  scaled,

but  certain  aspects, like  means,  are  still  there,

whereas  with  the  Standard  Normal  Variate, we  want  to  remove  that  for  every  function.

The  next  tool's   Multiplicative S catter  Correction.

It  is  similar  to  Standard  Normal  Variate,

the  results  end  up being  the  same,  similar.

But  in  this  case,  we're  thinking  about data  where  we  have  light  scatter.

So  some  of  these  spectral  data  come  where  we  have  to  worry  about

our  light  scatter  from  all  the  individual  functions  being  different

from  a  reference function  that  we're  interested  in.

Usually  this  is  the  mean.

So  what  we'll  do  is  we   will  set  a  simple  model

between  the  individual functions  to  this  mean  function.

You know, we will get coefficients

that  we  can  subtract   off  this  mean  feature,

divide  by  the  slope  feature,

get  us to  that  similar  standardizing  the  data,

and  in  this  case,  focused  on  this  light  scatter.

Okay,  so  at  this  point,

we're  thinking  about what  if  we  have  noise  in  our  data?

What  if  we  need  to  smooth  it?

So  the  models  that  we  want  to  fit, for  spectral  data,

these  wavelets, they  don't  smooth  the  data  for  you.

So  if  you  have  noisy  data,  you  really want  to  try  to  handle  that  first,

and  that's  where  the   Savitzky-Golay  Filter  comes  in.

What  this  is  going  to  do  is  fit  n- degree  polynomials

over  a specified  bandwidth

to  try  to  find  the  best  model  that  will  smooth  your  data.

So  we  search  over  a  grid  for  you,

pick  the  best  one, and  then  present  the  results  to  you.

And  I  do  want  to  note  that  the  data  are required  to  be  on  a  regular  grid,

but  if  you  don't  have  one, FDE  will  create  one  for  you.

We  have  a  reduce  option

that  you  can  use  if  you  want   some  finer  control  over  this,

but  by  default,

we  are  going  to  look at  the  longest  function,

choose  that  as  our  number  of  grid  points,

and  create  a  regular grid  from  that  for  you.

But  the  nice  thing  about   the  S avitzky-Golay Filter

is  because  of  the  construction   with  these  polynomials,

we  have  easy  access to  the  first  or  second  derivative.

Even  if  you  don't  have  spectral  data  and you  want  to  access derivative functions

this  will  be your  pathway  to  do  that.

And  if  you  do  request,  say, the  second  derivative,

our  search  gets   constrained  to  only  polynomials

that will allow us to  give  you a  second  derivative,  for  example.

But  this  would  be  the  way  to  access that,

even  if  you  weren't  even worried  about  smoothing.

You  can  now  get  to  derivatives.

The  last   preprocessing  tool  I'll  cover is  Baseline  Correction.

So  in  Baseline  Correction,

you  are  worried  about  having   some  feature  of  your  data

that  you  would  just  consider  a  baseline  that  you  want  to  remove

before  you  model  your  data.

So  the  idea  here  is  we  want to  fit  a  baseline  model.

We  have  linear,  quadratic,  exponential options for you,

so  we  want  to  fit  this  to  our  data   and  then  subtract  it  off.

But  we  know  there  are  important  features,

typically  these  peaks,

so  we  want  to  not  use  those  parts  of  the  data

when  we  actually fit  these  baseline  models.

So  we  have  the  option  here for  correction  region.

I  think  for  the  most  part  you would  likely  use  entire  function.

So  what  this  just  means  is,  what  part  are  we  going  to  subtract  off?

So  if  you  select  within  regions

only things  within  these  blue  lines are  going  to  be  subtracted.

But  I've  already  added  four  here.

Every  time  you  click  add   on  baseline regions,

you're  going  to  get  a  pair  of  these  blue  lines  and  you  can  drag  them around

to  the  important  parts  of  your  data, and  what  this  will  do  is,

when  you  try  to  fit,  say, a  linear  baseline  model,

it's  going  to  ignore  the  data  points that  are  within  these  two  blue  lines.

So,  function  one,  we  set  a  linear  model,

but  we  exclude  all  these  sharp  peaks that  we  really  want,

that  we're  interested  in.

And  so  then  we  take  the  result   from  this linear model

and  subtract  it off  from  the  whole  function.

The  alternative  to  that is  an  Anchor  Point,

and  that's  if  you  say, I  really  would  like

for this specific point  to  be  included in  the  baseline  model.

Usually  this  is  if  you  have  smaller  data

and  you  know,  okay, I  want  these  few  points.

These  are  key. These  represent  the  baseline.

If  I  were  able  to  fit, you know, say a  quadratic model to these points,

that's  what  I  want  to  subtract  off.

So  it's  an  alternative.

When  you  click  those,  they'll  show  up as  red  as  an  alternative  to  this  blue.

But  this  will  allow  you to  correct  your data

remove  the  baseline before  proceeding.

So  that  gets  us  to  how  we  model   spectral  data  now  in  JMP Pro  17,

and  we're  using  wavelets.

The  nice  thing  about  wavelets  is  we  have a  variety  of  options  to  choose  from.

So  these  graphs  represent   what  are  called  mother  wavelets

and  they  are  used  to  construct

the  basis  that  we  use, that  models  the  data.

So  the  simplest  is  this Haar  wavelet, which  is  really  just  step  functions,

maybe  hard  to  see  that  here, but  these  are  just  step  functions.

But  this  biorthogonal, it  has  a  lot  of  little  jumps

and  you  can  start  to  imagine,

okay,  I  can  see  why  these   make  it  a  lot  easier

to  capture  peaks  in  my  data

other  than  the  Haar  wavelet.

All  these  have  parameters  that  are   changing  the  shape  and  the  size  of  these,

so  I've  just  selected  a  couple  here to  just  show  you  the  differences.

But  you  can  really  see  where,

okay,  if  I  put  a  lot  of  these  together,

I  can  understand  why  this  is

probably  a  lot  better  to  model  all these  peaks  of  my  data.

And so here's  an  example  to  illustrate  that

with  one  of  our  new  sample  data,

a NMR  design  of  experiments.

So  this  is  just  from  one  function where  let's  start  with  B-Splines.

This  is  sort  of  the  go  to  for   most  data  place  to  start  in  FDE.

But  we  can  see  that  it's   really  having  a  hard  time

picking  up  on  these  peaks.

Now, there are,  we  have  provided  you  tools

to  change  where  knots are at in  these  beastline  models.

So  you  could  do  some  customization.

Probably  fit  this  a  lot better  than  the  default.

But  the  idea  is  that  now  you've  had  to  go  and  move  things around

and  maybe  it  works  for  some  functions, but  not  others,

and  you  need  a  more  automated  way.

So  that's  one  alternative to  that  is  P-Sp lines.

That  is  doing  that a  little  bit  for  you,

but  it's  still  not  quite  capturing the  peaks  maybe  as  well  as  wavelets.

It's  probably  doing  the  best   for  these data.

relative  to  wavelets and  almost  a  model -free  approach

where  we  model  the  data  just  directly on  our  shape  components,

this  direct  functional  PCA.

It's  maybe  a  bridge   between  P-Splines  and  B-Splines

where  it's  not  quite  as  good  as  P -Splines   but  it's  better  than  B -Splines

but  this  is  just a  quick  example  to   highlight

how  wavelets  can  really be  a  lot  quicker  and  powerful.

What  are  we  doing  in  FDE?

We  construct  a  variety  of  wavelet types

and  their  associated parameters  and  fit  them  for  you.

So  similar  to  the  S avitzky-Golay Filter,

we  do  require  that  the  data are  on  a  regular  grid.

And  good  news,  we'll  create  one  for  you.

But  of  course  you  can  go  to  reduce  the  control  that  if  you  would  like.

But  now  the  nice  thing  about   once  data are   on  the  grid,

we  can  use a  transformation  that's  super  fast.

So  P -Splines  would  also,  for  these  data,

be  what  you  would  really  want to  have  to  use,

but they can take a longtime to fit,

especially  if  you  have   a  lot  of  functions

and  you  have  a  lot of  data  points  per  function.

But  our  wavelet  models  are going  to  essentially  last

so  all  of  the  different  basis  functions

that  construct  this  particular   wavelet  type  with  parameter

and  what  that's  going to  allow  us  to  do  is  just  really  force

a  lot  of  those  coefficients  that  don't really  mean  anything  to  zero

to  help  create  a  sparse representation  for  the  model.

So  those  five  different  wavelet  types that  I  showed  before,

those  are  available, and  once  we  fit  them  all,

we're  going  to  construct   model  selection  criteria

and  choose  the  best  for  you  by  default.

You  can  click  through these  as  options  to  see  other  fits.

And  a  lot  of  times  these  first  few   are  going  to  look  very  similar

and  it's  really  just  a  matter  of,

there  are  certain  applications   where  they  know,

"Okay, I  really  want  this  wavelet type,"

so  they'll  pick  the  best one  of  that  type  in  the  end to use.

So  the  nice  thing  about  these  models is  they  happen  on  a  resolution.

They're  modeling  different resolutions  of  the  data.

So  we  have  these  coefficient  plots  where at  the  top  they're  showing

low-frequency,  larger  scale  trends, like  an  overall  mean,

and  as  you  go  down  in  the...

Or  I  guess  you go  up  in  the  resolutions,

but  down  in  the  plot,

you're  going  to  look at  high -frequency  items,

and  these  are  going  to  be  things that  are  happening  on  very  short  scales,

so  you  can  see  where  it's  picking up  a  lot  of  different  features.

In  this  case, it's taking...

A  lot  of  these  are  zero for  the  highest  resolutions.

So  it's  picking  up  some  scales  that  are at  the  very  end  of  this  function.

It's  picking  up  some of  these  differences  here.

But  this  gives  you  a  sense  of  kind of  where  things  are  happening at

both  location  and  then   is  that   high-frequency  or   low-frequency

parts  of  the  data.

So  the  last  thing  we've  added to  complete  the  wavelet  models

that's  a  little  bit  different  from  what we  have  now  is  called  Wavelets  DOE.

So  if  you've  used  FDE  before,

you've   likely  tried functional  design  of  experiments

and  that  is  going  to  allow  you  to  take   functional  principle  component scores,

connect  design  of  experience factors  to  the  shape  of  your  functions.

But  now, wavelet  models  in  particular,

they're  coefficients,  because  they  do represent  resolutions  and  locations,

these  can  be  more  interpretable and  they  have  more  direct  impact

to  understanding  what's happening  in  the  data,

that  may be a  functional  principle  component

isn't  as  easy  to  connect  with.

We  have  this  energy  function and  it's   standardized  to  show  you,

"Okay,  this  resolution  at  3.5,"

just   representing  more  on  this  end point.

That's going to have...

That's  where  the  most  differences are  in  all  of  our  functions,

and  it's  representing  about  12%.

So  you  can  scroll  down.

We  go  up  to  where  we  get  90% of  this  energy,

which  is  just  the  squared   coefficient  values

that  we  just  standardized  here.

But   energy  is  just   how  big  are these coefficients

relative  to  everything  else?

But  this, similar  to  Functional  DOE,

you  can  change the  factors,  see  how  the  shape  changes,

and  we  have  cases where  both  Wavelets  DOE

and  Functional  DOE  work well.

Sometimes  Wavelets DOE  just  get the  structure  better.

Doesn't  allow  for  some   maybe  negative  points

that  Functional  DOE might  allow  in  this  example.

So  it's  just  they're  both  there, they're  both  fast.

I mean, you  can  use  both  of  them  to  try  to analyze  the  results  of  wavelet  models.

But  that's  my  quick  overview.

So  now  I  just  want  to  turn  it  over  to  Clay to  show  you  some  examples

of using wavelet  models  with some  example  data  sets.

Thanks,  Ryan.

So,  as  Ryan  showed  earlier,  we  found  an  example

where  folks  were  trying   to  use  chemometrics  methods

to  classify different  vegetable  oils.

So  I've  got  the  data  set  opened  up  here.

Here  we  have  our each  row  of  the  data  set  is  a  function,

so  each  row  in  the  data  set represents  a  particular  oil,

and  as  we  go  across  the  table, that's  the  chromatogram.

So  I've  opened  this  up  in FDE  just  to  save  a  few  minutes.

The  wavelet  fitting  is  really  fast,

but  I  figured  we'd  just  go  ahead and  start  with  the  fit  open.

So  here's  what  our  data  set  looks  like.

You  can  see those  red  curves  are  olive  oils.

The  green  curves  are  not  olive  oils.

So  we  can  see  there's   definitely  some  differences

between  the  two  different kinds  of  oils  and  their  chromatograms.

So,  as  Ryan  said,  we  just  go   to  the  red triangle menu

and  we  ask  for  wavelets  and it  will  pick  the  best  wavelet  for  you.

But  like  I  said, I've  already  done  that,

so  we  can  scroll  down  and  look  at  the  fit.

Here  we  see  the  best  wavelet  that  we found  was  called  the  Symlet 20,

and  we've  got  graphs  of  each fit  here  summarized  for  you.

As  you  can  see,  the  wavelets have  fit  these  data  really  well.

But  in  this  case,

we're  not  terribly  interested in  fitting  the  individual  fits.

We  want  to  see  if  we  can  use  these individual  chromatograms

to  predict  whether  or  not  an  oil   is  an  olive  oil  or  not  an  olive  oil.

So  what  we  can  do  is,

we  can  save  out  these wavelet  coefficients,

which  we've  gotten  a  big  table  here, and  there's  thousands  of  them.

In  fact,  there's  one  for  every point  in  our  function.

so  here  we've  got  4,000 points  in  each  function.

This  table  is  pretty  huge. There's  4,000  wavelet  coefficients.

But  as  Ryan  was  saying,  you  can  see that  we've  zeroed  some  of  them  out.

So  these  wavelet  coefficients drop  out  of  the  function.

So  that's  how  we  get  smoothing.

We  fit  our  data  really  well,

but  zeroing  out  some  of  those  coefficients is  what  smooths  the  function out.

So  how  can  we  use  these  values  to  predict whether  or  not  we  have  an  olive  oil?

Well,  you  can  come  here  to  the  function summaries  and  ask  for  save  summaries.

So  what  it's  done  is  it  saves  out the  functional  principal  components.

But  here  at  the  end  of  the  table,  it  also saves  out  these  wavelet  coefficients.

So  these  are  the  exact  same  values

that we saw  in  that  wavelet  coefficient table  in  the  platform.

So  let  me  close  this  one.

I've  got  my  own  queued  up  just  so  that  I don't  have  anything  unexpected  happen.

So  here's  my  version  of  that  table.

And  what  we  want  to  do  is  we  want  to  use

all  of  these  wavelet  coefficients to  predict  whether

the  curve  is  from  an  olive  oil or  from  a  different  type  of  oil.

So  what  I'm  going  to  do  is,

I'm  going  to  launch the  generalized  regression  platform,

and  if  you've  ever  used  that  before,

it's  the  place  we  go  to  build  linear models  and  generalize  linear  models

using a  variety  of  different variable  selection  techniques.

So  here  my  response  is  type.

I  want  to  predict   what  type  of  oil  we're looking at

and  I  want  to  predict  it  using all  of  those  wavelet  coefficients.

So  I  press  run.

In  this  case,  I'm  going  to  use  the  Elastic Net

because  that  happens  to  be my  favorite  variable  selection  method.

And  I'm  going  to  press  go.

So  really  quickly, we  took  all  those  wavelet  coefficients

and  we  have  found  the  ones  that  really do  a  good  job  of  differentiating

between  olive  oils and  non -olive  oils.

So  in  fact,  if  we  look  at  the  confusion  matrix,

which  is,  this  is  a  way  to  look at  how  often  we  predict  properly,  right?

So  for  all  49  other  olive  oils,

we  correctly  identified  those   as  not  olive  oils.

And  for  all  71  olive  oils,  we  correctly identified  those  as  olive  oils.

So  we  actually  predicted  these  perfectly

and  we  only  needed  a  pretty  small  subset of  those  wavelet  coefficients.

So  I  didn't  count, but  that  looks  like  about  a  dozen.

So  we  started  with  thousands  of  wavelet coefficients  and  we  boiled  it  down

to just the  12  or  so  that  were  useful for  predicting  our  response.

So  what  I  think  is  really  cool  is,

we  can  interpret  these  wavelet coefficients  to  an  extent,  right?

So  this  coefficient  here  is resolution  two  at  location  3001.

So  that  tells  us  there's  something  going on  in  that  part  of  the  curve

that helps us differentiate between  olive  oils  and  not  olive   oils.

So  what  I've  done  is,

I've  also  created a  graph  of  our  data  using...

Well,  you'll  see.

So  what  I've  done  is  here the  blue  curve  is  the  olive  oils.

The  red  curve  is  the  non -olive  oils,

and  this  is  the  mean  chromatogram.

So  averaging  over  all  of  our  samples.

And  these  dashed  lines  are  the  locations

where  the  wavelet coefficients  are  nonzero.

So  these  are  the  ones  that  are  useful for  discriminating  between  oils.

And  as  you  can  see,  some  of  these   non-zero coefficients  line  up  with  peaks

and  the  data  that  really tend  to  make  sense,  right?

So, here is,

here's  one  of  the  non -zero  coefficients, and  you  can  tell  it's  right  at  a  peak

where  olive  oil  is  peaking, but   non-olive  oils  are  not,  right?

So  that  may  be  meaningful  to  someone

that  studies  chromatography and  olive  oils  in  particular.

But  so  we  like  this  example  because

it's  a  really  good  example  of  how  wavelets fit  these  chromatograms  really  well.

And  then  we  can  use  the  wavelet coefficients  to  do  something  else,  right?

So  not  only  have  we   fit  the  curves  really well,

but  then  we've  taken   the  information  from  those  curves

and  we've  done  a  really  good  job  of  discriminating  between  different  oils.

And  so  I've  got  one  more  example.

Ryan  and  I  are  both big  fans  of  Disney  World,

so  this  is  not  a  chromatogram.

This  is  not  spectroscopy.

But  instead  we  found  a  data  set  that  looks at  wait  times  at  Disney  World,

so  we  downloaded  a  subset  of  wait  times

for  a  ride  called  Disney's the  Seven  Dwarfs,  Mind  Train.

And  if  you've  ever  been  to  Disney  World,

you  know  it's  a  really  cool  roller coaster right  there  in  fantasyland.

But  it  also  tends  to  have really  long  wait  times, right?

You  spend  a  lot  of  time  waiting  your  turn.

So  we  wanted  to  see  if  we  could  use wavelets  to  analyze  these  data

and then use  the  Wavelet  DOE  function   to  see  if  we  can  figure  out

if  there's  days  of  the  weeks   or  months  or  years

where  wait  times are  particularly  high  or  low.

So  we  can  launch  FDE.

Here you  can  see, we've got

each  day  in  our  data  set, we  have  the  wait  times

from the time that  the  park  opens,  here,

to  the   time that  the  park  closes  over  here.

And  to  make  this  demo  a  little  bit  easier,

we've  finessed  the  data  set to  clean  it  up  some.

So  this  is  not  exactly  the  data,

but  I  think  some  of  the  trends that  we're  going  to  see  are  still  true.

So  what  I'm  going  to  do  is  I'm  going to  ask  for  wavelets,

and  it'll  take  a  second  to  run, but  not  too  long.

So  now  we've  found  that  a  different basis  function  is  the  best.

It's  the Daubechies 20

and  I  apologize  if  I  didn't pronounce  that  right.

I've  been  avoiding  pronouncing  that  word  in  public,

but  now  that's  not  the  case  anymore.

So  we've  found  that's  our  favorite  wavelet and  what  we're  going  to  do

is  we're  going  to  go   to  the  Wavelet  DOE  analysis,

and  it's  going  to  use  these  supplementary  variables

that  we've  specified  day of  the  week,  year  and  month

to see if we can  find  trends  in  our  curves  using  those  variables.

So  we'll  ask  for  Wavelets  DOE,

and what's  happening  in  the  background  is we're  modeling  those  wavelet  coefficients

using  the  generalized  regression  platform, so  that's  happening  behind  the  scenes,

and  then  it  will  put  it  all together  in  a  Profiler  for  us.

So  here  we've  got, you know,  this  is  our  time  of  day  variable.

We  can  see  that  in  the  morning.

The  wait  times  sort  of  start, you know, around  an  hour.

It  gets  longer  throughout  the  day, you know,

peaking  at  about  80  minutes, almost  an  hour  and  a  half  wait.

And  then,  as  you  would  expect,

as  the  day  goes  on   and  kids  get  tired  and  go  to  bed,

wait  times  get  a  little bit  shorter  until  the  end  of  the  day.

Now,  what  we  thought  was  interesting

is looking  at  some  of  these  effects, like  year  and  month.

So  we  can  see  in  2015,  the  wait  times  sort of  gradually  went  up,  right,  until  2020.

And  then  what  happened  in  2020? They  increased  in  February,

and then, shockingly,  they  dropped  quite a  bit  in  March  and  April.

And  I  think  we  all  know  why that  might  have  happened  in  2020.

Because  of  COVID,  fewer  people were  going  to  Disney  World.

In  fact,  it  was  shut  down  for  a  while.

So  you  can  very  clearly  see  a  COVID  effect  on  Disney  World  wait  times

really  quickly  using  Wavelet  DOE.

One  of  the  other  things  that's  interesting

is  we  can  look  at  what time  of  year  is  best  to  go.

It  looks  like  wait  times   tend  to  be  lower  in  September,

and  since  Disney  World   is  in  Florida, you know,

that's  peak  hurricane  season, and  kids  don't  tend  to  be  out  of  school.

So  it's  really  cool  to  see  that  our model  picked  that  up  pretty  easily, right?

So, but  don't  start  going to  Disney  World  in  September.

That's  our  time. We  don't  want  it  getting  crowded.

But  yeah,  so  with  just  a  few  clicks,   we  were  able

to  learn  quite  a  few things  about  wait  times,

Seven  Dwarfs  Mine  Train  at  Disney.

But  we  really  wanted  to  highlight

that these methods were  focused  on  chromatography  and  spectrometry,

but  there's  a  lot  of  applications  where  you  can  use  Wavelets,

and  I  think  that's  all  we  have.

So  thank  you. And thank  you,  Ryan.