Hello. Thank  you  for  joining  us  today

for  our  presentation  on,

"Measuring  Change in  Medication Assisted  Treatment

for  Participants  in  project  ECHO  Using Administrative  Healthcare  Claims"

We  are  excited  to  share  how  we  use  JMP as  a  key  tool  in  our  analytic  work.

Next  slide,  please.

My  name  is  Erica  Plante   and  I  am  a  senior  scientific  data  analyst

at  the  Institute for  Health  Policy  and  Practice

at  the  University  of  New  Hampshire.

I  am  joined  by  Dr.  Michelle  Capozzoli, Senior  Lecturer

in  the  Department  of  Mathematics   and  Statistics,

also  at  the  University  of  New  Hampshire.

Neither  Michelle  nor  I  have conflicts  of  interest  to  disclose.

Next  slide,  please.

Before  we  describe  our  work,

I  would  like  to  provide  a  brief overview  on  Project  ECHO.

Project  ECHO  was  founded  in  2003  by  Dr.  Sanjeev  Arora

at  the  University  of  New  Mexico.

Dr.  Arora  is  a  Physician  specializing in  Gastroenterology  and  Hepatology.

He  was  seeing  patients with  Hepatitis  C  die  at  alarming  rates

because  they  could  not  access  care

for  this  treatable  disease in  a  timely  manner.

He  sought  to  bring  providers  together to  form  a  community  of  practice

where  doctors  and  other  specialists can  learn  from  each  other.

ECHO  is  an  "all  teach,  all  learn"  model,

and  the  sessions  are  often  centered  around a  key  issue  or  condition.

The  University  of  New  Hampshire  launched its  project  ECHO  hub  in  2018

and  has  since  produced  a  number of  ECHO  programs,

including  the  Partnership  for  Academic Clinical  Telepractice,  Medications

for  Addiction  Treatment,  or  PACT-MAT.

Next  slide,  please.

The  primary  goal of  PACT-MAT  ECHO  is

to  increase  the  number of  nurse  practitioner  students

in  graduate  and  postgraduate  programs who  receive  waiver  training,

apply  for  the  waiver,  and  subsequently  prescribe  MAT.

Secondarily,  the  project  seeks  to  increase provider  self- efficacy

in  managing  patients with  Substance- use  Disorder.

The  program  developed a  learning  community

that  enabled  a  culture that  understood  addiction

as  a  chronic  disease and  was  prepared  to  address

the  range  of  issues  that  emerged  during the  process  of  treatment.

Specifically,  this  program  focused on  all  participants  becoming  proficient

and  culturally  competent in  prescribing  and  treating  SUD,

as  well  as  enhancing  capacity and  qualities  of  services  available

to  patients  in  their  communities through  their  providers.

Next  slide,  please.

After  the  completion of  the  first   PACT-MAT  session,

the  team  was  interested in  answering  some  questions

about  the  PACT-MAT  ECHO through  Claims  Data  Analysis.

But  here's  some  core  information about  the  analytic  project.

The  analytic  period  of  interest was  from  2018  through  June  2020.

This  was  to  capture  data prior to and after  the  first   PACT-MAT  ECHO  session.

The  project  was  funded

by  the  Substance  Abuse  and  Mental  Health Services  Administration ( SAMHSA),

as  part  of  150K 3-year  grant.

The  project's  principal investigator  was  Dr.  Marcy  Doyle.

We  were  wanting  to  ask  a  few  questions about  the  actual  ECHO  program

and  how  we  were  able  to  see if  provider  practices  had  changed

after  actually  participating  in  the  ECHO.

And  the  Center for  Health  Analytics  of  Informatics

at  the  Institute  for  Health  Policy and  Practice  at  UNH.

We're  fortunate  enough  to  have  access to  healthcare,  administrative  claims,

and  enrollment  data  for  commercial and  New  Hampshire  Medicaid  policies.

Therefore,  the  CHA  team  was  brought into  the  research  project

to  collect  and  aggregate  the  data,

and  UNH's  Department  of  Mathematics and  Statistics  was  brought  in

to  build  models  and  perform  analysis.

Next  slide,  please.

And  these  are  the  questions  that  we  asked as  our  core  research  questions.

Did  the  PACT-MAT  ECHO  Series  have  an  impact

on  participants'  MAT prescribing  practices?

And  can  we  successfully  perform a  case/ control  study  on  providers

using  administrative  claims  data?

Next  slide,  please.

When  collecting  health  care  claims,

we  included  all  members,   ages  zero  to  64  who  had  medical

and  pharmacy  enrollment in  the  month  of  interest.

PACT-MAT  participants  self- reported their  name,  titles,  NPI,

organization's  name  and  address, and  their  waiver  status.

We  cross- referenced  that  data against  CMS's  National  Plan

and  Provider  Enumeration  System, also  known  as  the  NPPES  Registry.

In  the  cases  of  a  mismatch   between  the  self- reported  data

and  the  NPPES  registry, the  self  reported  data  was  considered

the  most  up  to  date  and  was used  for  the  analysis.

Information  on  our  control  group  was sourced  only  from  the  NPPES.

Next  slide,  please.

And  claims  reflect  if  one or  more  service  lines  included

an  MAT  procedure  code  or  drug  code,

they  were  also  flagged

if an  Opioid-Related  Disorder diagnosis  code  was  found.

Medical  providers  were  selected as  having  billed  for  MAT  if  at  least

one  service  line  included  their  NPI or  one  of  the  MAT  CPT  codes.

Prescribing  providers  were  selected as  having  prescribed  MAT  if at least

one  pharmacy  service  line  included their  NPI  as  the  prescriber

and  at  least  one  of  the  MAT  NDC  codes.

The  case  and  control  populations  each  had two  pairs  of   datasets.

Next  slide,  please.

One  pair  for  each  insurance  type, commercial  or  New  Hampshire  Medicaid.

The  first  date  included  there the  providers  NPI  and  information,

as  well  as  total  aggregates by  month  of  all  providers,

patients  and  claims.

Patients  with  Opioid- Use  Disorder  (OUD),

patients  with  any  medication assisted  treatment,

and  patients with  both  OUD  and  MAT.

The  second  dataset  provided the  same   dataset  as  the  first  data set...

Same  data  as  the  first  data set, with  the aggregation

at  the  member  demographic  level,

such  as  age,  category,  county,  and  sex.

No  identifiable  member  data  was  applied to  the  statisticians.

Now  I'm  going  to  pass  the  presentation to  my  colleague,  Dr.  Capozzoli.

Thank  you,  Erica.

Once  we  obtain  the  data.

The  data  was  actually analyzed  by  Rebecca  L. Park.

She  was  a  UNH  master's  student under  the  supervision  of  myself

and  Dr.  Philip  Ramsey.

We  received  the  three  data sets.

One  was  the  practitioners  demographics such  as  name,

National  Provider  Identifier, title,  practice  address.

The  other  two  were  the  claims   datasets.

One  for  the  Medicaid and  one  for  the  commercial.

So  the  original  data  was extremely  large

for  both  the  commercial   and  for  Medicaid

for  each  practitioner,  it  tracked   each  of  their  MAT  patients'  history

over  the  study  period.

So  the  original  thought  was  try  to...

Use  their  pattern  of  behavior of  their  patients  over  time.

Quite  quickly,  it  became  apparent  that this  approach  was  a  little  bit  problematic

and  also  we  ran  into some  privacy  laws  in  trying  to  make  sure

that  all  identifying markers  were  not  available.

So  what  we  did   was  we  honed  in

on  several  of  the  variables for  the  demographics.

So  we  honed  in   on  the  National  Provider  Identifier,

their  title, and  the  city  of  their  practice.

And  then  from  the  claims  data, we  focused  in  on  the  month  and  year.

So  this  tracks  what  month  and  year

that  we  were  looking  at.

The  phase  of  the  program  was  the  Pre,

so  this  is  Pre  before  ECHO,

Ongoing  is  during  ECHO, Post is  obviously  after  ECHO.

So  then  we  aggregated  the  data and  so  instead  of  looking  at

every  single  visit, what  we  looked  at  were  months

and  we  looked  at  patient  totals.

So  the  first,  we  obviously  looked  at was  the  number  of  patients  total

for  that  practitioner  during  that  month,

the  number  of  patients with  Opioid-U se  Disorder,

the  number  of  patients  who  had  any  MAT,

the  number  of  patients with  OUD  who  had  any  MAT.

Further,  when  looking  at  the  Medicaid

versus  the  commercial  care, during  the  exploratory  analysis,

it  became  apparent  that  we  were  going to  need  to  focus  in  on  the  Medicaid  data

due  to  the  low  patient  numbers in  the  commercial  care.

So  here  for  example, when  we  were  looking  at  it,

it  became  apparent;

so  on  average  they  had one  patient  per  month  with  any MAT.

And  so  what  we  were  doing  was   there's  a  lot  of  sparse  data  here

and  it  was  just  not  conducive to  trying  to  fit  models.

So  we  focused  in  on  the  Medicaid  data.

Further,  we  initially  had  20  providers,

which  we  had  to reduce  down  to  nine  providers.

And  the  reason  being  is that  some  of  the  providers

had  too  many  months  of  missing  data.

Some  of  them, we  had  to  eliminate

because  the  majority  of  the  months, they  didn't  have  any  patients  who  had  MAT.

And  then  further, as  we  started  to  fit  the  models,

it  became  apparent  that  we  also  needed a  minimum  of  10  total  patients  per  month.

So  we  did  end  up with  a  very  smaller  sample  size

than  we  had  originally  thought we  would  have.

The  other  piece, as  we  were  exploring  the  data,

using  some  of  the  tools that  John  provides,

we  noticed  that  the  nine  practitioners  had on  average  the  number  of  patients

or  number  of  total  patients  differed.

So  for  example, they  could  have,

I  think  it  was  between  a  total  range of  one  to  161  patients.

So  between  that and  then  looking  at  the  trend  over  time,

so  we  looked  at the  average  number  of  patients  total

and  as  you  notice  this, the  blue  line,

you  notice  that  just  in  general there's  an  increase  of  patients  over  time.

We  also  looked  at the  average  number  of  patients

with  the  Opioid-Use D isorder

and  those  who  had  any  MAT, and  those  who  were  diagnosed

with  OUD  and  had  any  MAT.

And  if  you  notice,  all  four  have a  similar  trend  of  an  increasing  trend.

So  to  combat  that, because  what  we  want  to  know  is,

are the  practitioners  increasing their  prescribing?

Not  that  they  are  having more  in patients  over  time,

we  normalized  the  data by  creating  a  new  variable.

So  what  we  created  was the  proportion  of  patients

who  had  any  MAT  in  comparison to  the  total  number  of  patients.

And  the  reason  that  we  did  use the  patients  who  had  any MAT

is due  to  the  fact  that  a  diagnosis of  an  OUD  does  have  a  certain  stigmatism.

And  so  we  felt  that  it  would  be  better to  look  at  any  MAT.

So  Analysis  Considerations.

From  the  beginning  we  knew  that  we  had

a  small  sample  size of  practitioners,  only  nine.

The  other  thing that  became  apparent

through  a  lot  of the  graphical  representations  of  the  data

was  that  we  had  a  lot  of  noise.

And  so  in  taking   this  into  consideration,

we  decided  to  attempt  several  different  approaches.

The  first  approach  was  basically, your  means  comparison:

ANOVA and Matched  pairs.

Then  we  thought   about  bringing  in  that  time  variable.

And  so  we  did  look  at  them in  several  different  ways,

from  just  simple   Linear Regression, from  the  beginning.

We  did  consider Structural  Equation  Models.

Dr.  Laura  Castro- Shilo  from  JMP  had  come  to  one  of  Dr.  Ramsey's classes

and  given  a talk  on  these  models.

So  originally,  when...

We  were  looking  at  the  data, we  thought  that  these  models

might  be  appropriate.

But  it  became  apparent  quickly because  of  the  difficulties  that  we  had

that  they  just   were  not  working  for  us.

So  the  next,  we  worked with   Segmented Regression.

That  was  chosen

because  in  some  previous  work with  claims  data ,

it  was  brought  up  that  maybe the   Segmented Regression  would  work  well

because  we  had  data that  was  looking  at  pre  and  post.

So  we  thought  that with  our  pre, ongoing, and post,

that  maybe  the   Segmented Regression would  be  appropriate.

We  also  looked  at  Exponential  Regression and  Logistic  Regression.

We  also  looked at  their  Generalized  Regression  Models

and  including  just  the  regular and  then  zero- inflated  Poisson,

Binomial, and  Negative  Binomial  Models.

And  the  reason  why  we  decided to  look  at  the  Poisson,

and  Binomial  and  Negative  Binomial  is

the  data  is  inherent  that  it  has  counts and  so  we  thought  that  maybe

these  might  be  appropriate.

And  so  what  I'm  going  to  focus  on today  are  the  Means  Comparison,

the   Segmented Regression   and  the  zero- inflated  Poisson

and  this  is  going  to  be for  both  just  looking  at  the  ECHO  group

and  then  the  Matched  Pairs  comparing the  control  group  to  the  ECHO  group.

So  the  first  analysis  ignores the  time  variable

and  we're  just  looking  at  averages.

What's  the  average  proportion  of  patients in  your  pre,  ongoing,  and  post.

And  just  from  the  means, it  is  quickly  apparent  and  from  the  graph

that  our  pre  phase  is  definitely lower  than  the  ongoing  and post.

We  also  tested for the...

Because  we  have  small  sample with  practitioners,

we  did  look  at  the  variance and  we  noted  that  we  did  have  an  issue

with  non- equal  variance and  so  we  did  use  Welch's  test

instead  of  the  traditional  ANOVA

to  conduct  to  see  if  we  had  differences between  the  phases  and  obviously  we  do.

And  then  to  determine  statistically which  ones  were  different,

we  did  look  at   the  All  Pairs  Tukey- Kramer  Test.

And  it  did  confirm  that  our  pre  phase  was different  than  the  ongoing  and  post,

but  the  ongoing and  post  were  very  similar.

And  so  this  gave  us  some...

This  obviously  is  indicating that  we  do  have  some  differences.

The  ECHO  program  is  making  a  difference.

So  this   Segmented Regression, as  I  noted  before,

was  suggested  because  we  do  have the  three  phases:  pre,  ongoing,  and  post.

And  we  were  able...

Dr.  Ramsey  had  suggested that  we  use  a  script

that  is  done  by  David  Burnham, by  Pega  Analytics.

If  you  are  interested  in  this  script, the  link  is  here  on  the  website,

it  gives  you  the  code   and  it  also  gives  you

a  very  detailed  description   from  line  to  line,

what  he  is  doing  in  this  code.

And  so  what's  happening  here, when  we  ran  the  script,

We  were  able  to  fit  separate  regressions to  the  three  different  phases

as  well  as  get  the  fit,

an   R², and  test the  significance  of  the  slope.

And  so  again,  we're  seeing this  pattern  that  we  saw  with  the  ANOVA.

You  do  see  that  the  pre  is  definitely lower  than  the  ongoing  and post.

Unfortunately,  the  slopes for  all  of  the  three  phases

were  not  significant.

You  can  also  see  that  we  do  have a  significant  amount  of  variability.

So  next, what  we  considered was  the  fact  that  we  had  two  types...

Or  two  categories  of  practitioners;

we  had  those  who  were nurse  practitioners,  physician  assistants;

and  then  we  had  the  physicians.

Even  though  they  were  small  sample  sizes.

We  decided  to  see, "Okay,  can  we  see  some  kind  of  a  signal?"

"What  is  going  on  here?"

And  when  we  looked at  the  nurse  practitioners  again,

you  see  that  behavior  of  the  pre  being lower  than  the  ongoing  and  the  post.

And  what  we  noted  is  that for  the  pre  phase,

we  are  seeing  a  little  bit of  a  significance  even  though  it  is  0.07,

it  is  saying  that  there  is  a  signal  here and  we  do  have  something  going  on.

Unfortunately  for  the  ongoing  and  post,

we're  not  seeing  that  significance but  again  we  are  seeing  that  trend.

What's  interesting  is when  we  looked  at  the  positions.

And  we  noted that  obviously  we  do  not  have...

The  slopes  are  definitely not of  significance

and  there  doesn't seem  to  be  any  difference

whether  they're  in  the pre,  ongoing,  or  post.

So  it  seems  while  it  may  not  be a  practical  significance,

I mean it  may  not  be a  statistical  significance

of  practical  interest  is  the  fact that  the  ECHO  group  does  seem  to  be

benefiting  this  nurse  practitioner, physicians  assistants  group.

So  the  next  thing  we  tried  was the  Zero  Inflation   Poisson  model.

We  chose  the  Poisson  Model  due  to  the  fact that  we  had  counts

and  we  also chose  the  Zero  Inflation...

We  did  both  the  regular  and the  Zero  Inflation

and  we  chose  the  Zero  Inflation because  we  did  have  a  lot  of  zeros

in  our  data.

And  so  when  we  did  it,

you'll  notice that  we  do  have  this  slight  trend

of  increasing of  proportion of  MAT patients over time.

And looking  at  the P arameter  Estimates,   month  is  significant.

Note  that  the   Zero Inflation  is  zero.

So  if  you  look  at  it, it wasn't  really  doing  much  for  it

to  have  the   Zero Inflation  but...

It  was  informative  for  us.

Unfortunately,  when  we  went  to  evaluate  the  fit  of  the  model,

it  became  quickly apparent  that  it  was  a  poor  fit.

So  for  example,  when  we  looked  at your  generalized   R²,  it's  very  low.

And  then  we  have  this...

When  we  looked  at  the  actual   versus  predicted  plot,

what's  happening  is  that

here  are  the  predicted  values

and  they're  really ranging  between  0.1  and   0.4

where  the  actual  data  is ranging  between  zero  and  one.

So  what's  happening  is  that  the  data between  here  is  really  pushing  this  model.

And  so  obviously,   what  we  would  have  liked to have seen

to  have   is  more of  the  predictions  following  this  line.

Having  said  that,  we  did  see  some  trends and  even  though  we  didn't  have  maybe

a  statistical  piece, we  did  have  some  practical  interpretation.

So  moving  on,  we  did  have  a  control  group.

And  so  what  we  did

is  that  we  took the  nine  control  providers

were  directly  compared to  the  nine  ECHO  providers

and  they  were equivalent  in  title  and  city.

So  we  were  trying  to  match whether  they  were  a  nurse  practitioner,

physician's  assistant  or  a  physician,

and  then  where  their  practice  mainly  was,  their  primary  city.

And  the  piece  of  that  is  that  there  is a  very  different  demographic  when  we  go

from  the  South  of  New  Hampshire up  to  the  North  of  New  Hampshire.

So  we  wanted  to  make  sure that  we  captured  any  of  those.

So  when  we  did  the  Matched  Pairs  Test, we  created  a  confidence  interval

for  the  proportion  of  patients  who  had any  MAT to  the  total  number  of  patients.

We  looked  at  the  difference

between  our  control and  our  providers

and  we  know  in  that...

First  off,  when  you  look   at  the  confidence,  zero  is  not  in  there.

So  we  do  have  a  difference.

And  we  note  that  when  we  looked at  the  actual  means

you  have  about  your  treatment, ECHO  is  about 0.2...

Have 0.2  proportion  of  patients  have  MAT.

And  then  for  the  control  was  only  0.13.

So  we  do  have  a  difference  of 0.07 .

So  we  are  seeing  that our  ECHO  group  is

prescribing  MAT  more  frequently than  our  control  group.

So  the  next  was  to  try to  bring  in  that  again,

that  time  variable.

And  so  we  looked at  the   Zero Inflation  Poisson  Model

and  so  what  we  see  here,

so  first  off  the  ECHO  group  is  in  red

and  then  the  control  group  is  in  blue.

So  looking  at  it  graphically, we  are  seeing  an  increase  in  time.

It  is  evident  that  the  ECHO  group  is slightly  higher  in  prescribing,

a  proportion  of  prescribing than  our  control  group.

Again  looking  at  the  parameters in  our  model,  they  are  significant.

Unfortunately  when  we  moved again  to  assess  our  model,

we  had  a  very  similar

result  as  we  did  with  just  looking at  the  ECHO,

which  is  somewhat  not  surprising   in  the  sense  that  we  are  using

the  similar  data.

And  so  again  you're  looking at  your  Generalized   R²,  it's  very  low,

and  again  we're  noting that  our  actual  versus  predicted,

our  model  is  predicting  probably  between  now   0.05  and  0.4,

whereas  we  were  hoping   that  it  would  predict  along

the  range  from  zero  to  one.

So  findings.

So overall  we  were  able  to  detect

the  difference  in provider  diagnostic  patterns

before  and  after  they  predicted in  project  ECHO.

We  did  see  a  small  difference   in  provider  diagnostic  patterns

between  the  providers  that  did  participate in  project  ECHO

and  those  who  did  not.

One  of  the  things  that  we  did  not  control on  was  the  number  of  total  patients.

So  that  may  be something  to  consider  later.

And  then  we  also  noted  that  there  may  be a  difference  on  the  impact  of  project  ECHO

from  the  different  provider  level  title.

And  we  need to  maybe  further  delve  into  that.

So  next  steps, this  is  an  ongoing  project.

These  are  the  next  steps that  we  are  considering.

So  first  off,  what  we  would  like  to  do  is to  include  additional  providers

to  increase  the  size  of  the  database.

And  one  of  the  ways  that  we  are  looking at  doing  that  is  to  include

some  more  of  the  ECHO  periods.

So  we  do  have  one that  was  finishing  up  this  year.

So  we're  hoping  to  add practitioners  in  from  that  period.

And  we  also  may  want  to  consider the  methodology  for  detecting  MAT

and  the  medical  and  pharmacy  claims  data.

Also  we  would  like  to  analyze at  the  practice  level

with  case  control  studies to  help  combat  the  small  sample  size.

And  again,  the  overall  goal  is to  fit  an  appropriate  model.

So  we  wanted  to  thank  you  for  taking the  time  to  listen  to  our  presentation

on  the  PACT-MAT  ECHO.

If  you  have  any  questions, please  contact  us  at  our  following  emails.

Enjoy  the  rest  of  your  conference.

Thank  you.

Thank  you.