Hello. Thank you for joining us today
for our presentation on,
"Measuring Change in Medication Assisted Treatment
for Participants in project ECHO Using Administrative Healthcare Claims"
We are excited to share how we use JMP as a key tool in our analytic work.
Next slide, please.
My name is Erica Plante and I am a senior scientific data analyst
at the Institute for Health Policy and Practice
at the University of New Hampshire.
I am joined by Dr. Michelle Capozzoli, Senior Lecturer
in the Department of Mathematics and Statistics,
also at the University of New Hampshire.
Neither Michelle nor I have conflicts of interest to disclose.
Next slide, please.
Before we describe our work,
I would like to provide a brief overview on Project ECHO.
Project ECHO was founded in 2003 by Dr. Sanjeev Arora
at the University of New Mexico.
Dr. Arora is a Physician specializing in Gastroenterology and Hepatology.
He was seeing patients with Hepatitis C die at alarming rates
because they could not access care
for this treatable disease in a timely manner.
He sought to bring providers together to form a community of practice
where doctors and other specialists can learn from each other.
ECHO is an "all teach, all learn" model,
and the sessions are often centered around a key issue or condition.
The University of New Hampshire launched its project ECHO hub in 2018
and has since produced a number of ECHO programs,
including the Partnership for Academic Clinical Telepractice, Medications
for Addiction Treatment, or PACT-MAT.
Next slide, please.
The primary goal of PACT-MAT ECHO is
to increase the number of nurse practitioner students
in graduate and postgraduate programs who receive waiver training,
apply for the waiver, and subsequently prescribe MAT.
Secondarily, the project seeks to increase provider self- efficacy
in managing patients with Substance- use Disorder.
The program developed a learning community
that enabled a culture that understood addiction
as a chronic disease and was prepared to address
the range of issues that emerged during the process of treatment.
Specifically, this program focused on all participants becoming proficient
and culturally competent in prescribing and treating SUD,
as well as enhancing capacity and qualities of services available
to patients in their communities through their providers.
Next slide, please.
After the completion of the first PACT-MAT session,
the team was interested in answering some questions
about the PACT-MAT ECHO through Claims Data Analysis.
But here's some core information about the analytic project.
The analytic period of interest was from 2018 through June 2020.
This was to capture data prior to and after the first PACT-MAT ECHO session.
The project was funded
by the Substance Abuse and Mental Health Services Administration ( SAMHSA),
as part of 150K 3-year grant.
The project's principal investigator was Dr. Marcy Doyle.
We were wanting to ask a few questions about the actual ECHO program
and how we were able to see if provider practices had changed
after actually participating in the ECHO.
And the Center for Health Analytics of Informatics
at the Institute for Health Policy and Practice at UNH.
We're fortunate enough to have access to healthcare, administrative claims,
and enrollment data for commercial and New Hampshire Medicaid policies.
Therefore, the CHA team was brought into the research project
to collect and aggregate the data,
and UNH's Department of Mathematics and Statistics was brought in
to build models and perform analysis.
Next slide, please.
And these are the questions that we asked as our core research questions.
Did the PACT-MAT ECHO Series have an impact
on participants' MAT prescribing practices?
And can we successfully perform a case/ control study on providers
using administrative claims data?
Next slide, please.
When collecting health care claims,
we included all members, ages zero to 64 who had medical
and pharmacy enrollment in the month of interest.
PACT-MAT participants self- reported their name, titles, NPI,
organization's name and address, and their waiver status.
We cross- referenced that data against CMS's National Plan
and Provider Enumeration System, also known as the NPPES Registry.
In the cases of a mismatch between the self- reported data
and the NPPES registry, the self reported data was considered
the most up to date and was used for the analysis.
Information on our control group was sourced only from the NPPES.
Next slide, please.
And claims reflect if one or more service lines included
an MAT procedure code or drug code,
they were also flagged
if an Opioid-Related Disorder diagnosis code was found.
Medical providers were selected as having billed for MAT if at least
one service line included their NPI or one of the MAT CPT codes.
Prescribing providers were selected as having prescribed MAT if at least
one pharmacy service line included their NPI as the prescriber
and at least one of the MAT NDC codes.
The case and control populations each had two pairs of datasets.
Next slide, please.
One pair for each insurance type, commercial or New Hampshire Medicaid.
The first date included there the providers NPI and information,
as well as total aggregates by month of all providers,
patients and claims.
Patients with Opioid- Use Disorder (OUD),
patients with any medication assisted treatment,
and patients with both OUD and MAT.
The second dataset provided the same dataset as the first data set...
Same data as the first data set, with the aggregation
at the member demographic level,
such as age, category, county, and sex.
No identifiable member data was applied to the statisticians.
Now I'm going to pass the presentation to my colleague, Dr. Capozzoli.
Thank you, Erica.
Once we obtain the data.
The data was actually analyzed by Rebecca L. Park.
She was a UNH master's student under the supervision of myself
and Dr. Philip Ramsey.
We received the three data sets.
One was the practitioners demographics such as name,
National Provider Identifier, title, practice address.
The other two were the claims datasets.
One for the Medicaid and one for the commercial.
So the original data was extremely large
for both the commercial and for Medicaid
for each practitioner, it tracked each of their MAT patients' history
over the study period.
So the original thought was try to...
Use their pattern of behavior of their patients over time.
Quite quickly, it became apparent that this approach was a little bit problematic
and also we ran into some privacy laws in trying to make sure
that all identifying markers were not available.
So what we did was we honed in
on several of the variables for the demographics.
So we honed in on the National Provider Identifier,
their title, and the city of their practice.
And then from the claims data, we focused in on the month and year.
So this tracks what month and year
that we were looking at.
The phase of the program was the Pre,
so this is Pre before ECHO,
Ongoing is during ECHO, Post is obviously after ECHO.
So then we aggregated the data and so instead of looking at
every single visit, what we looked at were months
and we looked at patient totals.
So the first, we obviously looked at was the number of patients total
for that practitioner during that month,
the number of patients with Opioid-U se Disorder,
the number of patients who had any MAT,
the number of patients with OUD who had any MAT.
Further, when looking at the Medicaid
versus the commercial care, during the exploratory analysis,
it became apparent that we were going to need to focus in on the Medicaid data
due to the low patient numbers in the commercial care.
So here for example, when we were looking at it,
it became apparent;
so on average they had one patient per month with any MAT.
And so what we were doing was there's a lot of sparse data here
and it was just not conducive to trying to fit models.
So we focused in on the Medicaid data.
Further, we initially had 20 providers,
which we had to reduce down to nine providers.
And the reason being is that some of the providers
had too many months of missing data.
Some of them, we had to eliminate
because the majority of the months, they didn't have any patients who had MAT.
And then further, as we started to fit the models,
it became apparent that we also needed a minimum of 10 total patients per month.
So we did end up with a very smaller sample size
than we had originally thought we would have.
The other piece, as we were exploring the data,
using some of the tools that John provides,
we noticed that the nine practitioners had on average the number of patients
or number of total patients differed.
So for example, they could have,
I think it was between a total range of one to 161 patients.
So between that and then looking at the trend over time,
so we looked at the average number of patients total
and as you notice this, the blue line,
you notice that just in general there's an increase of patients over time.
We also looked at the average number of patients
with the Opioid-Use D isorder
and those who had any MAT, and those who were diagnosed
with OUD and had any MAT.
And if you notice, all four have a similar trend of an increasing trend.
So to combat that, because what we want to know is,
are the practitioners increasing their prescribing?
Not that they are having more in patients over time,
we normalized the data by creating a new variable.
So what we created was the proportion of patients
who had any MAT in comparison to the total number of patients.
And the reason that we did use the patients who had any MAT
is due to the fact that a diagnosis of an OUD does have a certain stigmatism.
And so we felt that it would be better to look at any MAT.
So Analysis Considerations.
From the beginning we knew that we had
a small sample size of practitioners, only nine.
The other thing that became apparent
through a lot of the graphical representations of the data
was that we had a lot of noise.
And so in taking this into consideration,
we decided to attempt several different approaches.
The first approach was basically, your means comparison:
ANOVA and Matched pairs.
Then we thought about bringing in that time variable.
And so we did look at them in several different ways,
from just simple Linear Regression, from the beginning.
We did consider Structural Equation Models.
Dr. Laura Castro- Shilo from JMP had come to one of Dr. Ramsey's classes
and given a talk on these models.
So originally, when...
We were looking at the data, we thought that these models
might be appropriate.
But it became apparent quickly because of the difficulties that we had
that they just were not working for us.
So the next, we worked with Segmented Regression.
That was chosen
because in some previous work with claims data ,
it was brought up that maybe the Segmented Regression would work well
because we had data that was looking at pre and post.
So we thought that with our pre, ongoing, and post,
that maybe the Segmented Regression would be appropriate.
We also looked at Exponential Regression and Logistic Regression.
We also looked at their Generalized Regression Models
and including just the regular and then zero- inflated Poisson,
Binomial, and Negative Binomial Models.
And the reason why we decided to look at the Poisson,
and Binomial and Negative Binomial is
the data is inherent that it has counts and so we thought that maybe
these might be appropriate.
And so what I'm going to focus on today are the Means Comparison,
the Segmented Regression and the zero- inflated Poisson
and this is going to be for both just looking at the ECHO group
and then the Matched Pairs comparing the control group to the ECHO group.
So the first analysis ignores the time variable
and we're just looking at averages.
What's the average proportion of patients in your pre, ongoing, and post.
And just from the means, it is quickly apparent and from the graph
that our pre phase is definitely lower than the ongoing and post.
We also tested for the...
Because we have small sample with practitioners,
we did look at the variance and we noted that we did have an issue
with non- equal variance and so we did use Welch's test
instead of the traditional ANOVA
to conduct to see if we had differences between the phases and obviously we do.
And then to determine statistically which ones were different,
we did look at the All Pairs Tukey- Kramer Test.
And it did confirm that our pre phase was different than the ongoing and post,
but the ongoing and post were very similar.
And so this gave us some...
This obviously is indicating that we do have some differences.
The ECHO program is making a difference.
So this Segmented Regression, as I noted before,
was suggested because we do have the three phases: pre, ongoing, and post.
And we were able...
Dr. Ramsey had suggested that we use a script
that is done by David Burnham, by Pega Analytics.
If you are interested in this script, the link is here on the website,
it gives you the code and it also gives you
a very detailed description from line to line,
what he is doing in this code.
And so what's happening here, when we ran the script,
We were able to fit separate regressions to the three different phases
as well as get the fit,
an R², and test the significance of the slope.
And so again, we're seeing this pattern that we saw with the ANOVA.
You do see that the pre is definitely lower than the ongoing and post.
Unfortunately, the slopes for all of the three phases
were not significant.
You can also see that we do have a significant amount of variability.
So next, what we considered was the fact that we had two types...
Or two categories of practitioners;
we had those who were nurse practitioners, physician assistants;
and then we had the physicians.
Even though they were small sample sizes.
We decided to see, "Okay, can we see some kind of a signal?"
"What is going on here?"
And when we looked at the nurse practitioners again,
you see that behavior of the pre being lower than the ongoing and the post.
And what we noted is that for the pre phase,
we are seeing a little bit of a significance even though it is 0.07,
it is saying that there is a signal here and we do have something going on.
Unfortunately for the ongoing and post,
we're not seeing that significance but again we are seeing that trend.
What's interesting is when we looked at the positions.
And we noted that obviously we do not have...
The slopes are definitely not of significance
and there doesn't seem to be any difference
whether they're in the pre, ongoing, or post.
So it seems while it may not be a practical significance,
I mean it may not be a statistical significance
of practical interest is the fact that the ECHO group does seem to be
benefiting this nurse practitioner, physicians assistants group.
So the next thing we tried was the Zero Inflation Poisson model.
We chose the Poisson Model due to the fact that we had counts
and we also chose the Zero Inflation...
We did both the regular and the Zero Inflation
and we chose the Zero Inflation because we did have a lot of zeros
in our data.
And so when we did it,
you'll notice that we do have this slight trend
of increasing of proportion of MAT patients over time.
And looking at the P arameter Estimates, month is significant.
Note that the Zero Inflation is zero.
So if you look at it, it wasn't really doing much for it
to have the Zero Inflation but...
It was informative for us.
Unfortunately, when we went to evaluate the fit of the model,
it became quickly apparent that it was a poor fit.
So for example, when we looked at your generalized R², it's very low.
And then we have this...
When we looked at the actual versus predicted plot,
what's happening is that
here are the predicted values
and they're really ranging between 0.1 and 0.4
where the actual data is ranging between zero and one.
So what's happening is that the data between here is really pushing this model.
And so obviously, what we would have liked to have seen
to have is more of the predictions following this line.
Having said that, we did see some trends and even though we didn't have maybe
a statistical piece, we did have some practical interpretation.
So moving on, we did have a control group.
And so what we did
is that we took the nine control providers
were directly compared to the nine ECHO providers
and they were equivalent in title and city.
So we were trying to match whether they were a nurse practitioner,
physician's assistant or a physician,
and then where their practice mainly was, their primary city.
And the piece of that is that there is a very different demographic when we go
from the South of New Hampshire up to the North of New Hampshire.
So we wanted to make sure that we captured any of those.
So when we did the Matched Pairs Test, we created a confidence interval
for the proportion of patients who had any MAT to the total number of patients.
We looked at the difference
between our control and our providers
and we know in that...
First off, when you look at the confidence, zero is not in there.
So we do have a difference.
And we note that when we looked at the actual means
you have about your treatment, ECHO is about 0.2...
Have 0.2 proportion of patients have MAT.
And then for the control was only 0.13.
So we do have a difference of 0.07 .
So we are seeing that our ECHO group is
prescribing MAT more frequently than our control group.
So the next was to try to bring in that again,
that time variable.
And so we looked at the Zero Inflation Poisson Model
and so what we see here,
so first off the ECHO group is in red
and then the control group is in blue.
So looking at it graphically, we are seeing an increase in time.
It is evident that the ECHO group is slightly higher in prescribing,
a proportion of prescribing than our control group.
Again looking at the parameters in our model, they are significant.
Unfortunately when we moved again to assess our model,
we had a very similar
result as we did with just looking at the ECHO,
which is somewhat not surprising in the sense that we are using
the similar data.
And so again you're looking at your Generalized R², it's very low,
and again we're noting that our actual versus predicted,
our model is predicting probably between now 0.05 and 0.4,
whereas we were hoping that it would predict along
the range from zero to one.
So findings.
So overall we were able to detect
the difference in provider diagnostic patterns
before and after they predicted in project ECHO.
We did see a small difference in provider diagnostic patterns
between the providers that did participate in project ECHO
and those who did not.
One of the things that we did not control on was the number of total patients.
So that may be something to consider later.
And then we also noted that there may be a difference on the impact of project ECHO
from the different provider level title.
And we need to maybe further delve into that.
So next steps, this is an ongoing project.
These are the next steps that we are considering.
So first off, what we would like to do is to include additional providers
to increase the size of the database.
And one of the ways that we are looking at doing that is to include
some more of the ECHO periods.
So we do have one that was finishing up this year.
So we're hoping to add practitioners in from that period.
And we also may want to consider the methodology for detecting MAT
and the medical and pharmacy claims data.
Also we would like to analyze at the practice level
with case control studies to help combat the small sample size.
And again, the overall goal is to fit an appropriate model.
So we wanted to thank you for taking the time to listen to our presentation
on the PACT-MAT ECHO.
If you have any questions, please contact us at our following emails.
Enjoy the rest of your conference.
Thank you.
Thank you.
