My presentation is going to be on Risk-based Quality Management, Quality Risk Management, Quality by Design, Associated Tools and Challenges in a Clinical Trial Setting. I'm Chris Wells, and I work for Roche Products Limited, and I just would like to say that the views expressed in this presentation are mine and not necessarily those of Roche Products Limited.

Risk-based quality management, known as RBQM, and quality risk management known as QRM, what are they all about? Risk-based quality management provides a framework for managing the overall quality of a clinical trial. It does this by identifying the critical processes and critical data to ensure resources are focused mainly on those areas. The purpose of risk-based quality management is to ensure that risks are identified and managed proactively and systematically throughout the clinical trial process.

Whereas quality risk management is, again, a systematic process, but of evaluating, assessing, controlling, and communicating risks that can affect the quality of the clinical trial. The purpose of quality risk management is to identify potential risks to quality and implement the appropriate measures to mitigate those risks. QRM is a vital component of a comprehensive quality management system and is essential for ensuring patient safety and regulatory compliance. Quality risk management focuses on all processes and all potential risks.

But why is this important? RBQM and QRM are both essential components of a comprehensive quality management system in clinical trials. While they may have different focuses and methodologies, they are complementary approaches to managing risk and ensuring trial quality. Implementing both RBQM and QRM is crucial for ensuring patient safety, data integrity, and regulatory compliance.

Let's have a look at quality by design. Quality by design is a framework for embedding quality into the design, the conduct, and the monitoring of clinical trials. For example, if you will need standard protocol templates, standard electronic case report forms, and incorporation of prior knowledge and design of experiments utilized.

There are four main components in quality by design: the people, the processes, the tools, and the culture. With regard to the people, you need skilled professionals used to dealing in risk management. You need end-to-end processes for clinical trial quality by design and quality risk management. You need electronic systems to ensure consistent approaches across the enterprise. The culture, a clinical trial culture and clinical trial quality needs to be owned by the study team.

Now, ICH guidelines. These are produced by the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH). It's unique in bringing together the regulatory authorities and pharmaceutical industries to discuss scientific and technical aspects of pharmaceuticals and develop their ICH guidelines. In total, there are 14 guidelines covering four elements: quality, safety, efficacy, and multidisciplinary. But RBQM are governed by ICH E6 R2 and the soon-to-be-coming ICH E6 R3. These recommend a risk-based approach to quality management.

Quality by design is governed by ICH E8 R1, and quality by ICH Q9. These aim to design quality into the trials. Efficacy are governed by ICH E9. The principles outlined in ICH E9 (R1) for the Estimands, or actually these are involved, the primary and secondary, key secondary endpoints, are relevant "whenever a treatment effect is estimated or a hypothesis is related to a treatment effect is tested, whether it's related to efficacy or safety."

There are a number of tools and metrics used in RBQM. Our main metrics are quality tolerance limits (QTLs), and key risk indicators (KRIs). They are key metrics of any risk-based quality management system. QTLs are planned to identify systematic deviations at a study level, whilst KRIs typically trigger risk-mitigating actions at a site performance level. An example of a KRI would be average data entry cycle time at site. For a QTL, it could be the proportion of patients who withdrew from study drug early.

There's a couple of examples to the right of the screen here. The top is the Bayesian Hierarchical Model, which is an example of a methodology that can be used for quality tolerance limits. There's an example of a KRI metric.

Statistical monitoring is another metric used. It uses complex statistical algorithms to discover data outliers and anomalies across the trial, the results of which will inform various monitoring or escalation of communication actions. JMP Clinical is an ideal tool for utilizing when running statistical monitoring.

Data quality oversights. These are a range of other tools built to detect anomalies or data outliers. For example, query management. We would ask, are some sites querying more than other sites? Do these queries lead to changes in the data? Are the queries on only critical data? These need to be acted on and enable you to be able to bring down eventually the number of queries across a study and make sure that that study is concentrating on queries that lead to changes in the data and ensure they are only querying critical data.

The biggest challenge is implementation within RBQM. The reasons within the challenges on implementation are things like methodology. There are a whole range of methodologies. Up until recently, industry has not really been communicating as to which methodology each company may be using. There's been no standardization. But recently, several organizations have got together and created papers on this subject. It's starting to become much more of a sharing environment. This can only be good for the whole RBQM process.

Some methodologies require you to access historic data. Again, this can be a real challenge because maybe data has been archived, it's difficult to get hold of for whatever reason, and maybe even that you do not have a match for the current study that you're investigating.

Study design is a huge challenge because of the variety of studies that you have in the portfolio. You can have small studies, platform studies, things like basket studies and cohort studies, where you actually only have a small number of patients within each group. That can make it very difficult, depending on the methodologies you're utilizing. There are also very complex designs and decentralized trials where perhaps sites are not used in the normal traditional way.

The types of parameters to use for a KRI or a QTL are also sometimes quite difficult to understand. But I think with the implementation of the Estimand framework, it's starting to make it clearer for teams exactly where to focus their reviews. Lastly, even with statistical underpinning, it's not always an exact science.

Another element that is difficult is if there is no senior leadership support. This is crucial for RBQM to be a success. I think it's becoming more understood from senior leaders how important this is across the industry. But it can be complex to implement and comprehend and not quite so clear-cut as control charts within the manufacturing environment. Tools and processes are also complex to understand which is the best to use and the ability to embed. Oftentimes within the study teams, a lack of belief as to whether they are meaningful.

A whole raft of challenges here that the teams setting up RBQM have to face. That concludes my presentation, and I'd like to pass over to Sam.

Thanks, Chris, for that overview of risk-based quality management in this new framework for doing it using the key risk indicators and quality tolerance limits and some of the challenges associated with that. I'm going to show you how you can do some of this work with JMP and also in JMP Clinical.

I'm the Senior Product Manager for Health and Life Sciences at JMP. JMP Clinical is a focused and specialized product for clinical trial data review. We give users straight out-of-the-box functionality to do thorough reviews of clinical trials at the study site and subject level. It is utilized across the pharmaceutical industry and by several regulatory agencies across the world. It is focused on safety review, medical monitoring, and study monitoring.

It's one of our four main JMP products: JMP, JMP Pro, JMP Clinical, and JMP Live. JMP Clinical is an extension of JMP, runs on top of JMP Pro using JMP scripting language to automate all of the actions and the reports that you see in JMP Clinical.

JMP Pro is a comprehensive statistical and data analysis and data visualization program. Because of its flexibility that it's customizable with JMP scripting language, it allows you to develop applications on top of it. It also allows you to connect to other types of tools like SAS and Open-Source tools: R, Python, Matlab.

JMP Clinical also utilizes data standards. Having a data standard enables you to develop standard analysis. We use the CDISC data standards, primarily the Study Data Tabulation Model or SDTM or the Analysis Data Model, ADM. These are data standards that are mandated for new product submissions by regulatories in the US and Japan, and it's also recommended in many other regions.

Having that well-defined data model lets you develop standardized analysis approaches for it. JMP Clinical has been using that for many, many years. We actually have a recent article that's linked there where you can see some examples of how the data standards enables standardized reporting.

Let me move on to showing you how to do some of this in JMP. JMP is showing here, this is actually JMP Clinical that's running. I have actually done some work on my own to look at some data just to develop a simple example of how you would put together one of these key risk indicators and establish limits.

We're going to focus on two different types of data you can generate from a trial. One is the adverse events, the things that happen to subjects that are harmful, or medical and other adverse events that may or may not be related to the drug. What you want to look in this case is you want to see, what's the rate of adverse events in general?

I've got a plot here that shows the number of adverse events for each of the sites in the study. In this case, there are several studies in the site. I guess, what, about 25 sites or 30 sites here. This is the adverse events that are occurring every week. They're being recorded every week by subjects in the trial. You notice some interesting pattern there. You notice this up-down pattern. You also notice that a lot of times it gets high and then it drops down.

As a matter of fact, if I go here to the control panel, and I just remove the site ID and just plot that and just show that, and that's just sorting the sum of the counts there for the average counts, you see it goes up and then it comes down. You think, "What's going on there?" Subjects are entering the trial at different times. The study starts, and then subjects start to get entered at different rates at different sites, and so there's a point where you have a few subjects, and you have more subjects and more subjects, and then subjects in the trial, then there's fewer and fewer subjects as you go along in time.

Because the adverse events are proportional to the number of subjects and probably proportional to the exposure that the subjects have to the drugs or the placebo in the trial, that's also indicative. In this case, if you look at this plot again, what's happening here in this trial is the dosing is every two weeks, and this is a weekly chart. You see these little spikes. It looks like maybe every two weeks of adverse events. It's probably an indication that the rate of adverse events or the number of adverse events is related to the exposure.

What we can do is we can go in, and we can actually look at the exposure. Exposure just means how many days was the subject exposed to the treatment, and make a similar graph here. I'm just going to look at the exposure by week. You can see that that exposure, this is across all of the sites, but that exposure is getting more, more, more, more, and then dropping down. If we break it out by site, and we look at that plot, make a similar plot to what we had before, and you can see a very similar pattern of this up, down results in the exposure as well.

That makes you think that if I want to get an accurate picture of the rate of adverse events, I need to adjust it by the exposure to the drug. An easy way to do that is just take a ratio. Take the number of adverse events and divide that by the exposure.

I did a lot of data work to put this together. Here's a data table in this. Let's put the two tables together, and then it's just taking the ratio of the number of adverse events divided by the sum of the exposure. We can look at that graph, and you can see that this is an overlay of all of the sites. You see that's, okay, generally in a range and fairly stable, with the exception of some points that seem to be outliers.

Say, now I've got a measure. I've defined a metric, which is the exposure adjusted adverse event rate. Now I want to set limits on it. How should I set limits on it? One is if you know just from a clinical standpoint what it should be; you just have good scientific knowledge, expertise; you can just set a limit and just draw it on the plot. But lots of times, what we do is we look at the data and say this was a historical study, and we would just want to look at what's the variation in that data and set limits that are at the edges of the variation of the data or somewhere based on the variation of the data.

In this case, we can look at the distribution of those results. It turns out that if I take the log of this exposure adjusted adverse event rate, it looks very normally distributed. That means the data itself is probably logged and normally distributed, which is nice. It means I can fit a distribution to the data and I can estimate percentiles from that, or I can just use the observed percentile. There's a lot of data here, so we can probably just use the percentiles.

One approach might be, let's just take maybe the 95th percentile and the 5th percentile of that, and of course, do the backward transformation, go from the log, transform back to the original scale, and use those as limits. Additionally, you might want to have some tighter limits, maybe some warning limits. Maybe you don't investigate those things outside the warning limits as thoroughly or be as concerned about them, but you might want to put that as a limit as well. Maybe you could even use something like the 20th to 80th percentile as warning limits.

The last way you can determine an appropriate limit, if you want to have a limit that you think is going to cover most of the data and only really look at things that are extreme and outside the normal range of data, you can calculate a tolerance interval with different coverages. In this case, I've got a couple of different tolerance intervals calculated, but I'm going to show on the graph the 95% coverage, 95% tolerance interval on the graph.

If I go back here to the data table, going to plot that. There's a nice graph. It's got in the green, the green range is inside the warning limit, so that's the 20th, the 80th percentile. Things in the green, probably okay. I've also plotted everything on a log scale on the Y axis because it seems to be logged normally distributed.

Then the yellow range is the range where that's the... Anything in the yellow is in the warning zone. Probably want to look at it, but maybe not as concerned. Then outside the yellow range is you've failed, and you really need to do a more serious investigation. You could also use the dash lines, which are the upper tolerance limit and the lower tolerance limit.

It's a little confusing because in the ICH E6 terminology, they call the limit the quality tolerance limit, but I'm saying it's just a tolerance limit, which is a statistical tolerance limit. There's two different meanings. There's a statistical tolerance limit. It's just made from the data, but there's the quality tolerance limit is essentially the specification. You don't want to fail that. If you fail it, you need to investigate.

Nice way to visualize the data. We could also just go here, and maybe we didn't want to break it out by site. We can just remove the site ID and just look at it as a trend plot, and there. Now you can see there's a fair number of events that are happening here. The points that above are indicative of a higher rate of adverse events or maybe what you call overreporting. You want to say, "What's going on with that?" That could be a signal around the treatment itself, but it might also be the way the drug is being administered, maybe the way adverse events are being recorded. There's different reasons why they could be high.

But then there's also the ones that are really low and unusually low. Each one of these represents a different site on a different week. It could be maybe adverse events are being recorded properly. You want to make sure that that's also happening, that you are recording the adverse events. That's the adverse exposure adjusted rate.

The other thing you might want to look at is deviations. Deviations is when you have the protocol for the study was not followed for some reason. Maybe someone didn't fill out a form correctly. Maybe a lab test that someone was supposed to get was not happening. It didn't get recorded. Maybe a subject missed a visit where they were supposed to have a regular visit, and they missed it. There's lots of reasons why you could have a deviation. Then again, this is the count of number of deviations at each site by week.

If you look at that, it's not quite as related to exposure, but if I take away the site ID here… I'll start over. I'm going to look at week, and I'm going to make week nominal in this case, just so it's free like a category, and then look at the number of deviations and just do a bar chart here, which has the sum. There we go. You can see that the number of deviations also has that similar pattern where it's increasing and then decreasing, but then it picks up again and drops off. This may be related to exposure.

The other challenge with this is that the exposure period for this data is roughly this region right here in here. But then there's things that can happen after it, the deviations and problems that get found later on. One thing I did do is I tried to do the exposure adjusted deviation rate and plot that. That seems okay, but what happens here at the end, first of all, since you don't have weekly exposure in the latter period of the study, probably need a different metric. Probably need to do something like cumulative exposure at a site and just use that as it goes along. When you get the end, you just have the total amount of exposure that happened.

But I didn't do that in this case, but that'll be something to work on in the future. Again, I calculated in a similar way some limits using percentiles and overlaid those on the graph.

Finally, I just want to mention, for monitoring things over time, I'm just doing this in JMP. We don't have something in Clinical right now that does exactly this framework; although, we are working on that for JMP Clinical and planning to provide some features for that in the future.

But another thing we do have in JMP Clinical is we do have a risk-based monitoring report. What this does is it'll take an overall view of the study. It'll take the entire study data, calculate key risk indicators, and compare them to defined criteria. Essentially, the same thing we were doing before. It's just the methodology is a little bit different than this current methodology.

But if I run this report in JMP Clinical, this is coming in version 18. We had it in version 8. It was not in version 17, but now we have it in version 18. I'm going to run this report. Again, it will calculate several key risk indicators and compare them to some defined thresholds that was in the dialog. You chose a threshold file. Then it will display it in a nice summary graph.

There's the heat map of the risk indicators based on their score, whether they were red, yellow, green. Again, red would essentially be like you failed a quality tolerance limit. Yellow is in a warning zone. Green is it's probably okay. There's different metrics here. There's metrics based on safety. For instance, the number of adverse events per randomized subject, per week, the serious adverse events per week, the deaths per patient week, percent of deaths per randomized subject, percent of discontinued randomized subjects.

Discontinuation would be something that you want to be concerned about why are patients dropping out. They're dropping out because the drug is not working, they're getting adverse events, or they're not liking the site that they're at. There's different reasons. There's also things around enrollment. These are protocol violations. Did someone not sign their informed consent form? How many people failed the screening and weren't allowed to be in the study?

Then overall risk indicators, which are just weighted averages of those individual risk indicators. We've got that where you can look at that overall across all the sites. We can also, if you want to drill down into a particular site, you can look at a bar chart of those results for individual sites.

That's all we have today or to talk about. But this is an example of how you can use JMP and JMP Clinical to do risk-based quality monitoring, do the actual reporting and analysis, and use it to explore your data and find out where those issues might be coming from in a study. Thanks.