Recurrence: An analysis of adverse events whose time has come in clinical trials

[#1]

Most analyses of adverse events (AEs) are based on the presence or absence of events at the patient level. For example, if a patient experiences one or more headaches, they are considered a “success” in that the event occurred (= 1). Patients who do not experience a headache are considered failures (= 0). In traditional summaries of safety, the number of patients who experience one or more headaches are tabulated for each treatment arm, and the proportion (or odds ratio or risk ratio) of subjects experiencing a headache are compared between treatment arms.

However, this analysis has important limitations. First, patients who experience one event versus many events are weighted identically with no distinction made for the number of events a patient may experience (except for patients who experience no events). Second, the timing at which these events occur is entirely ignored within the analysis, with incidence summarized over extremely large (and often varying) durations of follow-up. Study teams have no insight into whether events occur early in treatment compared to late, or if the occurrence of events is constant, increasing, or decreasing over time. The Recurrence Report in the forthcoming JMP Clinical 19 makes it easy to screen a clinical trial for noteworthy safety concerns. Throughout this blog post, we illustrate methodologies using data from patients with probable mild-to-moderate Alzheimer’s disease.

When using CDISC standards, the AE or ADAE domains summarize events that were experienced by patients in the study, but more information is needed to perform a recurrence analysis. Dates representing the start and end of an appropriate follow-up period need to be defined for all patients, not just those patients who experience a particular AE. Further, the derivation of the appropriate follow-up dates will depend on the types of AEs of interest for analysis (Figure 1).

Figure 1. Follow-up for particular adverse events

The events of interest can be broken down into these subgroups:

• All Events: Any AEs that may occur on study once informed consent is signed until the patient ends study participation.
• Treatment Emergent Events: Any AEs that may occur on or after the first dose of study therapy determined by the treatment arm.
• On Treatment Events: Any AEs that may occur on or after the first dose of study therapy and before or at the last dose of study therapy.
• Pre-Treatment Events: Any AEs that may occur prior to the first dose of study therapy.
• Off Treatment Events: Any AEs that may occur after the last dose of study therapy.

During his time at the U.S. Food and Drug Administration, Ohidul Siddiqui described the limitations of traditional incidence analyses for AEs and how an analysis using the mean cumulative function (MCF) can provide greater insight into patient safety. Briefly, the MCF is the mean number of cumulative events that occur up to a particular point in time. This non-parametric analysis makes no assumptions about the form of the MCF, and at any given point along the curve, the MCF can be interpreted as the average of the individual cumulative AE curves of all patients. Analyses of recurrence using MCFs have been used extensively in manufacturing applications to assess the likelihood of product failures over time, and the JMP Recurrence Analysis platform has been used to provide important insights for many years. However, recurrence analyses have not been readily conducted in medical product development and subsequent regulatory review. There may be several reasons for this: 

1. Limited software. As Siddiqui describes, MCF methodologies are available in PROC RELIABILITY, which is a part of SAS QC software. In my experience, this software was rarely available to statistical teams involved in the design and analysis of clinical trials. PROC PHREG, a procedure to perform Cox’s proportional hazards models for time-to-event data (among others), has the ability to perform recurrence analyses with some additional assumptions (and with what I think is an extremely challenging way to incorporate follow-up information), but I generally did not see its use in practice. While the lack of SAS QC may have been a greater limitation in the SAS-dominated regulatory environment of the past, recurrence analyses still do not appear to be regularly performed, even with the availability of JMP, R, or Python. For example, recurrence analyses are not described in FDA standard safety outputs.
2. Data visualization burden. Perhaps the lack of regular application of recurrence analysis is due to the data visualizations required to help communicate the story behind the data. For example, has anyone anywhere in the history of humankind ever produced a Kaplan-Meier analysis without the accompanying Kaplan-Meier curve to communicate the results of a particular endpoint? This is highly unlikely. But given the large number of AEs present in the typical clinical trial, this presents a very cumbersom analysis burden for the statistical team. Further, there needs to be a straightforward way to review multiple recurrence analyses quickly to identify the most important signals among the multitude of events.
3. Rare events. While it is true that the most problematic AEs tend to be rare, making recurrence analysis a bit of overkill to apply in practice, there are some therapeutic areas with a deluge of AEs that traditional approaches to AE analysis do an extreme disservice. In addition, there are many commonly occurring AEs that would benefit from more informative summaries across most therapeutic areas. Even in situations where events are rare, considering the explicit timing of those events in an analysis is far more informative than merely summarizing the incidence for huge blocks of follow-up.
4. Analysis literacy. Perhaps these methods are not being used because statisticians and data scientists have done a poor job explaining why recurrence analyses are needed and how the resulting output should be interpreted. Here, I am very hopeful. Clinicians have grown very comfortable in reviewing and interpreting Kaplan-Meier plots, and plots of MCF are not extremely different. Perhaps with additional training and examples, non-statisticians (and even many statisticians) will be comfortable interpreting MCF plots.        

JMP Clinical 19 solves many of the above issues with Recurrence Report. The analysis is quick, highly visual, and interactive. Further, Recurrence Report leads with a summary across all AEs to help identify particular AEs with notably large MCFs or differences between treatment arms. Treatment emergent adverse events are summarized in the analysis described in Figure 2 and beyond. The CDISC Pilot study includes data from 254 patients randomized to one of three treatments (xanomeline high dose, low dose, or placebo) for a 26-week treatment period; 242 different AE preferred terms were observed.

In the Siddiqui paper, the confidence intervals for MCF were obtained at the end of follow-up for each treatment arm. Rarely, in practice, will treatment arms have identical maximum follow-up time. JMP Clinical 19 takes the approach of analyzing treatment arms at the minimum of the maximum (e.g., the minimax) follow-up for each treatment arm so that all treatments are compared at a particular slice of time. In Figure 2, the minimax follow-up time is 210.6 days, corresponding to roughly 30 weeks.

Figure 2. Recurrence report dashboard

The initial summary of all TEAEs is presented using a dot-forest plot (Figure 3), which is commonly used to summarize incidence analyses among AEs. Here, the dots in the left panel represent estimates of the MCF, while the confidence intervals in the right panel represent treatment differences in MCF between the two doses of xanomeline and placebo at the minimax follow-up. The control group for the confidence intervals can be modified using the combo box. The y-axis is sorted by the maximum difference with placebo with pruritis (itching, to us non-clinical people) exhibiting the greatest difference. Here, estimates of the MCF are 0.567, 0.414, and 0.145 for xanomeline high dose, xanomeline low dose, and placebo, respectively. So, for example, at the minimax follow-up of 210.6 days of treatment, the average number of cumulative pruritis events experienced by patients in the xanomeline high dose arm was 0.567. In other words, a pruritis event is experienced by roughly 1 in every 1.76 (1/0.567) patients at 210.6 days after the start of treatment for patients in the high dose xanomeline arm.

Figure 3. Dot-forest plot for TEAEs

The remaining plots in the report are specific to an individual preferred term, which allows for a deep dive into specific events based on the contents of the dot-forest plot. By default, the preferred term with the maximum difference with placebo is presented (here, pruritis), but users can select any event using the Report Filter in the Display Options (left side of Figure 1).

A pair of event plots, sometimes referred to as swimmer plots, illustrate the occurrence of pruritis events over the trial (Figure 4). The left plot summarizes individual patients sorted by treatment arm and length of follow-up. Bubbles indicate the occurrence of pruritis events with bubble size indicative of the number of events occurring at any given time. In practice, the time of day that AEs occur is rarely captured, which means that event plots will tend to summarize the number of events that occur on a given day. In our example, while it appears some patients experience more than one event, with some even experiencing more than one pruritis event on a particular day, most patients do not suffer a pruritis event over the course of the study. The right plot collapses the information in the left plot to a single treatment-specific line, drawn to the maximum follow-up of each arm (recall, this is 210.6 days for placebo). Here, it is more straightforward to interpret what is happening at the treatment level. Patients on the xanomeline arms experienced pruritis events right away, with most events reported through Day 100 and limited events reported thereafter. Most placebo events appeared to have occurred between Days 50-100. Note that bubbles represent the events that occur for a particular patient for a particular time.

Figure 4. Event plots for pruritis

Figure 4 displays greater statistical detail by summarizing the estimates and pointwise 95% confidence intervals for the MCFs (left plot) and the treatment differences between MCFs (right plot). Plots of the MCF allow us to interpret whether there is any change in the rate at which events occur over time. A relatively straight MCF implies that events occur at roughly a constant rate. A curve that concaves up (like a cup) implies that the rate of events is increasing over time, while a curve that concaves down suggests the rate of events is slowing. For our pruritis example, the curve concaves down, implying that the rate of itching events decreases, though the two xanomeline arms had a relatively constant rate up to Day 100 or so. Because AEs are self-reported, whether this decrease in event rate is due to an actual slowing in the rate of itching events (due to improved tolerance or poorer treatment compliance) or due to reduced reporting requires additional data.

Figure 5. Plots of MCF and treatment differences for MCF for pruritis

The plot of treatment differences in MCF (right) illustrates that after approximately Day 25, both active treatment arms illustrate an excess of itching events compared to placebo (since the shaded area excludes 0). Note that the control group for the confidence intervals can be modified using the combo box.

Finally, the dual forest plot presented in Figure 6 gives a snapshot of the estimates and 95% confidence intervals for the MCFs (left) and treatment differences in the MCFs (right) at the minimax follow-up. Here, the estimate and 95% pointwise confidence interval for the xanomeline high dose MCF is 0.567 (0.368 and 0.765). In other words, we are 95% confident that the average number of cumulative pruritis events after 210.6 days of follow-up is between 0.368 and 0.765 events, or one pruritis event for every 1.3 (1/0.765) to 2.7 (1/0.368) patients. The control group for the confidence intervals can be modified using the combo box.

Figure 6. Dual forest plot of MCF and treatment differences for MCF for pruritis

JMP Clinical 19 makes it easy to perform a more informative analysis of AEs with Recurrence Report, taking into consideration event recrudescence, as well as the timing at which the events occur. One can hope that with better tools available to analyze patient safety that we gain greater insight into the risk that novel medical products may hold. The regular use of recurrence analyses may one day allow for more informative drug labels, with the average number of cumulative events expected to occur after specific exposure times to new therapies (Table 1). Imagine being able to understand the average number of headaches, or the average number of mild, moderate, or severe events likely to occur after 90 days of exposure to treatment. Now that seems useful!

Table 1. Cumulative means of the number of events experienced according to drug exposure

	Xanomeline High	Xanomeline Low	Placebo
Pruritis
30 Days	0.191 (0.078, 0.304)	0.170 (0.041, 0.299)	0.012 (0*, 0.036)
60 Days	0.390 (0.226, 0.554)	0.319 (0.161, 0.476)	0.024 (0*, 0.058)
90 Days	0.439 (0.266, 0.612)	0.380 (0.213, 0.547)	0.104 (0.018, 0.190)
120 Days	0.537 (0.343, 0.731)	0.414 (0.244, 0.584)	0.145 (0.043, 0.247)
Erythema
…	…	…	…

Table 1. Cumulative means of the number of events experienced according to drug exposure

Note: Values are cumulative means and 95% confidence intervals.

* Negative lower limit displayed as 0 since cumulative means cannot be negative.