Funnel Plot add-in identifies unusual subgroups

[#1]

Funnel plots are used for continuous, binary, and count endpoints to assess the performance of mutually exclusive subgroups when compared to the entire population (Spiegelhalter, 2005). Within clinical trials, funnel plots are often used to assess site performance and whether additional training and oversight are needed to maintain data quality (Zink et al., 2018).

As an example, consider the bar chart on the left of Figure 1 that summarizes the percentage of patient deaths that occur among the sites in the nicardipine clinical trial (Haley et al., 1993). The plot is annotated with the overall death rate of approximately 25%. If concerned about safety, a natural inclination would be to follow up with sites exhibiting the highest death rates to determine if there are factors leading to an excessive number of deaths.

Figure 1. Nicardipine example

However, an extremely high death rate may not be cause for alarm. Take, for example, a site that may only have a single patient enrolled. The death rate at this site has two possibilities, 100% and 0%. With an overall death rate of 25%, a death rate of 100% at a site with only one patient enrolled doesn’t necessarily require follow up. The bar chart could report the site sample as labels, but this doesn’t really solve the problem for sites with more than one patient enrolled. Ideally, interest lies in flagging sites where the rate of death is unusual considering the number of patients enrolled at that site.

Enter the funnel plot, which is presented in the right-hand side of Figure 1. Here, the funnel plot presents the proportion of deaths at each site by the sample size of patients treated at that site. A vertical reference line (red) summarizes the overall death rate, while curved lines connect the upper and lower limits of the 99.8% and 95% confidence intervals (CIs) produced at each and every sample size presented along the X axis. The regions produced by these reference lines can be used to categorize the site markers and triage the necessary interventions. For example, the markers in the right panel of Figure 1 are:

1. Green, if the death rate was below the overall death rate (mild risk).
2. Yellow, if the death rate is above the overall death rate (moderate risk).
3. Red, if the death rate exceeds the upper 99.8% limit, which is approximately three standard errors above the overall response (severe risk).

The corresponding bars in the left panel are colored accordingly, which highlights that relying on excessively high death rates alone is insufficient for identifying problematic sites.

Additional categorizations are possible. For example, consider:

1. Green, if the death rate is within the limits of the 95% CI (mild risk).
2. Yellow, if the death rate exceeds the 95% CI and is within the 99.8% CI (moderate risk) with rates above the overall average (indicating excess deaths) and rates below the average (indicating scarcity).
3. Red, if the death rate exceeds the 99.8% CI (severe risk) with rates above the upper limit (indicating excess deaths) and below the lower limit (indicating scarcity).

Of course, while fewer deaths are ultimately a good thing, an absence of deaths may be a cause of concern (e.g., delayed reporting, misconduct).

The Funnel Plot add-in in the JMP Marketplace makes the production of funnel plots easier than ever, creating plots for continuous, binary, and count endpoints, with an option to adjust the intervals of binary outcomes according to the duration of follow-up (Liu et al., 2006). Users define intervals according to specific confidence limits or multiples of the standard error (Figure 2).

Figure 2. Funnel Plot dialog

For example, Figure 3 summarizes the exposure-adjusted incidence rates of blood transfusion per 100 days.

Figure 3. Exposure-adjusted incidence rates of blood transfusion per 100 days

The manual describes possible future refinements to the add-in. If you find it useful, please share your thoughts with me through the JMP Marketplace!

References

1. Hayley EC, Kassell NF & Torner JC. (1993). A randomized controlled trial of high-dose intravenous nicardipine in aneurysmal subarachnoid hemorr.... Journal of Neurosurgery 78: 537-547.
2. Liu GF, Wang J, Liu K & Snavely DB. (2006). Confidence intervals for an exposure adjusted incidence rate difference with applications to clinica.... Statistics in Medicine 25: 1275-1286.
3. Spiegelhalter DJ. (2005). Funnel plots for comparing institutional performance. Statistics in Medicine 24: 1185-1202.
4. Zink RC, Dmitrienko A & Dmitrienko A. (2018). Rethinking the clinically based thresholds of TransCelerate BioPharma for risk-based monitoring. Therapeutic Innovation & Regulatory Science 5: 560-571.