Good morning. My name is Ray Bunkofske, and I'm from Regeneron Pharmaceuticals. I will be presenting an application I wrote in JMP to make it easier to detect variation in our assay key performance indicators, and then track down the root cause of any unusual results. There are a variety of summary statistics from each assay, and the KPIs are linear combinations of them that I've been shown to be useful for detecting issues with these results.

Across the business, there are tens of different assays, each with several parameters to be monitored and reviewed when problems are detected. The idea is to provide a single application where users can detect which assays are having issues and determine likely sources for the variation in those assays. There's also the ability to display metrics for management on long-term health and identify sources of systematic variation. It leverages the multivariate analysis supported by JMP to provide a single page with a normalized results from every assay to make it quick to identify emerging trends. The primary metrics I'm using are a hotelling T² and distance to the model. These are useful as multiple KPIs may be incorporated into a single control chart, which eliminates the need to go through multiple charts for each assay.

It also supports conventional univariate charts, so users can go to a particular parameter, review the results, and do a bit of drill down into possible reasons for any odd results. We'll switch over to JMP now. Here is where the individual control charts may be reviewed. For each parameter, the control limits are stored in a column property, and the top left is the list of the available assays and the number of times each assay has been run in the last year. Then a couple of columns showing the parameters that have been tagged as being the highest for any given week. Finally, there's a stoplight chart, which is green if everything's within 1 sigma, yellow if it's between 1 and 3 sigma, and then red if it's outside 3 sigma. To view a specific chart, you simply highlight the one you're interested in. Click Show Parameters. Those parameters appear over here, and then you can pick which ones you wish to review, whether it's one of them or many of them. Then you can select Create Charts, and the parameters are shown below. Now, here you can see an example where I've got a couple of points that are very much outside the control limits.

If you want to expand the scale, the easiest way to do that is to click rail outlier points, create the charts again, and then any high fliers or low fliers are plotted as stars on the 3 sigma control limit. You can move among the plots with the next and the prior. You've got a little bit of legend capability where you can select points that have laboratory fails, notice of events, general fails. You can color by the version of the assay. Who did the analysis? These are pseudonyms. The managers have a key, so they know that name 8 is Sally and name 35 is Johnny.

Then you also have the sample type, whether it's a non-routine or release or a stability. Then they're colored accordingly. The other thing you can do is go to what I call the situation at a glance page. Here we've got a lot going on, so I'll start at the top left. You can select a time frame. In this case, it's the last 60 days. Here are the assays with data in the last 60 days, how much, and the max hotelling T², and the max DModX. Anything with at least one of these over the upper control limit of 1 is colored red.

Here we have assays that didn't have any data in the last 60 days. Then over here is the individual observations that triggered the red points here. Sliding down, we have a plot of the hotelling T² for all the assays. Each point represents the maximum value of the hotelling for a given work week. All of them have been normalized, so the limit 1. These two vertical bars are the last two weeks, just to give your eye something to reference with. You can see that some of the spline fits have bumps and there's no points there. That's because those points are off the chart. If you click on rail, it will replot those points at the 3 sigma limit to make it easier to see what's really going on there. Then, if you want to see details of any one of these, you can simply click on the plot, and it will take you to the details page. That's CTP number 21. Here's the capability analysis that JMP does. We've got the stability on the X axis, capability on the Y axis. This plot right now doesn't look very good. If you remember, we had some outliers on this chart.

I'm going to take those outliers out and rerun the report and see if that changes where things come. Indeed, it does. My capability is now above 1.33, and my stability is still good. The detailed numbers are up here at the top. Moving down, here are the individual hotelling T², where each point is each assay, not the week max anymore and the DModX. Again, I can color by an LTIF or a notice of event or a general fault. The SOP version, the analyst, or the sample type. If you hover over a point, JMP pops up a little Pareto of the individual parameters showing you what's going on. You can then click on that and hover again, and you'll get the control chart of that parameter, which then shows up in a separate window, and you can look at the details there. I'll just make that go away for now.

The other thing you can do is maybe you want to drill down and see, we have a whole bunch of laboratory test investigations right here at the end. What's different between these points? I'll call that Group A. I'll set everybody else to Group B, and then I'll do a predictor screening, and that shows up like this, where I see that the difference between the two groups is pipette number 5, pipette number 3, reagent number 1, and so forth.

Anything with a portion over 0.1, it creates a plot for. You can slide down and see different kinds of plots, depending on what the data are. You can decide for yourself if that's something that warrants investigation. In this case, it looks like probably not. We've just got one very different pipette down here, and the same story on this one. The other thing you can do here is you can highlight some points and hit make a plate list, and then you see a listing like this, which for each of the points you've highlighted, shows you whether there should be three plates, and they should all say true, meaning they passed. We've got some that one passed, some that they all passed. It makes it pretty easy to see how many plates were run for each assay. Then, if there's any notes associated with them, they show up here, and you can go up into our tracking system and look at what's going on there. If you want to see more details, you can look at the remarks and highlighted points. Click on Show Remarks. Then what you'll see is for each of the assays, that number again, reference number, and then the notes that were put in by the analyst at the time the experiment was run.

Moving around a little bit. For the assay you picked, the parameters are shown. If you want to drill down on any one of them, just simply highlight the one you're interested in. Click on Create Chart. It makes a control chart for the parameter with all the various statistics that go along with it. Some of the other tabs we've got, we've got a Validity Report. Here on the left, is all of our individual SOP analysis. What percentage of the time they met requirements, did not meet requirements. Then, if you want some more detail, you can come over here to the right. In this case, we've got that same assay highlighted. Then you can see by month what's going on. You see how many assays were done during each month, what percentage of the time that month we had known lab errors, the details down here in a table of the percentage of the time any individual parameter was having difficulty, which lines up with this legend here on the top right. If you want to drill down some more, like maybe here in March of 2025. We'll pick March of 2025, and it plots it by work week.

You can see the results for a given week, or you can plot by Aliquot and see the results for any given assay. The advantage here is if you see some that are crazy and everybody else is doing well, your assay is probably okay, but maybe there was a new analyst or a problem with the agent at this point, and so you need to maybe dig into that a little bit. The other reporting thing we have here is known lab error reporting. By default, it lists all the assays across the bottom, and then it's color-coded by month. You can see that assay number 1 and assay number 33 seem to have the most issues. You can also look at different kinds of errors, and you can have it show up in pie charts or stacked bar charts, whatever makes the most sense for what you're looking at. Then if you want to drill in some more, you can pick a specific assay, and then it'll do a plot one bar for each month, or you can turn it around and pick a month and see the results for all the different assays. I'm going to switch back to PowerPoint.

We built this tool to give weekly reviews and provide a simple place to demonstrate that the assays are running in control. When things do go awry, a straightforward way to determine likely causes for the variation. Being able to include reagents and batch has been helpful as well as identifying certain analyst pipette combinations that maybe don't work as well as they might. Finally, I'd like to acknowledge my colleagues that made all of this possible. I wouldn't have been able to do it without them. We've got the assay team. These are the managers from the groups that are doing the assays, the folks developing the assays, the groups that are providing the equipment and reagent support, and then the automation team that actually puts all the pieces into place so that the equipment is running or connecting to our analytics applications. Thank you. Are there any questions?