Good morning, everyone. Thank you all for being here today. My name is Simin. I'm currently a fifth-year PhD candidate in the Department of Statistics at Virginia Tech, and also a former summer intern with the Design of Experiments & Reliability group at JMP. Today, I will demonstrate how JMP supports reliability analysis to evaluate the safety of autonomous vehicles and emerging area in Artificial Intelligence. The title of my talk today is Exploring Autonomous Vehicle Safety: A Reliability Analysis Powered by JMP.

Before diving into today's talk, I'd like to share a little bit about myself so that everyone can get to know me better. My name is Simin, a fifth-year PhD student in the Department of Statistics at Virginia Tech. Go Hokies! My research focus on the reliability of AI systems, AI-driven statistical analysis, and large language model performance evaluation. Yes, I'm not the one building large language models, but I'm the one testing their robustness and reliability. My research interest aligns closely with our team's mission, so being part of the DoE and Reliability Group is a great fit for me.

Beyond research, I'm also actively involved in several professional activities and organizations. I'm a lead statistical collaborator at Statistical Applications and Innovation Group, also called SAIG in short, which provides statistical consulting services across the Virginia Tech campus. I also serve as a committee member for corporate partner events in the Department of Statistics. Recently, I was honored to be selected as the secretary for the Mu Sigma Rho, Virginia Tech Chapter, where I help support student engagement and activities.

I know this presentation is for a broad audience of JMP users. Some of you might be wondering: what exactly is reliability? When we think about reliability, a few key ideas come to mind. Consistency, robustness, confidence, proven, tested, dependable. These are all words people naturally associate with reliability, and they apply whether we are talking about a physical product, a software system, or even a machine learning model. In short, reliability is about trust through a consistent performance.

Now, let's take a closer look at a more formal definition from the field. According to Meeker (2021), reliability has been informally described as both failure avoidance and quality over time. In other words, a system is considered reliable if it avoids breaking down and continues to perform well throughout its lifetime.

This definition applies across mining domains. For example, in product reliability, we are typically referring to physical components or systems. In AI systems reliability, the idea extends to software reliability, making sure the code and infrastructure are robust and don't crush, and model robustness and performance, ensuring the model behaves fairly, even though when faced with unexpected or noisy inputs.

Now that I guess everyone have a general idea of reliability, let's turn to AI systems. As AI systems become increasingly prevalent, reliability in AI is emerging as a new and important area. Artificial Intelligence systems, often called AI system in short, have a wide range of applications, including autonomous vehicles, I would just call it AVs in short, AI-powered drones, feature recognition, and conversational AI. Among those, AVs stand out as one of the most impactful applications, where safety and reliability are essential, making it important to understand and model their real-world behavior and uncertainty.

In many studies, we observe events that occur repeatedly over time for each subject. This is known as a recurrent event process, and the resulting data are called recurrent event data. In the field of autonomous vehicles, we observe not just one, but multiple types of recurrent events. There are two different types of events involved in autonomous vehicles. One is called disengagement events, the other one is called collision events. A disengagement event occurs when the autonomous driving systems hands control back to the human driver, often due to system limitations, unclear road situations, and potential risks. A collision event, on the other hand, involves an actual crash during autonomous operation. Together, this genre is what we refer to as multi-type recurring event data.

We also have previously published papers related to AV reliability. First, Planning Reliability Assurance Test for AVs focuses on this engagement event, developing statistical methods to plan AV reliability assurance test using recurrent event data. Second, DR-AIR introduces an established repository for AI reliability data and defines common measurements and data types, providing usable clean data sets to support this emerging research area. Together, this paper provides a promoting direction for analyzing the emerging field of AI reliability.

You might be wondering, where does the data come from for AV reliability analysis? In the field of AVs, the California DMV has been running an autonomous vehicle testing program since 2015. As part of this program, companies testing AVs in California are required to submit, first, annual disengagement reports, and then, second, report any collisions within 10 days of the incident.

This slide show two examples. On the left side, a screenshot of the disengagement reports, and on the right side, the collision reports, which listed detailed incidents by manufacturer and AI-AIR. Returning to the two AV event types, disengagement and collisions. Today, we will mainly focus on the coalitions to demonstrate AV safety. As AI advances, establishing reliable current AV performance is important for public trust. This leads to the following questions. First, how should AV safety be modeled? Second, can we trust current AV safety levels? Third, how can JMP help analyze recurrent collision event data? And, most importantly, what is the relationship between autonomy level and safety?

Here is the end-to-end workflow for today's talk. We will start with data collection and cleaning, importing the source of PDFs and extracting the fields needed for the reliability analysis. Next is the data exploration. We visualize exposure-based measurements. Mileage serve as an exposure because it reflects how much driving occurred. More Mileage means more chances or opportunities for a collision. We then examine collision event patterns over time and visualize a risk measure, the collision event rate, which quantifies the likelihood or possibility of a collision regardless the mileage driver. A higher rate means a more expected collision for the same exposure.

Finally, in the data analysis, we apply reliability modeling to aggregate data, focusing on the reliability growth models, and then examine how autonomy level relates to safety level. I will work through each step in JMP to demonstrate its capabilities for AV reliability analysis.

Since we will start with collision events, let me recap how the California DMV reports look. The DMV records every AV company to file a report for any crash that causes property damage, injury, or even death. On the website, these public reports are listed by manufacturer and by year, which is shown on the left side of the slide. Today, I will just focus on the Waymo. Waymo begins as Google's self-driving project and now runs a wide hailing service with driverless cars in selected cities. Waymo driver combines cameras, radar, rudder, maps, and even machine learning to act like a careful, predictable driver. Waymo has extensive operations and consistent detailed reports, which makes its data well-suited for our reliability analysis.

Here is how we import and clean the California DMV collision reports. The left shows the public portal you saw earlier. The right shows a typical collision report. Let's open one to see the fields that it contains. For each PDF, we extract key fields like: manufacture, date and time, vehicle year, make, and model, location, and all other collation details. Then load the structured data into JMP, assign the right types, and standardize names. This gives us a clean analysis ready table for further exploration.

Let's look at the structure of the collision event datasets after we extract the key fields from the public collision reports. This table shows the field we kept, including manufacture, date, year, model, make, vehicle, model, and basic accident details, but does not include unique identifiers like VINs. As noted at the bottom, collision data are only available at the aggregate level. In other words, there's no VIN level information. We will keep this in mind as we proceed to the final analysis stage.

This is our cleaned JMP table for Waymo based on the extracted collision report fields. At the monthly level, it includes days, the number of AV testing, total mileage measured in kilo-miles, and the number of collision events. Remember, these are all aggregate counts, no VIN-level detail. Based on this, we add accumulated events to see how collision build over time. In addition, we also compute a safety risk measure for AVs. By measuring how often these events occur over time, we obtain the recurrent rate of collision events, which we use as a proxy for safety.

This raises another question. How do we create a column based on the existing column information in JMP? The detail click path is summarized here. First, let's go to Columns and choose New Columns. Then set Column Name to Collision Event Rate. Third, set Data tab to Numeric and Modeling tab to Continuous. Then open Column properties, choose Formula, and enter Custom Formula for the new column. For example, we use a cumulative sum to create the cumulative events' column, and the collision event rate is computed as the number of collision events divided by total mileage.

Here's what the JMP data table looks like after adding the new information. With the new formulas applied, it now includes two new columns: cumulative number of collision events and collision events rate. These two features drive our plops and feed reliability models we will fit next. Based on the new JMP table, let's explore the data with exploratory data analysis. Because mileage is the exposure, more mileage means more opportunities or chance for a collision.

We will start by visualizing mileage over calendar tab. Here is a clear step-by-step for plotting the mileage chart in JMP. First, open Graph and choose Graph Builder. Second, drag Date to the X-axis. Third, drag total mileage measured in kilo-miles to the Y-axis. Then choose the line element from the top two bars. This chart shows monthly mileage, the exposure. We see an earlier increase, a sharp drop in 2020, and a bounce back with the pics in 2021 to 2022, and an even graduate decline and a modest uptake at the end.

Two things I want to point out here. First, after 2020, there are months with zero mileage, which means way more AV testing post during COVID-19. Second, mileage rebalance after work, showing more extensive testing post-COVID. Let's go one step further by looking at the cumulative number of collision events and the monthly number of collision events to see safety trends for AVs. To build this visualization in JMP, for the cumulative plot on the left, first, let's open Graph Builder. Second, drag Date to the X-axis. Third, drag Cumulative number of collision events to the Y-axis. Then choose the line. Add smaller if you want a clearer trend. The slope for the cumulative plot tells the story.

A flatter slope means fewer events. A steeper slope means more events per month. We see a slow build earlier and flattened near 2020. Then a steep clamp in 2021 to 2022. After that, the slope eases with some ups and downs. For the monthly collision counts on the right. First, let's open Graph Builder. Then, drag Date to the X-axis. Third, drag number of collision events to the Y-axis. Then, choose Bar for our visualization. From the visualization on the red, you can see that the same pattern month by month, low counts earlier, a deeper round, 2020, higher spike in 2021 to 2022, and mixed levels afterward. This just give us a general idea of how the number of collisions happens based on the year we are currently interested in.

Up to now, we have to look at two measures separately, collision counts and mileage. How do we combine them? We switch from row counts to exposure adjusted risk measure, the collision event rate. This lets us fairly compare months with different mileage. First, how to build it in JMP. Still follow the similar patterns as what we just explained in earlier slides. Let's open the Graph Builder and then drag days to the X-axis. Third, drag collision event rate to the Y-axis. Then choose bar for our visualization.

In this visualization, you will see spikes and dips. Higher bars mean a higher chance of a collision per male than a month. Lower bars mean lower risk. Overall, the collision rate is relatively low, indicating Waymo is managing AV safety while. Now, let's move into reliability analysis in JMP, which is our key parts in today's talk. We will first use reliability groups with Piecewise Weibull NHPP Change Point Detection to see when and how collision risk shifts over time. The method splits the timeline into phases, each with its own trend.

How to do it in JMP? Here is the step-by-step instruction in JMP. First, let's open Analyze, and then go to the Reliability and Survival module. Choose Reliability Growth. Second, let's focus on the Date Format. Third, drag Date to Timestamp. Then, drag Number of Collision Events to Event Count. Next, run the platform. In the output, select Piecewise Weibull, NHPP Change Point Detection. In the plot on the right side, we can see the dots show cumulative collisions.

The green band shows the fitted curve with uncertainty, and the slope reflects risk. Steeper means a higher event rate. Flatter means improving reliability. You can see a suggested change point near the highlighted region. After it, the slope shifts, indicating a different risk pattern. The table shows the estimates from Piecewise Weibull, NHPP Change Point Detection. The model finds a change point in August 2022. Before that point, beta[1] is around 2.09, which means that the collision rate was rising quickly. After the change, beta[2] is around 1.17, much closer to flat, so the rate rises more slowly. Lambda sets the overall scale. The 95% Confidence intervals give the uncertainty around each estimate.

However, looking closely at the cumulative events plot from the reliability growth model, the timeline bridges into phases. In each phase, the slope tells a different story. We see several different slope shifts, suggesting the risk pattern changes across periods. This motive is dividing the calendar into different phases and refitting the model to check for changes in the results. Followed by the logic I just explained earlier, I've added a new column called Phase and refit the reliability growth model. First, we create a Phase label.

How to do it in JMP? To be more specific, first, let's go to Columns and choose New Column. Second, let's set Column Name to Phase. Third, set Data Tab to Character and Modeling Tab to Nominal. Then, fill the column by time ranges. Either select rows for each date range and type the label or add a formula in column properties. The plot on the right now marks phases boundaries with blue-dashed lines. Within each phase, the slope of the cumulative curve show the piece of events. Steeper means more collision per unit time, flatter means improvement. Comparing slopes across different phases help and link risk changes to reward shifts in operations of software.

Next, let's rerun the reliability growth analysis with Phase. We will fit the model using Piecewise Weibull NHPP. This model segments the timeline into different phases, each with its own trend. In JMP, let's follow the steps. First, let's open Analyze, choose Reliability and Survival Modules, and specify the Reliability Growth Model. Second, let's go to the Dates Format tab. Third, drag Date to the Timestamp, Number of Collision Events to the Event Count, and 'Phase' to Phase. Then click Okay to rerun everything.

From the right side, we can see that the green curve and band show the fitted cumulative events and uncertainty. The blue dashed lines mark phase boundaries. Within each phase, the slope tells the piece of collision events. This table shows the Piecewise Weibull NHPP estimates by different phases. Focus on the beta. The general idea is if the beta is greater than 1, the collision rate is rising in that phase. If beta is above one, the rate is roughly flat. If beta is less than one, the rate is falling. That's an improvement in reliability.

Here, phases A, B, C, D, E, G, and I have beta greater than 1, indicating rising risk in those periods. However, phases F and H have beta less than 1, suggesting reliability improvement. That's how those two different models fit and how the model results change if we compare this to different mode of fittings. Remember, we also want to explore the relationship between autonomy and safety. To a proxy autonomy, we will look at disengagement events. We will use the same approach as before. Collect the most aggregate monthly disengagement data for analysis. To connect autonomy with safety, I add a disengagement event rate.

We will use this alongside the collision event rate to see whether monthly with fewer disengagement also have a fewer collisions. To create the disengagement event rate in JMP, here is the step-by-step instructions. First, let's go to Columns, choose the New Columns. Second, let's set Column Name to Disengagement Event Rate. Third, let's set data tab into numeric and choose Modeling tab as Continuous. Then enter the custom formula. The number of disengagement events is divided by the total mileage. Based on this formula, we can get the monthly disengagement events rate.

On the left is the collision event rate, the proxy for the safety in AVs. On the right is the disengagement event rate. Our proxy for autonomy both vary month to month. The key idea here is when the disengagement rate drops, the collusion rate often stays lower, but we need a direct comparison to be sure. Next, I will put these two rates on a scatter plot with a smother to quantify the relationship between these two measurements. For this visualization, the X-axis is the disengagement rate. The Y-axis is the collision rate. The line with the shaded band is the fit with 95% confidence interval.

From here, we can see a slight upward slope. Months with more disengagement per mile tend to have more collisions per mile. The relationship is statistically significant but modest, so there's a lot of spread. This indicates a correlation between these two measurements. The key takeaway here is, as autonomy improves, the collision rate generally stays lower on average, at least for Waymo, the specific manufacturers we are currently focusing on.

Let's do a quick wrap up of what we did to explore autonomous vehicle safety. We started with data collection and cleaning, importing the DMV reports into JMP and creating new columns from the road field to make sure the JMP data table was analysis ready. Then we move to the data exploration part. We visualized the exposure, which is mileage over time and plotted collision events and their risk to get an exposure adjusted view of risk. Finally, in the data analysis, we fit two reliability growth models on the aggregate data using Piecewise Weibull NHPP Change Point Detection to identify phases and slope shifts, and further used the Piecewise Weibull NHPP to segment the timeline into different phases. In addition, we explore how autonomy relates to safety for AVs by comparing the disengagement rate with the collision rate.

Last but not the least.