Hello, my name is Caleb King. I'm a developer in the DoE and reliability group at JMP. Today I figured I'd showcase what I think is a bit of an overlooked platform in the reliability suite of analysis tools, and that's the reliability growth platform. I thought I'd do that in the context of something that's become pretty popular nowadays, and that's autonomous vehicles. They're fast becoming a reality, not so much science fiction anymore. We have a lot of companies working on extensive testing of these vehicles.
It's nice to test these vehicles on maybe a nice track at your lab or something like that. But nothing beats real actual road testing, which is why early in the 2010s, the state of California's Department of Motor Vehicles actually put together a testing program that allowed these companies to test their vehicles on roads within the state. Now, as part of that agreement, each company was required to submit an annual report which would detail out any type of disengagement incidents. Or heaven forbid, any crashes that happened involving their autonomous vehicles.
Those had to be reported to the Department of Motor Vehicles, the DMV. Now, one benefit of the DMV being a federal institution is that these reports are actually available upon request. In fact, we can go there right now to the site and you'll see that you can at least access the most recent reports. We have the 2021 reports. They're still compiling the 2022. You could also, if you want, email. If you wanted some previous ones, I did that with a brief justification of what I was doing. They were pretty quick to respond.
Now, we have different types of reports and different types of testing. We're we're focusing on testing where there is a driver in the vehicle and the driver can take over as necessary. This isn't a fully autonomous vehicle. You do have to be in the driver's seat to do this. We're using these disengagement events as a proxy for assessing the reliability of the vehicles. Obviously, we don't have access to the software in these vehicles. If you worked at those companies, you could probably have more information. We obviously don't.
But they're a proxy because if you want our vehicle to be reliable, that means it needs to be operating as you intend within the environment. Any time you have to take over the AI for some reason, that could be a sign, "I t's not exactly operating as I intended." We can use it as a bit of a proxy. Again, it's not the best approximation, but it's still pretty good. Of course, I'm not the first one to think of this. This is actually an informal extension of some work I've done recently with my advisor, Yili Hong, and a bunch of other co-authors where we actually looked at this type of data from a recurrent events perspective.
I'm going to do a slightly different approach here. But there is a preprint of this article available if you want to check it out that does something similar. Let me go in and describe the data you for you real quick. I'm not going to be looking at every company doing testing. There's so many out there. I'm going to focus on one, and those would be events submitted by Waymo, which was Google's self driving car project. Now they're their own subsidiary entity. These are their annual reports. Let me define what we mean by disengagement events.
I'm in the driver's seat and if something's happening, I'm in autonomous mode and I need to take over and take over driving. That's a disengagement event. I disengage from autonomous mode. That could be for any reason. They, of course, need to report what that reason was. We're just using that as our proxy measure here. These annual reports are going to go all the way back to about 2015, 2014. That's when Waymo started participating in this program. The 2015 report actually contains data back to 2014. They start in the middle there.
Each report essentially covers the range from December of the previous year to November of the current year. The 2016 report would contain data from December 2015 up to November of 2016. That way, they have a month to process the previous year's numbers. There are two primary sources of data we're looking at in each report. The first one is going to list each incident that occurred, when it happened that could be as detailed as day and time, or it could just be the month.
Again, there's not a lot of good consistency across years. It's something we ran into. But they at least give some indication of when it happened. They might say where it happened and they can describe what happened. It could be very detailed or it could just be falling into a particular category that they give. Then the second part of data is going to list the the VIN or partial VIN of the vehicle, so the vehicle identification number. Something to identify the vehicle and how many autonomous miles that vehicle is driven that month. You might see later on when I show this data, there might be a bunch of zeros. Zero just means I either didn't drive that vehicle or I just didn't drive it in autonomous mode.
In either case, I was not doing active testing of the autonomous mode of the vehicle. Now, as I mentioned earlier, there was a bit of inconsistency. Prior to 2018, when they listed the disengagement events, they actually don't give the benefit of the vehicle. We don't know what vehicle was involved. We know how many autonomous miles it drove that month, but we have no idea what vehicle was involved. Starting in 2018, that information is now available. Now we can match vehicle to the incident, which means when we do this analysis, I'm going to do it at two different levels.
One is at an aggregate level where I'm going to be looking at each month all of the vehicles being tested at that time. Then looking at the incident rates overall in an aggregate measure. The second will be then I will zoom in at the vehicle level. I'll look at it by VIN. For that data, I'll only be going back to 2018. For the aggregate level, I can take all of it. Now, before we get through the analysis, actually wanted to show you some tools that JMP has available that allowed us to quickly process and accumulate this data. Again, to show you how easy it is and show off a few features in JMP. Some of them are really new, some of them have been around for a little while.
Let me start by showing you one thing that helped us, and that was being able to read in data from PDFs. Prior to 2018, a lot of these data were compiled in PDFs. Afterwards, they put them in an Excel file, which made it a lot easier to just copy and paste into a JMP table. But for those PDFs, how did we handle that? Let me give you an example using data from 2017. This is actually one of the best formatted reports we see from companies. Some summaries here, some tables here and there. This in appendix A is the data I'm looking at.
You can see here, this is the disengagement events. We have a cause, usually just a category here. They have the day, which is actually the month. A bit of a discrepancy there, the location and type. But this is basically just telling us how many disengagement events happen each month. Then we have a table here or a series of tables actually here at the back. This is showing us for each vehicle, in this case, it only gives partial event information. There is not a lot of information available in these early reports and then showing you how many autonomous miles were driven each month. How can we put this into JMP? Well, I could just copy and paste, but that's a bit tedious. We can do better than that.
Let me come here. I'm going to go to my File, I'm going to go to Open. There's my PDF. I'm going to click Open and JMP has a PDF import wizard. Awesome. Now what it's going to do is it's going to go through and look at each page and identify whatever tables it finds there. It's going to categorize them by the page and what the table is on that page. Of course, when you save it out, you can, of course, change the name.
Now, I don't want every table on every page. What I'm going to do is I'm going to go to this red triangle on this page and just say, "Ignore all the tables on this page. I don't want these." I'll say, ""Okay," I'll do the same here. It's a nice summary table, but it's not what I want. Then I start saying, " This is the data I want." Now, we're going to notice here, this is formatted pretty well. It's gotten the data I want. If I scroll to the next one, this is technically a continuation of the table from before. However, by default, JMP is going to assume that every table on each page is its own entity.
What I can do to tell JMP that actually this is just a continuation is to go to the table here on the page, click the red triangle and say for the number of rows to use as header, there actually are none. This is a way to tell JMP that actually that's a continuation of the previous table. We'll check in the data here, and it looks like it did it now. I'm going to check here at the bottom and I noticed, "Oh, I missed that October data. That's okay. I'm going to do a quick little stretch there and boom, I got it." That's okay. You can manipulate the tables. I f it didn't catch something, you can stretch and manipulate the table to adjust it. You can also add tables it didn't find.
In this case, I missed this. That's okay. I'm going to drag a box around it. Boom. There's a new table for you, JMP. I'm going to go in here. It's going to assume that there are some header rows. Actually, there are none. Okay, great. Now it's captured that part of the data. There's a bit of an empty cell here. That's just a formatting error because this is technically two lines, so they didn't put this at the top. It's okay. Easy fix on the back end. Now for these tables, what we notice if we go to that is it actually thinks, "Well, this is actually one table." Unfortunately, it's technically not correct because there are two tables, but it's an easy fix. I can simply go to each one and say, "It's actually not a continuation JMP. This actually has its own header."
It says, "Okay," and you can do that for each of these tables. I won't do it for all of them. I'm just doing this to illustrate. What we'd have to do is we'd probably end up with a bunch of tables here where we'll have to horizontally concatenate. That's just the way they decide to format in the report. But JMP has a lot of tools to help us with concatenating and putting tables together. But you can see this is a lot easier than trying to copy and paste this into JMP, making sure that the formatting is all good. JMP is going to do a lot of that for us.
Okay, another helpful tool that came out recently in JMP 2017 is, if you've probably heard of it, the JMP workflow. That was super helpful because obviously we have multiple reports over multiple years. We'd like to at least combine across all the years into two reports, one with the disengagement events, one with the mileage. What we did is we created an initial... We followed some steps to set up the table in a way that we can then concatenate them together into one table, and then we saved it into a workflow.
That's what I have demonstrated here. This is a workflow builder that we put together for that. I'm going to demonstrate it using this data set. This is particular for our mileage. What we have here is a table. This represents what we would have a raw output from one of the reports. Here we, of course, have it broken down by VIN number. We've got a lot of information here. We'd like to reformat this table. First thing I'm going to do, I'm just going to walk through each step. I'm not going to show too many details in each step. You'll see what they are. It's pretty self explanatory.
This first one is going to change the name of this column to vehicle. That way it matches a column in our concatenated table. I'm going to go over, I'm going to delete this total column. I don't need that. Then I'm going to do a stack across all the dates. You can see I've got that here. We conveniently called it stacked table, very informative. Now, one thing I need to do here, I put a pause here, that's the little stop sign. In. That's because I would usually need to go in and change the year.
Now, something I could do right now, I couldn't really figure out a way to get a variable, say year, that you could just fill out, put the year there, and then it automatically fill it in here. That's maybe something I can go to community.jmp.com , go onto the wish list and say, "Hey, it'd be nice if I could do this." But for right now, I just put in the years. It was pretty easy to do compared to doing this multiple times. Pretty straightforward. But again, I can also highlight for you how you can actually go in and adjust the script itself. You can go in and tailor this to your needs. What this is going to do is recode these so it shows the month and the year. I'll do that real quick. There we are.
The next step is going to take this. Right now, this is a category, it's a string, I want a number. That's what I do next. Now, this isn't pretty. This is just the number of seconds since some date in 1900, I believe. Obviously, that's not pretty. I'd like to show something more informative. That's what I do in the next step. Now, it shows the month and the year. I'm going to stop here. I'm not going to continue because at this point I'd have another table open. This next step would then concatenate the tables and then close off these intermediate tables. What I'm going to do is I'm going to reset stuff. I'll reset, click here. I'm going to reopen this table. I'm going to do this just so you can see how fast this goes.
Here I'm going to click over here, I'm going to click Play, I'm going to click Play again. Look how fast that was. Now, imagine doing this for multiple reports. How much faster that is than repeating the same steps over and over and over again. This workflow was really helpful in this situation.
Now, I'm going to close all these out because it's time to get into the analysis. Let's do that. I'm going to start with the aggregate level data. Here's my table. I compiled across all the data, all the time periods. We have the month, we have how many disengagement happened in that month. I got a column here for the cumulative total. I've got here how many autonomous models were driven. I got two columns here that I'm going to talk about in just a second. You'll have to just wait.
What I'm going to do is I'm going to go in, I'm going to go to Analyze. I'm going to go under Reliability and Survival, and then I'm going to scroll all the way down until I reach Reliability Growth. I'll click that. Now we have multiple tabs here. I'm only going to focus on these first two because these last two concern if I have multiple systems. I'll revisit those when we get to the actual vehicle information. For right now, let's pretend that we're looking at the whole AV system, the artificial intelligence system in these vehicles. Think of it as one big system.
There are two ways that I can assess this. One, I can do as time to event, essentially how many months until a certain event happened or days if we had that. Or I could do it via a particular time stamp. Basically, what was the time at which it occurred? I do have that type of formatted data. I have it at the month. The month is a fine timestamp. It just says in that month I had this many events happen. That's all I need to put in. I have all the data I need. I'll click OK.
Now, great thing about this is before you do any analysis, you should, of course, look at your data, visualize your data. It's nice because the first thing it does is it visualizes the data for you. Let's look at it. One thing we're looking at, we're looking at cumulative events over time. What we expect is a behavior where early on we might have what I'll call a burn in type period where I have a lot of events and it's happening. I'm tweaking the system, helping fix it, helping improve it.
Then ultimately, what I'd like to see is this plateau. I'd like it to increase and then flatten off. That tells me that my number of incidents is decreasing. If it goes completely flat, that's great. I have no more incidents whatsoever. I wish the were like that, it is not. But we can see patterns here in the data. Let's walk through. We have a burn in period here, early 2015, and then about mid 2016, we flatten off until about here. We see a little blip, about summer of 2016, something happens. We get a few more incidents. We level off again until we get to about here, about late spring of 2018.
Something else happened because we start going up again. They're not very steep. This one's a bit longer. Then we pretty much at here, we almost flattened out. We've reached the period where we're really having no incidents happen, essentially, till the end of 2020. Then something happens in 2021 and where we've reached essentially another burn in period, something's going on. Essentially what we've got is four phases, if you will, happening in the growth of the system. Something's changed two or three times to impact the reliability.
Another way to visualize this, I'll run this plot. This uses some data. I'm plotting again the cumulative total. I'm also plotting something what I call the empirical mean time between failures. It's a very simple metric to compute. It's just the inverse of the number of disengagements. It is a very ad hoc, naive way to try and estimate the mean time between incidents. But I plotted here so that you can see, you'll notice these four peaks that correspond to the bend in the curve. But there are four of them indicating these four places where something has changed in the system to affect its reliability.
What we can do then is try to figure out, what are those breakpoints? One way you could do that is the reliability growth platform has a way to fit a certain model. I'll pause here to talk about the model a bit. All of these are actually the same model with slight modifications. They're all what we call a non-homogeneous poisson process. That is a fancy way to describe a counting process. I'm counting something, but the rate at which the accounts might occur per unit time is changing over time. A Poisson process just means that at a constant rate, so the rate at which incidents would occur would to be constant, that would be equivalent to seeing a straight line.
It's very easy to model, but it's bad for reality because obviously we don't want the rate to stay the same. We would like it to decrease to essentially zero. That's why we have a non-homogeneous poisson process. We want it to change over time. Here we have a model where we can actually let JMP try and figure out a change point in the process. If I run it, what it's going to do is it's actually going to catch this big piece and say, "Hey, something really changed there. For most of it, it was the same thing, but after this point, it really changed." Now here it's only going to change one at a time. I have talked to the developer about, wouldn't it be nice if we could identify multiple change points?
A pparently that's a bit of an open research problem, so me and him might be working together to try and figure that out. But what I did is I essentially eyeballed it and said, "I think there are certain phases. I think there's about three or four phases, and I did it empirically, which is where you get this column." I'm going to run that script. Let me show you how I did it. I come here under redo, go to relaunch, and all I did was I added the phase column here. This tells you that there are different periods where the reliability might have changed significantly, excuse me, in some way.
If we think of that, we're going to look at the key metric here as the mean time between failure. We're going to see early on, so this is in months, this here is about three days, 4- 5, about a week, and this is about a day, day and a half. Early on, we have a bit of a low time. It's pretty frequent. We can also look here, I'll show you the intensity plot. That might be another thing to interpret. What we're looking for is we'd like the mean time between failures to be long. We'd like it to be a long time between incidents, ideally infinite. That means nothing ever happens, and our intensity to decrease.
What we're looking here is, we get a bit of a good start. About middle of 2016, we're doing really well. In fact, we get to about a week between an incident for any vehicle. There was a bit of a blip, but we primarily get back to where we were until we get to the end of 2021, where now it's essentially about a day between incidents for any vehicle. Something big happened here at the end of 2020 with these vehicles with this software system, if you will.
Again, you can see here with the intensity, you can almost do one curve and we get down to about six or seven incidents per month. Whereas here it's almost 30, essentially, once a day. We've been able to look into here and discover what's going on, at least at the aggregate level. Before we get to the vehicle level, I'm going to run one more graph that's looking at, we've got all these autonomous miles. Could it be that if I drive it more often, maybe I encounter more incidents? Could that have an effect? T here's a quick way to assess that. Just using a simple graph. We'll just plot autonomous miles versus the total disengagements.
We see here for are a few number of disengagements, that might be true. The more you drive, the more you might see. But in general, long term, not really. There's really no big, strong correlation between how many autonomous miles driven, how many engagements you see. There's something else going on. T hat's actually what we found in the paper that I mentioned earlier is that the mileage impact was very minimal.
Now, let's zoom in to the individual vehicle. We're not going to have all the data, even though I actually do have it here. But we're not going to have complete data for all of the vehicles. Let me break it down. I have the month, I have the vehicle identification number. Notice some of these, it's only partial. I have here what I call a VIN series. This is very empirical. I'm just taking the first four digits of the VIN. You'll see here, I'm going to scroll down a bit and we'll see. Let's see, maybe I will drag down a little bit. There we go.
Some of these VINs, a lot of them actually start with the same four digits, 2C4R. I'll call them the 2C4 series. There's a bunch of vehicles that have this as their starting one. This identifies a particular fleet of vehicles, at least from an empirical view. If I scroll down, we're going to run into a different series, which I'm going to call the SADH series. This is the one that was introduced about 2021. That's when I saw the Venn numbers change to the SADH designation.
Again, I have how many of miles, I have a starting month, when did that vehicle start? I'm going to use this to compute the time to events. First, I'm going to do a plot. I think this is the most informative plot you'll see for this analysis. What I've done here is I've essentially created for you a heat map. You can see I've got the heat map option select. Selected, I got for each vehicle and over time, essentially a cell, and that's just going to indicate, was I driven in autonomous mode anytime that month?
I got it color coded by the series. These vertical lines correspond to the transitions between those empirical phases I mentioned earlier. What this is telling us is basically, can we identify what might have caused those transitions? Here we see an initial series of vehicles, and it looks like there wasn't a big change in what vehicles were introduced here. Maybe there was a bit of a software upgrade for this particular series that may have introduced those new incidents. Here we see that a new series was introduced, a smaller number of vehicles, maybe pilot series. Then a bunch of them introduced about that same period where we saw the other transition.
Here, this seems to correspond to a new fleet of vehicles with maybe a slightly updated version of the software. Here, we see a clear distinction. Obviously, in 2021, a completely new series of vehicles was introduced. We have a bit of the old vehicles still there in the mixture, but most of them are the new vehicles. That probably explains why we got a new batch of new incidents. We got a burn in period for this new series of vehicles. This is cool because now we have a bit more explanation as to what was going on with the aggregate data, which is why it's important to have this information.
Now let's break it down by VIN. I have right here script to indicate we've got a table here and it's similar to the table I have previously. Notice some of these have been excluded and this is because if for that particular vehicle, the total number of incidents was less than three, the platform is not going to be able to fit a model for you because it needs at least three incidents per vehicle. That makes sense. I have only one or two, that's not really enough information to assess the reliability. If I have three or more, now we're talking, I can do something.
I also have the month since the start. I have some cumulative information there which month it started. I'm going to go ahead then and run the platform. Don't worry, I will show you how I did this. I'm going to go to the redo, relaunch. I'm going to get rid of that. That's some leftover stuff. I'm looking at one of these two last platforms. These are about multiple systems. Now we're thinking of each vehicle as its own system. The concurrent just means I'm going to run each vehicle one after the other.
That's not what's happening here. The vehicles are essentially being run in parallel. Multiple vehicles are being driven at a time. Here I need a column to indicate the time to event, in this case, how many months since the start until this many events happened. I have the system ID. The one thing that's not shown here is I actually took the VIN series and used that as a by variable, which is why we have the where VIN equals so and so. There's only going to be two because the one with a little asterisks has no information about incidents. That's because that was two earlier that was prior to being able to tie the VIN to the vehicle.
But they were there for completeness, I'm cancelled out of that. Now, what we're going to see here is a list of models that you could run. These first four, I'm not going to be able to do because I only have one phase. Essentially, the phase now corresponds to the VIN series. But there are two that I could run and the only difference is, do I want to run a model for all of them saying these are all part of an identical system? Makes sense. These are all vehicles, they probably run the same software. Maybe I can run a model for all of them. Or I can have a model where I fit it to each individual vehicle.
Before I run those models, let's take a look at this plot. Again, start with the visualization and what we see plotted is all the vehicles. Notice here there's a bit of a shallow slope to this. Essentially there's a bit of a steep curve, but then it levels off pretty quick. This is a pretty good sign of reliability going on here. I'm going to compare them, I'm going to scroll down to the next one, the set series. Now, initially the axes here, just so you know, when you run it next time, the axes are going to only go to the complete set of data. This would actually be a smaller range.
I fix them so they had the same range. You can clearly see that this is much steeper than this. Clearly, we have more incidents happening with this new series than this one. But we can do a quick model fit. I'm going to do the identical system. Again, it's a non-homogeneous poisson process. Although in this case, I'm going to ignore the estimates for right now, if you want to look at that, you can. I'm going to go straight to the mean time between failure and you'll notice that for all the months it's pretty much flat.
What it's essentially done is this is just a poisson process. The rate is constant, which is good for modeling, not so good in terms of assessing it. It's just saying across this whole time for this particular series, we pretty much reached for any one vehicle, a mean time of about five months between incidents. Now, let's compare that to what we saw with the aggregate where it was about a week, that's across any vehicle. It's just saying for any vehicle, it was about a week between an incident for any vehicle. Whereas this seems to be implying it's about five months for any one vehicle.
You can think of it as the running in parallel, so you can see it's staggered. Any one vehicle, it could be about five months. Again, this is an average, there's a lot of range between there. For one vehicle, it's a pretty long time. But in aggregate, they're probably staggered enough that it seems like it's about a week for any one vehicle. They can be consistent like that. But this is still pretty good. That's about five months between an incident, a disengagement event here for that series. If we run to the SADH series and do the same model, let me go here. There we see clearly increasing. I'm going to hide that. If we look at this, it says, "No," early on we probably had about two months. That's a bit of a start there. But we've dropped to less than a month, almost two to three weeks between incidents.
Clearly, there's a bit more work to do on this series. Again, it was just introduced, so this is probably more of the burn in phase. If we get the 2022 data, we might start to see it level off like it did in the previous series. This actually might flip and be more of a level curve. That's about all I want to show for these platforms. I can show you some of the individual distinct systems, but there are a lot of systems here and so it's going to get crowded very quickly. There's a plot for each one. There's estimates for each one. You can look at the mean time between failure for each one.
If there are particular vehicles you wanted to call out and see how they might differ, this is what you can do. You can see some increase, some decrease, but overall more or less flat. You can also look at intensity plots. If you find that more interpretable than the mean time between failure, you have other metrics that you can incorporate here. Okay, t hat's all I want to show for this platform. Now, of course, there's data I didn't include here. For example, we could break it down by cause.
For some of this data, cause might be, I just need to take over because it was getting too close to the side of the the road. Or maybe the car stopped at the stop sign, did what it was supposed to, started rolling, and some other driver blew through the stop sign coming the other way. In which case, maybe that might not necessarily be a reliability hit. The car did what it was supposed to. Somebody else wasn't. It'd be interesting to break it down by that, also by location. The number of incidents, you get more when you're in the city, maybe on the highway, something like that.
Real quick, we should look at the mileage impact. Again, same information, one or two incidents, sure, that might change. But overall it's going to be flat. The mileage impact on the incident rate is minimal. Of course, this is just one of many platforms available in the reliability suite. You can see there's a ton of options, very flexible for helping assess reliability. Again, that's all I have to show you. Hopefully, I've been able to demonstrate for you how well JMP can help initiate discovery and analysis. Hopefully, you discovered a lot of things about this particular company's autonomous vehicles. I hope you enjoy the rest of the conference. Thank you.