I'm  a  senior  research statistician  developer

here  in  the  Design  of  Experiments and  Reliability  group

at  JMP  Statistical  Discovery.

Its  quite  a  mouthful.

A s  you  probably  guess from  the  title  of  my  talk,

I'm  going  to  be  talking  about autonomous  vehicles,

and  specifically  how  we  can probably  assess  the  reliability

of  autonomous  vehicles

using  some  of  the  reliability platforms  like  JMP,

so  that  may  be  one  day we'll  be  like  this  fine  gentleman  here

relaxing  in  our  autonomous  vehicles as  they  take  us  wherever  we  want  to  go

without  being  scared that  we  won't  end  up  there.

L et's  get  into  that.

O f  course,  autonomous  vehicles are  fast  becoming  a  reality.

We  have  a  lot  of  companies  now  involved in  extensive  testing  of  these  vehicles.

Now,  of  course, some  of  that  testing  early  on

is  basic  testing  of  the  software, testing  in  simulated  environments,

but  ultimately, in  order  to  have  a  very  reliable  vehicle,

you  need  to  test  it  out  on  the  roads,

and  so  to  help  out  with  that, a  couple  of  years  back,

the  California  Department of  Motor  Vehicles

put  together  a  testing  program that  would  allow  these  companies

to  opt  in  and  test  their vehicles  on  their  roads.

As  part  of  that  arrangement,  though,

every  year  these  companies have  to  provide  a  detailed  report

on  any  disengagement, and  heaven  forbid,  crashes

involving  their autonomous  vehicles  to  the  DMV.

Now,  because  the  DMV is  a  federal  institution,

these  reports  are  actually publicly  available  upon  request.

In  fact,  I'll  pull  up  a  link  here real  quick  to  the  California  DMV  website

where  you  would  be  able  to  go  to.

In  this  case,  you  could  access reports  from  the  last  two  years.

If  you  wanted  more, you  just  send  an  email  to  this  link

there  pretty  quick on  getting  back  to  you.

Now,  already  a  lot  of  people are  using  this  data  in  academic  research.

As  a  quick  example, I'll  click  on  this  link  here

to  a  paper  on  a  project that  I  was  personally  involved  in

with  my  former  advisor and  one  of  his  new PhD  students,

in  this  case,  analyzing  it  using what's  called,  recurrent  events  data.

I'll  talk  a  little  bit about  that  more  later.

But  again,  there's  already  research looking  at  how  can  we  use  this  data

to  assess  the  reliability of  autonomous  vehicles,

maybe  use  it  to  introduce new  methods  and  so  forth.

I t's  already  been starting  to  be  widely  used.

Now  for  this  particular  talk, I'll  be  looking  at  disengagement  reports,

using  that  as  a  way  to  measure  reliability

because  of  course, we  can't  access  the  actual  software,

that's  all  proprietary  for  the  company.

But  through  the  use of  these  disengagement  events,

we  can  infer  the  reliability of  the  vehicles  for  a  particular  company.

A gain,  I'll  be  showing  off,

this  is  a  JMP statistical  discovery  conference,

so  I'll  be  showing  off  some  tools  in  JMP that  allow  us  to  get  that  information.

Wi th  that,  let's  quickly  talk  about what  data  are  we  looking  at?

W hat  are  in  these  reports?

F irst,  these  are  annual  reports,

and  I'm  going  to  be  focusing on  a  single  company,

those  coming  from  Waymo,

which  is  formerly  this Google  self-driving  car  project.

T hese  are  reports  submitted by  Waymo  to  the  California  DMV.

Now,  I've  been  talking  about a  disengagement  event.

What   do  I  mean  by  that?

Well,  there's  two  types  of  testing.

The  first  involves there  being a  driver  in  the  seat,

the  vehicle  actually  can't  operate without  a  driver  in  the  seat.

But  at  some  point, they  can  turn  on  autonomous  mode,

and  if  anything  happens where  the  driver  has  to  take  over,

that  was  not  planned as  part  of  the  testing,

that's  a  disengagement  event.

The  other  type  of  testing,  of  course, is  a  completely  driverless  vehicle.

We're  not  going  to  focus  on  those.

We're  just  going  to  focus on  the  driver  necessary  systems.

Now,  these  annual  reports go  all  the  way  back  to  about  2015,

even  containing  information  back  to  2014,

which  is  about  when  this  whole California  DMV  program  started.

Waymo  was  actually  one  of  the  first companies  to  get  involved  in  this.

T hese  reports  contain, first  of  all,  they  contain  data

typically  from  December of  the  previous  year,

to  November  of  the  current  year of  the  report,

the  exception  being  the  2015,

which  actually  goes  back a  bit  further  into  2014.

F or  example,  the  2016  report  would have  data  from  December  of  2015,

all  the  way  up  to  November  of  2016.

T here's  two  general types  of  data  present.

The  first  is  going  to  tell  you  all about  the  disengagement  event.

What  was  the  cause? Where  did  it  happen?

S ometimes it  might  give  you  the  day and  the  VIN  of  the  vehicle

involved  in  the  incident.

The  other  type  of  report  is,

how  many  miles  were  driven in  autonomous  mode

for  a  vehicle,  for  a  particular  month.

Now,  you  may  have  noticed  that I  said, in  the  disengagement  event  report,

sometimes  we  know the  date  and  the  event.

Yes,  that's  right.

It's  not  very  consistent across  the  entire  time.

In  fact,  that  information  actually wasn't  available  until  2018.

Prior  to  that,  we  only  know  what  the  event was  and  how  many  happened  within  a  month.

Starting  in  2018, they  started  providing  that  information,

most  likely  because  the  DMV encouraged  them  to  do  so.

W hat  it  means  then  is  that

you  actually  have  two  levels of  resolution  with  our  data.

T he  first  is  at  a  month  level,

and  for  that  we  can  go  all  the  way back  to  when  testing  began.

I'm  going  to  be  looking at  that  type  of  data

using  our  reliability  growth  platform.

T he  other  type  of  data is  actually  daily  and  at  the  VIN  level,

so  individual  vehicle  level.

I'll  be  using  the  recurrent  events platform  to  analyze  that.

Before  I continue  further, let  me  show  you  where  those  are.

You'll  find  them  under  the  Analyze  Menu, go  down  to  Reliability  and  Survival.

You'll  notice  a  ton  of  platforms  here.

The  one  I'll  be  highlighting are  Recurrence  Analysis.

Th is  is,  we  got  individuals, in  this  case,  vehicles,

that  can  experience a  recurring  event  over  time,

in  this  case,  a  disengagement  event,

so  we'll  be  using  that  for  the  VIN  level, and  then  the  reliability  growth

I'll  be  using  to  analyze the  monthly  aggregate  data.

The  same  idea,

although  reliability  growth is  more  for  if  I  have  a  big  system

that  could  encounter m ultiple  events,

sometimes  they  have  to  be  repaired as  part  of  the  event,

so  they're  experiencing  it  over  time.

It  might  go  through  different  phases  in,

so  the  underlying   process or  software  might  be  evolving  over  time.

That's  something  that  the  reliability growth  platform  is  intended  to  handle.

Now  this  could  also  handle the  VIN  information  as  well.

It's  fine  with  that.

I'm  just  picking  these  two because  I  want  to  show  off  the  breadth

of  the  stuff  that  we  have  here  at  JMP.

While  I'm  doing  that,

I  also  want  to  show  you  a  bit  of  how  we use  some  JMP  tools  to  compile  the  data.

Obviously,  there's  a  lot  of  reports,

so  how  do  we  compile  all that  together  into  a  single  report?

I'll  touch  briefly  on  some  of  the  tools that  we  used  in  JMP  to  help  with  that.

L et's  dive  right  into  that.

F irst  I'll  talk  about  reading in  some  of  the  earlier  reports.

Y ou've  already  noticed  there's  a  bit of  inconsistency  in  what's  reported.

There's  also  a  bit  of  inconsistency in  the  format  of  the  reporting,

so  some  of  the  very  early  reports were  in  the  forms  of  PDF.

Let  me  show  you  an  example using  the  2017  report.

W e  have  a  nice  report  here  from  Waymo.

It's  very  well  formatted.

In  fact,  this  is  probably  one of  the  better,  nicer  reports

that  we've  seen  amongst  the  companies that  were  participating  at  this  time.

A  bunch  of  tables  in  here,

The  ones  we  really  want  are  the  ones here  at  the  end  in  the  appendices.

This  is  an  events  report.

Notice  days  a  bit  of  a  misnomer.

It's  actually  the  month  where  it  happened,

some  information  about  the  operation, what  caused  it.

We  definitely  want  this  information.

This  is  a  table  that  we  want.

We  also  want  a  table  here  at  the  end

which  shows  the  autonomous  miles  for  each event,  in  this  case  the  last  four.

Don't  even  get  the  complete  Vin for  each  vehicle  for  each  month.

Now,  of  course  it's  in  the  PDF,

so  we  really  don't  want  to  have to  go  in  and  manually  type  these  in.

It  would  be  nice  if  JMP  had  a  way to  read  in  tables  from  a  PDF.

Well,  you're  in  luck because  I  can  go  here  and  do  open.

There's  my  pdf  I've  marked  it  to, Select  all  files,  not  just  JMP  files.

There's  my  report, I'm  going  to  click  Open

and  we  have  our  fancy PDF  reader.

Now,  what  it's  gone  through  is  it's  trying to  identify  everything  that's  a  table

in  that  report  and  it's done  a  pretty  good  job.

Scroll  through  here.

You  can  see  captured  most  of  those  tables.

In  fact,  probably  too  many tables  don't  need  all  of  these.

You  can  scroll  through  them  individually using  this  dropdown  menu

and  you  can  see  it's  labeled  by  page and  the  table  number  on  that  page.

I  don't  need  all  of  these  tables as  I  mentioned  before,

I  just  want the  ones  that  are  in  the  appendices.

That's  easy.

I'll  go  to  the  Red  Triangle  menu that  suddenly  appeared  on  the  page

and  say,  Just  ignore  all the  tables  on  this  page.

I'll  scroll  down  and  do the  same  thing  for  this  page.

Okay,  now  I've  got  the  tables I'm  interested  in.

We  still  need  a  bit  of  tweaking  to  do because  as  you'll  see,

they're  essentially  each  table on  each  page  is  its  own  entity.

So  these  are  actually all  the  same  table.

How  do  I  tell  JMP  to  do  that?

Well,  now  I  can  go  to  the Red  Triangle  menu  on  the  table.

I'm  going  to  scroll  down  here to  number  of  rows  to  use  this  header

and  tell  it  to  use  none.

Now  what  this  means  is if  there's  a  table  prior  to  it

that's  telling  JMP  that,  hey,

that's  actually  part of  the  previous  table.

That's  why  there's  no  header  on  this.

I'm  going  to  click  zero.

You'll  notice  that  table, there's  one  less.

You've  also  noticed  it's  added two  columns  in  this  case.

What  page  is  that  table  occur and  what  table  on  that  page.

If  you  wanted  that  information, JMP  automatically  provides  it.

In  this  case,  I  don't  really  need  it,

but  I  can  delete  it  later  after I'm  done  reading  these  then.

We  also  notice we  missed  a  bit  of  that  table

probably  because  the  formatting  change

notice  it  was  able  to  pretty  easily  find what  the  roads  were  for  this  table,

but  for  this  one,  not  so  much.

Mainly  because  of  this  guy.

Some  we  just  didn't  put  it  just  right, so  it  didn't  fit  the  pattern.

That's  okay,  I'm  going  to  click, I'm  going  to  drag  around

and  that  tells  JMP, "Hey,  there's  you  a  new  table",

and  I  can  go  in  right  now  thinks  it's a  zone  table.

I  can  just  tell  it no,  it's  part  of  the  previous  one.

There  we  are.

I'm  going  to  scroll  through  and  yeah,  it missed  some  of  this  information.

But  again,  notice  it's  trying to  maintain  that  formatting,

but  because  of  the  way  it  was  input, it   missed  that  last  row.

That's  okay  I  can  input  that information  later.  It's  not  a  big  deal.

If  we  go  to  this  next  page  in  this case  JMPs   done  the  opposite.

It's  actually  concatenated  these two  together,  which  makes  sense.

It   sees  these  are pretty  close  together.

Maybe  they  are  the  same  table.

Unfortunately,  they  are  not.

As  you  can  see, they're  two  distinct  tables  here,

whereas here  they're  concatenated  together.

I  need  to  tell  JMP.

No,  that's  actually  a  different  table.

That's  pretty  easy.

Just  say  how  many  rows.

No,  there's  actually  is  a  header  row,

and  that  tells  JMP, that's  a  different  table.

Now  it's  created  a  different  table.

I  would  have  to  repeat that  for  a  lot  of  these.

Unfortunately,   the  way  they've  input  this,

I'm  going  to  have  to  do a  bit  of  work  later  on

to   get them  in  the  right  format.

But  that's  okay.

There  are  other  tools  and  JMP  to  do  that.

For  right  now  for  the  sake  of  time I'm  not  going  to  go  into  that.

I  just  want  to  show  you  how  we could  read  it  in  from  a  PDF.

Now,  thankfully,  starting  in  about  2018,

they  started  using  Excel for  their  reports.

It  was  a  lot  easier  to  just  copy and  paste  directly  into  JMP.

Now  that  we've  got  the  raw  data  tables  in,

I'm  not  done  yet  because  I  would  like to  reformat  some  of  these  data  tables

to  get  them  into  a  nice  common  formatting

that  can  then  use  to  concatenate  them all  together  into  a  single  table.

Now  I'll  show  you   how  I  did  that,

and  it's  actually  quite  a  lot of  things  that  I  want  to  do.

Here's  an  example of  the  Waymo  2022  report.

We  in  this  case,

we  have  the  date  of  the  incident, the  VIN  number  of  the  vehicle  involved.

This  is  just  extra  information that  I  actually  don't  want  to  keep.

It's  just  saying,  can  I  drive  by  itself?

Is  it  completely  driverless? No.

Yes,  sir. Of  course  there's  a  driver  present.

Who  initiated  it. I'm  not  going  to  use  that  information.

I'm  going  to  keep  the  location and  the  description.

This  is  essentially  the  cause.

I  will  probably  rename some  of  those  columns.

I  also  need  to  summarize  over  this  table,

because  from  this  I'd  like  to  create an  aggregate  events

so  I  can  use  it with  the  monthly  aggregate  data.

I would  need  to  aggregate  summarize  over the  VIN  numbers  over  the  dates  by  month.

Do  all  that, so if  you  bear  with  me,

I've  a  lot  of  about  five  minutes or  so  to   do  that.

I  know  it's  a  bit  dangerous  to  kind of  do  that  live,  but  just  bear  with  me.

First  thing  I  need  to  do  is I  need  to  open  this  guy

and  then  I need  to  click  this  button.

Okay.  Give  me.

Let  me  take  a  little  break  here. That  was  a  lot.

Thank  you  for  bearing  with  me  on  that.

As  you  can  see,  thanks  to  something  new called  the  JMP  workflow,

I  was  able  to  accomplish  all of  that  in  a  fraction  of  a  second.

Well,  maybe  a  little  over  a  second, but  still  a  lot  faster.

I've  got  a  data  table here  aggregated  by  month.

I got  the  location,  the  cores, how  many  events  happened  in  that?

I  can  relabel  some  of  that  and  I've  got my  individual  data  table

that   has  the  date,  the  VIN  and  the actual  individual  events.

I've  got  that  set  up.

Now  I  can  save  those  in  different locations  to  be  concatenated  later.

What  is  this  new  workflow?

Well,  let  me  open  up  a  new  one  for  you just  so  you  can  see  it  if  can  get  there.

There  we  go.

This  is  what  you  would  start  off  with.

What  you  do  is  you  click  this  red  record button  and  on  an  individual  data  table  you

would  then  it  would  then  record  the actions  you're  taking  on  that  data  table.

This  is  very  useful  if  you  have  an  this like  I  have  here,  I  have  a  lot  of  reports.

I'm  going  to  be  doing  the  same thing  for  each  report.

Then  having  to  instead  repeat  all  those actions  for  each  of  those  tables,

I  just  record  it  for  one  table and  then  save  it  as  a  workflow.

Then  when  I  ran  it, what  it  would  do  is  it  would  open  up

a  window  saying,  "Hey,  couldn't find  that  original  data  table.

Can  you  show  me  what  the  new  one  is?"

Select  it  and  it  runs  it.

Now,  in  this  case,

it  didn't  do  it  here  because  I  already had  it  primed  for  this  data  table.

But  typically  if  the  data  table  has changed  in  the  name,

so  long  as  it  keeps  the  same  formatting,

that's  the  only  thing you  have  to  worry  about.

If  a  column  did  change a  name,  that's  okay.

JMP  also  knows  to  ask,

"Hey,  couldn't  find  that  column. Can  you  give  me  a  replacement?"

This  is  very  helpful.

Let's  say  you  have  a  daily  report that  you  have  to  collect  data  on

so  you  would get  a  data  table  of  that  report.

Maybe  there's  a  couple  cleaning  things you  need  to  do  on  that  data  table.

Save  a  workflow,  open  that  data  table,

click  Run,  select  the  right data  table  and  you're  done.

This  is  a  great  time  saver.

This  really  helped  save  us  a  lot  of  time

when  we  were  compiling these  data  together.

That  was  something  that  was just  recently  introduced  in  JMP  17.

I'm  sure  you're  going  to  love  that.

Let  me  exit  out  of  all  of  these. I've  already  got  those  tables  done,

and  want  to  make  sure we're  going  to  go  on  to  the  analysis

because  want to  make  sure  we  have  time  to  do  that.

Let's  start  with  the monthly  aggregate  level.

Here  I've  compiled  it  across   all  the  time  periods

up  until  the  most  recent,  that  being  2022.

Let's  quickly  walk through  what  I've  got  here.

I've  got  date  in  this  case, it's  the  month.

How  many  vehicles  are  actively being  tested  in  that  period?

How  many  total  autonomous  miles were  driven  in  that  month?

This  is   how  many different  event  types  happened.

This  is  the  total  number of  disengagement  events  that  occurred.

Let's  do  a  quick  visual look  at  this.

JMP  is  all  about  graphs and  visual  exploration.

Let's  look  at  that.

Here  I've  got  plotted  the  cumulative number  of  events  over  time

and  the  individual  events  and  we  notice is  an  interesting  little  pattern  here.

There  seem  to  be  there's  clear  spikes and  then  drops  and  it  seems  to  happen

about  4  or  5  times  where we  see  this  spikes.

You  can  also   see that  with  the  cumulative  data.

Ideally  what  you  would  like with  this  type  of  data,

there'd  be  a  bit  of  an  increase at  the  beginning

and  then it's  going  to  flatten  out.

Ideally,  it  becomes  completely  flat.

If  you  do  that,  you  have  achieved the  perfect  autonomous  vehicle.

You  have  no  more  disengagement incidents  to  worry  about.

But  of  course  that's never  going  to  happen.

You  just  want  it  to  be as  flat  as  possible.

What  we're  seeing,  of  course, is  a  bit  of  a  rise  or  a  burn  in  period,

if  you  will,  and  then  it's starting  to  flat  out.

But  we  have  that  repeated  a  couple of  times  most  significantly  here.

That's  a  pretty  big  jump  there.

Of  course,  you  can  see that  correlating  here.

We've  got  our  burn  in  period.

Then  as  we  drop in  the  number  of  incidents,

our  curve  up top  is  going  to  flatten  out.

This  is  going  to  be  the  key  piece  of information  we are going  to  be  looking  at

in  our  reliability  platforms.

We've   got that  information  there.

Now  I'll  go  ahead  and  quickly address  one  question  that  might  come  up.

Maybe  you  might  think,  well,

maybe  the  more  you  drive  it, the  more  incidents  you  might  have.

Maybe  we  should  try  and  account for  that  and  I  have.

Here's  another  graph  where  I've  looked at  the  correlation  between,  in  this  case,

the  log  of  the  mileage  and  the  log of  the  total  disengagement  events.

There  is  a  bit  of  a  trend.

It's  a  very  small  one.

In  fact,  the  R²  is  not  good  at  all.

There's  a  lot  of  spread  in  the  data.

Here's  actually  the  prediction  equation with  its  individual  prediction  boundaries,

there's  a  lot  going  on.

There's  plenty  of  room for  a  flat  line  there.

This  is  just   saying  that  there

might  be  something  like  that  happening, but  it's  clearly  not  the  driving  force.

Yes,  if  you  drive  it  a  little  bit  more, you  might  experience  a  few  more  incidents

that   make  sense.

But  it's  clearly  not  a  big  driving  force.

Something  else  is  clearly  going  on.

Hopefully  that  helps  address a  side  question  that  might  come  up.

Well,  let's  get  into  the  actual reliability  analysis.

I'm  going  to  run  this  script  here,

and  this  is  going  to  run that  reliability  growth  platform.

Don't  worry.

After  I've  run  it,  I'm  going  to  click  here under  the  Red  Triangle  redo  relaunch.

This  brings  up  the  initial  launch.

This  is  what  the  initial launch  window  looks  like.

You  have   four  different ways  of  entering  the  data.

The  first  is  a  time  to  event.

If  you  had  recorded  here's  how  many days  or  months  up  until  an  event  happened,

you  would  use  this  one.

Then  you  could  of  course, maybe  multiple  happened  at  that  time.

You  could  put  that  there.

We  have  the  dates,  which  is  what  I'll  be using  because  we  instead  of  an  actual

timed  event  we  just  had  saying  that  in this  month  this  many  events  happen.

We  have  a  timestamp and  an  event  count  for  that.

You  also  have  it if  you  have  individual  information,

maybe  we're  doing  concurrent testing  or  testing  in  parallel.

You  can  do  that  information  here.

Notice  there's  a  spot  for  the  system I D.

If  I  were  looking  at  the   individual  event  information,

this  is  where  I  would  go.

But  for  right  now,  we're  going to  stick  with  the  dates.

I'll  revisit  the  phase  stuff  in  a  moment.

You'll  notice  it's  plotted  that  cumulative curve,  which  we  saw  earlier.

There  it  is  right  there.

Now  the  metric  of  interest  here  is what's  called  mean  time  between  failures.

It's  clearly  oriented towards  reliability  data

where  we're  looking  at  failures.

Think  of  this  as  what's  the  average  month

between  incidents, disengagement  incidents.

There's  a  bit  of  a  non-parametric

measure  here  that's  just  sort  of  a  moving window,  looking  at  it  over  time.

But  you  can  get  a  sense  of  how  good we  get,  maybe  how  poor  we  get.

I'm  actually  going  to  fit  a  model.

The  model  I'll  be  fitting is  here  under  fit  model.

This  was  a  bit  of  a  mouthful.

It's  a  piecewise  viable  in change  point  detection.

Let  me  break  that  down  for  you.

The  typical  model  used  in  modeling this  type  of  account  type  data

is  what's  called  a  Poisson  process.

What  that's  essentially  says  is that  the  rate  at  which  incidents  occur

in  this  case  rate  per  month is  constant  over  time.

That  would  be  the  equivalent of  a  straight  line  here

from  a  mathematical perspective,  that's  a  nice  model.

It's  very  easy  to  analyze ,

from  a  practical  perspective that  is  a  horrible  model

because  we  never  get  to  improve.

Our  rate  of  incident  is  always  the  same.

We  don't  want  a  constant  rate.

We  want  the  rate  to  go to  zero,  preferably.

That's  what  I  mean by  that  flattening  of  the  curve.

What  we  want  is  what's  called a  Non-homogeneous  Poisson  process,

fancy term  meaning  the  rate  changes  over  time.

The  weibel  part  says  how  exactly does  it  change  over  time?

It  consists  of  two  parts.

There's  an  initial  rate  lambda Think  of  that  as  like  a  constant  out  front

and  multiply  that  by  time raised  to  some  power  beta.

That  value  beta  is  going  to  tell you  how  the  rate  changes.

If  beta  is  greater  than  one, my  rate  is  increasing  over  time.

That's  bad,  we  don't  want  that.

That  rate  were  zero.

We're  at  a  Poisson  process. It's  constant,  we  also   don't  want  that.

What  we're  looking  for  is  a  value  of  beta that's  between  0  and  1  saying,  you  know,

okay,  it'll  increase  a  little  bit, but  then  it's  going  to  start  to  die  off,

it's  going  to  slow  down  getting  us that  leveling  off  that  we  want.

When  we  estimate  these, we're  looking  at  a  value  between  0  to  1.

We're  also  looking  visually for  that  flattening  off.

For  the  meantime,  between  failures, what  we  want  is  that  to  get  very  large,

ideally  it'd  be  infinite, meaning  it'll  never  happen,

but  of  course  that  can't  happen.

We  just  want  very  large mean  times  between  failures.

I'm  going  to  fit  that  model  here.

I'm  going  to  use  a  change  point

because  that's  saying  that  at  some  point

maybe  that  value  of  beta  in  my model  is  going  to  change  significantly,

mainly  because  of  this  guy  here.

I'm  going  to  run  that  because it  looks  so  different.

Yes,  according  to  this,

there  is  a  change  point  right  there, a  pretty  significant  one.

If  you  look  graphically,

that's  not  a  great  fit, but  it's  not  a  really  bad  fit  either.

But  it  is  detecting  that  something significant  happened  in  2021.

If  we  go  to  this  plot,

this  is  showing  you  how  does  that  mean time  between  failure  changed  over  time

and  it's  doing  it  for  each in  this  case  phase.

This  is  sort  of  the  empirical phase  based  on  this  change  point.

In  the  first  phase,  it  says,  well, we  got  to  about  as  good  as  0.15  months.

That's  roughly  five  days between  incidents,  not  too  bad.

Roughly  a  week.

Again,  this  is  across a  fleet  of  vehicles.

For  any  individual  vehicle, it  could  be  longer  or  shorter.

But  for  the  fleet,

essentially  the  software  itself,  about a  week  between  disengagement  incidents.

After  2021, it  drops  significantly  to  about  5%.

That's  almost  a  day  and  a  half.

That's  quite  a  drop.

We  can  look  at  the  parameter  estimates. There's  my  data,  that's  just  my  initial.

Notice  For  beta  one  it's  less  than  one,  we're  doing  great.

For  the  second  phase,  not  so  much.

In  fact  we  can  see  maybe  we could  do  a  little  bit  better.

Notice  there  were  about  four  of  those, so  maybe  I  can  invoke

some  sort  of  empirical  grouping  on  that,

which  is what  I've  done  in  the  other  script.

It's  going  to  run  this.

All  I  did  here. I'll  do  the  redo,  relaunch?

I  just  added  a  column  called Empirical  Phase

that's  just  got  a  bunch  of  letters  in  it

that  just  distinguish the  phases  from  each  other.

It's  a  categorical  factor.

That's  all  it  is,  and  that's  what these  vertical  lines  represent.

This  is  a  combination  of  me repeatedly  using  the  change  point.

It  only  detects  one  change  point.

You  can  do  a  trick  where  I  can  select  some of  the  data,  hide  and  exclude  it,

and  then  do  it  again.

That's   a  way  for  it  to  drive detect  different  change  points.

Some  of  this  is  also just  me  eyeballing  it.

Full  confession.

Now  for  this  one,  the  model  all fit  is  called  a  piecewise  Weeble.

The  same  thing  is  what  we  did  before.

But  instead  of  it  detecting the  change  point,  it's  saying,

I've  already  got  those  in  for  you.

Each  time  you  see  one  of  these  changes,

these  different  phases, that  value  of  beta  is  going  to  change.

It's  going  to  connect  them  all  together.

But  that  value  of  beta  might  change,

indicating  a  change  in  the  process overall  based  on  these  phases.

This  is  a  much  better  fit.

It's  maybe  it's  a  bit of  a  too  much  overfitting.

I  will  acknowledge  that.

But  if  we  look  at  this, what  it's  essentially  saying  is

for  that  first initial  period  of  changes  in  the  software,

were  you  about got  almost  as  good  as  0.4  months.

That's  almost  12  days  between  incidents.

It's  still  saying  that  we  got  really good  up  until  2021  and  then  we've  dropped.

Clearly  something  is going  on  in  around  2021.

Something  significant  has  changed

in  the  underlying  software  across the  fleet  of  vehicles  that's  dropped.

That's  increased  its  incidence  rate.

There's  those  parameter  estimates.

Again,  notice  I'm  going to  ignore  that  one.

That's  a  burning  period.

A  lot  of  these  are  less  than  one

until  we  get  to  the  season  D's, which  are  afterwards.

Clearly  something's  happening because  it's  larger  than  one.

We're  already  we've got  a  bit  of  a  story.

Something  clearly  happened  in  2021 to  change  the  incident  rate.

Maybe  when  we  look  at  the  individual  data, we'll  figure  out  what's  going  on.

Let  me  clear  out  all  of  this  and  let's move  over  to  the  individual  vehicles.

Quick  summary. We  have  our  date.

The  actual  date  of  the  incident,

the  vehicle  identification  number,  the VIN this  VIN  group,  it's  nothing  fancy.

It's  just  the  first four  digits  of  the VIN.

You're  going  to  see  why  I've created  a  group  in  a  moment.

I've  got  the  location,  the  cause.

I've  reduced  the  text  of  the  cause. It  was  very  wordy.

I  just  reduced  it  to  a  basic  description.

We'll  cover  that  in  just  a  moment.

The  starting  and  ending  month.

I've  got  this  information basically  telling  me

how  long  was this  vehicle  under  test?

This  particular  vehicle,

it  was  basically  being  tested in  autonomous  mode  on  and  off

from  December  of  2017  to  about  November, maybe  October  of  2019.

We're  going  to  need  that  information.

I'm  going  to  first  look  at  some basic  some  graphical  summary.

First  thing  I'm  going  to  do, I'm  going  to  run  this.

This  is  just  going to  summarize  over  the  causes.

Let  me  look  at  the  total  number

of  disengagement  events for  each  vehicle  each  month.

Then  I'm  going  to  run  this  plot

and  you're  going  to  see it's  going  to  look  interesting,

like  coral  under the  sea,  waving  about  very  pretty.

A  lot  of  information  going  on.

But  the  basic  thing  I  want  to  tell  you  is

the  way  to  interpret  this  is   like we  did  with  the  cumulative  events  graph.

That's  what  these  are.

You  want  to  see  a  bit  of  a  burn  in, but  it  ultimately  flattening  off.

Now  I've  got  a  colored  by  individual  then, which  is  why  you  don't  see  the  labels.

There'd  be  way  too  many.

I'm  going  to  color  by  a  different variable  this  time.

We're  going  to  color  by  the  VIN  group.

Come  here  under  color.

Again,  that's  just  the  first  four digits  of  the  VIN  number,

which  clued  me  in  because  let  me  throw in  the  legend  so  you  can  actually  see  it.

Show  legend.

I'm  done.

Notice  there  was  a  significant  change in  the  VIN  number

indicating  maybe a  significant  change  in  the  vehicle.

Notice  when  the  change  occurred right  around  2021.

Very  interesting.

You  can  also  see  a  general  trend in  the  behavior  for  these  early  vehicles.

Now,  some  of  these  probably actually  started  earlier  than  this.

There's  a  bit  of  censoring  involved, but  already  they're  pretty  flat.

We've  got  a  bit  of  burn in  and  then  some  flattening  off.

That's  great, that's  what  we  want  to  see.

These  guys,  there's  almost a  straight  line  up.

We've  got  a  lot  of  incidents  happening in  a  very  short  period  of  time.

Now,  later  on  here,  later  in  2022,  there are  starting  to  lean  off  a  little  bit.

We've  got  some  that  did  start  and  they're starting  to  be  pretty  shallow  curve.

That's  good.

But  clearly  something's  happened  here.

This  seems  to  indicate that  a  different  vehicle

and  perhaps  a  different  type of  software  was  introduced  about  2021.

It's  having  some  issues.

It's  struggling  a  bit, or  at  least  initially  did.

About  here  it's  performing  about  as well  as  we  saw  early  on.

They've  they've  done better  in  fixing  it.

They   introduced  something new  and  had  a  bunch  of  bugs.

That's  what  happens  when you  introduce  new  things.

Let  me  exit  out  of  this.

Let's  look  at  maybe  something about  a  different  type  of  cause.

Now,  remember,  in  the  previous  video,

we  didn't  really  look  at  the  cause  because it  was  aggregated  across  all  the  vehicles.

But  let's  look  into  that  real  quick.

Maybe  there  might  be something  going  on  there.

Before  we  actually  get into  the  recurrence  data.

Here  I've  just  done a  basic  bar  chart  of  the  causes,

and  a  lot  of  them  are things  you  might  expect  to  be  a  cause.

An  unwanted  maneuver

equals  doing  something  you  don't  want it  to  do,  a  perception  discrepancy.

The  vehicle  didn't see  something  it  should  have

or  thought  it  saw something  that  wasn't  there.

Then  there's,  of  course,  software, a  general  catchall  software  discrepancy.

Something  happened  with  the  software.

There  are  some  interesting  causes.

For  example,  other  people being  jerks  on  the  road.

A  reckless  road  user  car didn't  know  what  to  do.

Adverse  weather,  incorrect  behavior prediction,  hardware  discrepancy.

These  are  the  basic  causes.

Now,  I'm  going  to  use a  feature  called  a  Graph  Lit

to   investigate  things  further.

I'm  going  to  right  click  inside  this  bar.

I'm  going  to  hover  label, I'm  going  to  click  Bar.

What  it's  done  is  it's  added a  little  bar  chart  Graph Lit.

When  I  leave  my  bars  there with  a  hover,  here's  my  little  Graph Lit.

Now,  by  default,  it's  going  to  use the  first  categorical  factor  it  sees,

which  was  the  variant  number.

I  don't  want  that.

I  actually  want  the  date.

I'm  going  to  open the  control  panel  here.

I'm  going  to  put  the  date here  and  replace  it.

I'm  going  to  right  click  in  the  date.

Because  it's  a  date.

This  was  recently  introduced.

Think  it  was  either  16  or  17.

I  can  actually  bend by  a  particular  type  of  date,

in  this  case I  want  to  bend  by  a  month?

Perfect.

I  also  would  like to  color  by  the  VIN group.

Just  so  that  we  get  a  clear  comparison of  the  different  types  of  groups.

Notice  there's  a  bit  of  purple  here.

That  means  there  is  some  overlap, which  makes  sense.

There  was  a  bit  of  overlap  from  end of  2020  to  2021  with  a  new  groups,

but  now  we  can  clearly  see,

compare   visually  the  groups  across the  types  of  incidents  just  visually.

I'm  going  to  come  here,  I'm  going  to  go back  down  to  save  script  to  a  clipboard.

I'm  going  to  exit  out  of  this

because  right  now, if  I  hover,  it's  no  longer  there,

but  I  can  right  click go  to  hover  label  and  paste  Graph Lit.

Now  it's  going  to  show  me the  graph  for  each  of  them.

In  fact,  I'm  not  going  to  click and  zoom  in

because  we  can see  what's  happening.

There's  not  too  much  there's this  is  the  unwanted  maneuver.

There  might  have  been  a  bit  more unwanted  maneuvers,

but  that's  more  of  a  general  spike

in  overall  terms, not  much  difference,  I  would  say,

for  the  perception discrepancy  pretty  much  on  par.

There's  a  bit  more  in  general  here, but  I'd  still  say  comparable.

Sort  for  discrepancy.

Whoa,  let's  look  into  that.

Okay,  This  is  very  interesting.

Apparently  with  the  old  group,

there  was  hardly  any  software discrepancies,  maybe  one.

Whereas  with  the  new  group,  there's  a  lot.

Interesting.

There  could  be  some  explanations.

It  could  be  that  the  way  the  category was  defined  maybe  changed  over  time.

Maybe  there  were  other  things.

It  was  re  categorized as  other  things  for  the  old  one

before  they  change the  definition  for  this  new  group.

That's  a  very  valid  explanation.

It  could  also  be  taken  at  face  value.

Maybe   there  were  a  lot  of  other different  causes  with  the  old  one.

Maybe  if  we  look, we're  able  to  look  back  early  on

at  the  individual  vehicle information  for  the  earlier  data,

maybe  we  would  have  seen  a  spike in  the  software  discrepancies.

Because  this  is   later in  that  series  testing,

maybe  we  did  those  out  a  little  bit.

We're  just  seeing  these  early  bugs happening  with  the  new  vehicles.

Let's  actually  now  go  into  the  recurrence event  analysis  now  for  current  events.

That's  just  saying  have  an  individual object  that  can

experience  an  event  repeatedly, in  this  case  a  disengagement  event.

Over  time.

I'm  going  to  run  the  script

and  then  do  the  redo  relaunch  so you  can  see  what  I  did.

Do. I'm  going  to  do  relaunch  by  group.

In  this  case,  here's  what  you  would get  if  you  opened  it  up.

In  this  case,  you  have an  age  or  event  timestamp.

When  did  it  happen? Have  a  timestamp.  The  date.

What's  the  system? That's  the  VIN.

What's  the  cause?

Well,  hey,  great  to  have a  column  called  cause

it's  going  to  basically  break it  down  by  the  different  causes.

You  can  look  at  these  different  causes.

The  timestamp. When  did  I  start  testing  on  this  thing?

Basically,  when  does  it  undergo  start?

When  did  I  stop  looking  at  it?

What's  the  timestamp?

I  put  the  VIN  group  in  by, I  could  have  put  it  in  grouping,

but  there  is  a  bit  of  an  issue  where  if there's  not  a  lot  of  data

in  the  causes  for  both  of  them,

it's  not  going  to  be  able  to  compare  them.

For  right  now,  I'm  just  going to  use  a  by  group, a VIN  by  group.

We  can  still  do  a  sort of  visual  comparison.

For  the  scaling  you  can  select  that

I've  done  today  because  that's the  level  of  resolution  we  have.

It'll  do  it  in  a  day  just for  interpretation,  ease  of  that.

Okay,  so  we're  looking at  this  first  group.

This  is  the  early  group  two  for  our  group.

Notice  it's  broken  it  up by  all  the  different  causes

and  they're  pretty  similar.

The  hardware  discrepancy  seems  like the  biggest  one,

if  I'm  reading  that  correctly. Yes,  that's  hardware.

Notice  this  is  what's called  a  mean  cost  function.

It's  similar  in  interpretation to  that  cumulative  function.

You're  looking  for  the  same  behavior,

a  bit  of  a  rise,  but  you ultimately  want  it  to  flatten  out.

This  is   saying  what's the  average  sort  of  incidence  rate

for  a  single  vehicle  by  this  age?

It's  a  little  bit  harder  to  interpret  than

just  what's  the  mean  total  count because  it's  accumulating  over  time.

Just  at  a  high  level  think,

I  want  to  rise  but  also  want  it to  flatten  out  at  some  point.

So that  I'm  getting  no  more  incidence for  that  vehicle.

That's  what  we're  seeing with  a  lot  of  these.

There's  a  bit  of  a  rise  and  it's starting  to  shallow  out.

If  you  want  detailed  information, you  can  look  under  each  one.

We  can  do  some  fit  models, I'm  not  going  to  at  this  point,

because  we're  starting  near  the  end,

but  it  would  be  similar  types of  models  that  you  would  fit  here.

Let's  look  at  the  SAD  Group,  if  you  will.

There's  clearly  a  difference  here.

Notice  this  big  spike  and  again, that  is  the  software  discrepancy.

What  we're  seeing  is  we've essentially  got  a  new  vehicle.

You  can  you  can  clearly  see  if  we compare  that  is  starting  out  later.

We  had  a  couple,  maybe  some  early on  while  the  other  vehicles  being  tested.

Makes  sense.

You  got  to  sort  of  pilot group  being  tested.

Then  when  they  did  full  scale  testing, the  software  started  having  some  issues.

Now  it's  starting  to  get  better.

We're  seeing  that  level  out.

We  saw  that  in  the  previous  data.

There's  clearly  a  difference, but  they  are  doing  better.

These  two,  the  unwanted  maneuver, the  perception  discrepancy,

I  would  say  visually  they're on  par  with  what  I  see  here.

Clearly  the  software is  the  big  issue  here.

What's  the  big  takeaway  here?

What  would  we  say  about  this?

Well,  it  seems  that  we've  got  two different  series  of  vehicles.

We've  got  one  that  they  were working  on  for  a  while.

Then  in  2021,  they've  introduced a  new  series  of  vehicles.

For  right  now,  especially  in  2021,

there  is  a  bit  of  issues with  the  software  going  on.

A  lot  of  software  discrepancy issues  causing  incidents.

They  seem  to  be improving  that  now.

That   makes  sense.

You've  introduced  something, there's  going  to  be  bugs.

Just  give  it  some  time.

All  the  other  ones  seem  comparable with  what  was  going  before.

Overall,  I've  got  a  pretty  good feeling  about  this  new  series.

Give  it  some  time  to   burn  in.

Well,  I  hope  I've  given  you  a  flavor of  all  the  things  that  JMP  can  do,

especially  in  the  reliability  platform  as well  as  in  preparing  some  of  your  data.

If  you  have  any  questions  about  this,

this  journal  will  be  available where  you  can  find  this  talks.

Some  of  the  data  tables  will be  available  there  as  well.

You  get  to  play  with  all  of  this.

There  are  also  some  other  graphs  didn't really  go  into  some  other  analyzes,

so  feel  free  to  explore.

That's  what  we  do  here  to  JMP statistical  discovery.

If  you  have  any  questions  about  it,

I'll  be  in  some  of  the  Meet  the  Expert sessions  so  you  can  find  me  there.

I'd  love  to  chat  with  you  about  this

or  anything  else  you  might  have  about designing  experiments  or  reliability.

Thank  you  for  your  time.