Prediction using JMP recurrence analysis

[#1]

Hi all,

I tried to predict future repairs and cost for our machines using JMP recurrence analysis, but I’m encountering overprediction issue. I trained a model (log linear NHPP) using lifetime data of the bots up to 90 days ago and then predicted currently added repairs and costs. Basically I use data 90 days ago for modeling and prediction, and valid the results using current data. The results show about 30% overprediction. I also tried the proportional NHPP model and got a similar amount of overprediction. The fitting of the models looks good to me. In terms of prediction on total repairs, the model works well (less than %10 overprediction). Only the prediction for newly occurred repair events are not well.

My question is:
Is there a way to reduce this overprediction? I’m considering predicting events over a 45-day period instead of 90 days to have a short window. but I’m wondering if there are other approaches I can try.

I know there are other models for prediction. Since I have been using recurrence to study repairs and get familiar with it already, I want to make the most of this study.

Thanks.

[#2]

Would it be possible to post an anonymized (Tables > Anonymize) or subset version of your data with your analysis attached? That might help us suggest alternatives.

[#3]

Hi, that is good suggestion.

I’ll explain what I did using the following screenshot, which shows data for one bot. The real dataset contains about 15,000 bots. Please don’t use my example values to fit the model — the predicted values are based on the cumulative formula generated from all 15,000 bots.

The first three columns in table on the left includes data 90 days ago,  BOT ID, Age, and Cost (1 means a repair, 0 means end of study), and these three columns were used to fit the JMP Recurrence Analysis proportional Poisson Process model (There are additional 15k bots that I did not show. Just want to use one bot to show what I did). The cumulative formula from the model was then obtained. The fourth column, Age Plus 90 Days (current age), was fed into this formula to calculate the fifth column, Predicted Repairs in 90 Days (prediction for current time).

The value 6.28 represents the predicted total repairs for this bot at age 941. Therefore, the repairs for this bot in the most recent 90 days are predicted to be 6.28 − 3 = 3.28. To calculate total numbers, I sum all predicted repairs at the oldest age for each bot to get total prediction. I sum all true repairs  (90 days ago) for each bot to get total true repairs. Then I get the difference (total predicted repairs for most recent 90 days) between those two sums.

The right table (current data) shows that only 3 repairs actually occurred up to 941 days, which is the same as 90 days ago. so the true repairs for this bot in the most recent 90 days are actually 3-3=0. I also calculate the true repairs for all bots in the most recent 90 days.

When I compare the two numbers, I consistently see an overprediction. For example, for all 15,000 bots, the number of true repairs in the most recent 90 days is 6,996, while the predicted repairs for the same bots are 9,306 — a 33.0% overprediction. I’ve tried several time windows and always get around a 30% overprediction.

I’m concerned that I might have done something wrong.
What do you think? Thanks

I also attached a simulated data in case who want to play around with it. 

There are three columns in the data, Machine ID, cost (1 means repair and 0 means end of study), age. The question is how to predict repairs during the following 90 days using the data. Thanks.

[#4]

Depending on the shape of your actual data and on the model parameters you select, you can get very different models. Your simulated data has an anomaly at the end that can also alter predictions. From the data provided, I get very different models depending on if I include Age^2 or not. How well is your model fitting your data?

[#5]

The anomaly at the end is caused by only two machines at risk and those two machines are the oldest ones.

I just use the proportional PP model without scale or shape effect and the figure looks like following

There are two things that I don't understand:
why do you pick age as Scale Effect and Shape Effects? It does not generate a curve when you do so. Why is it like that? Where is the fit curve?
Why do you try age^2?

Thanks.

[#6]

I'm not sure how much of an issue this is, but for Recurrence models, you're system data should be in chronological order, as I don't think JMP will automatically reorder them, but this will make a difference in your model. Here are two machine ID's in your sample data that were not in order. If this isn't something you see in your data, then you can ignore this.

 

The Recurrence Analysis is for when your data is a repairable system. A repairable system doesn't mean that it can be repaired, but what is the status after it has been repaired. After the event, are the bots considered to be exactly the same as just before the event occurred? Or are the bots considered "as good as new"? If the bot is "as good as new", this is a non-repairable system because you essentially have a brand new item.

The reason this is important is because if your data as a repairable system, it is modeled as an exponential system and the events should occur more frequently over time. If your data is not a repairable system and you try to use the Recurrence Analysis, it will often over predict your forecasts because it assumes things are going to keep getting worse. A non-repairable system should reach a consistent amount of repairs over time, which will often be much less than the exponential system.

If the bots are "as good as new", it might make more sense to model each event independently (i.e. age is not cumulative) and use the Life Distribution platform.

[#7]

Hi, 

Thanks for you explanation.

I tried examples with and without  chronological order. It seems that JMP recurrence analysis automate this with right order.

Our system is repairable and it is only assumed to be as good as when it was before the repair. So the bots wear out with aging. That is the reason we choose the recurrence analysis. I think we found the reason for the over prediction. We did not subtract time when the bots were under repair, leading to a immortal bias (fancy word I just learnt). I am excluding the time under repair and see if I will see the over prediction again.

Thanks for your suggestions.