Share your ideas for the JMP Scripting Unsession at Discovery Summit by September 17th. We hope to see you there!
Choose Language Hide Translation Bar
Highlighted
FR60
Level III

Model Metrics

Hi

in order to predict an electrical parameter by using an equipment parameters  at a particular production step of our semiconductor Fab we used a Random Forest. If we look at the relative error (%) of the model we can see that is is very small being around 0.5% (see attached picture). The RMSE is low too, but R2 is only 52%.

So if we look at the percentage error we can say that the model is fine while this seems to be not true if we look at R2. How to solve?  

 

Thanks   Felice

 

Picture1.jpg

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Model Metrics

Thank you for providing that plot.

 

It shape of the markers looks like a rhombus. I am not sure what to expect but it is unusual.

 

It also demonstrates why the R square is about 0.5 and why predictions will not be very accurate. Imagine predicting the response at an input of 131. I assume that the predicted response is the abscissa. It appears to be unbiased but with high variance. There is a lot of data so the confidence interval on the mean response might be relatively narrow but the prediction interval would be much wider.

Learn it once, use it forever!

View solution in original post

5 REPLIES 5
Highlighted

Re: Model Metrics

The RMSE is an estimate of the standard deviation of the response. It is assumed to be constant over the entire range of the response in the model. The R square estimates how much of the response variance is not the random error component. So your model might be significant, but it will not be very accurate.

Learn it once, use it forever!
Highlighted
FR60
Level III

Re: Model Metrics

I triyed to calculate the systematic error MBE in percentage and this is only 0.01%, that is pratically negligible, and the absolute error is only 0.54%. If we use Accuracy=100-MAE=99.46% that is really a good value. Same for RSME that in percentage is only 0.007% by indicating a very small average deviation from true values. Once again seems that R2 is not giving the complete picture.

What do you think?

 

Felice 

Highlighted

Re: Model Metrics

Can you show a plot of Actual by Predicted?

Learn it once, use it forever!
Highlighted
FR60
Level III

Re: Model Metrics

Hi

in attached the correlation requested.

 

 

Felice

Highlighted

Re: Model Metrics

Thank you for providing that plot.

 

It shape of the markers looks like a rhombus. I am not sure what to expect but it is unusual.

 

It also demonstrates why the R square is about 0.5 and why predictions will not be very accurate. Imagine predicting the response at an input of 131. I assume that the predicted response is the abscissa. It appears to be unbiased but with high variance. There is a lot of data so the confidence interval on the mean response might be relatively narrow but the prediction interval would be much wider.

Learn it once, use it forever!

View solution in original post

Article Labels