Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

- JMP User Community
- :
- Discussions
- :
- Model Metrics

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Created:
Mar 12, 2020 3:38 AM
| Last Modified: Mar 12, 2020 3:59 AM
(1384 views)

Hi

in order to predict an electrical parameter by using an equipment parameters at a particular production step of our semiconductor Fab we used a Random Forest. If we look at the relative error (%) of the model we can see that is is very small being around 0.5% (see attached picture). The RMSE is low too, but R2 is only 52%.

So if we look at the percentage error we can say that the model is fine while this seems to be not true if we look at R2. How to solve?

Thanks Felice

1 ACCEPTED SOLUTION

Accepted Solutions

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Thank you for providing that plot.

It shape of the markers looks like a rhombus. I am not sure what to expect but it is unusual.

It also demonstrates why the R square is about 0.5 and why predictions will not be very accurate. Imagine predicting the response at an input of 131. I assume that the predicted response is the abscissa. It appears to be unbiased but with high variance. There is a lot of data so the confidence interval on the mean response might be relatively narrow but the prediction interval would be much wider.

Learn it once, use it forever!

5 REPLIES 5

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Model Metrics

The RMSE is an estimate of the standard deviation of the response. It is assumed to be constant over the entire range of the response in the model. The R square estimates how much of the response variance is not the random error component. So your model might be significant, but it will not be very accurate.

Learn it once, use it forever!

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Model Metrics

I triyed to calculate the systematic error MBE in percentage and this is only 0.01%, that is pratically negligible, and the absolute error is only 0.54%. If we use Accuracy=100-MAE=99.46% that is really a good value. Same for RSME that in percentage is only 0.007% by indicating a very small average deviation from true values. Once again seems that R2 is not giving the complete picture.

What do you think?

Felice

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Model Metrics

Can you show a plot of **Actual by Predicted**?

Learn it once, use it forever!

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: Model Metrics

Hi

in attached the correlation requested.

Felice

Highlighted

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Thank you for providing that plot.

It shape of the markers looks like a rhombus. I am not sure what to expect but it is unusual.

It also demonstrates why the R square is about 0.5 and why predictions will not be very accurate. Imagine predicting the response at an input of 131. I assume that the predicted response is the abscissa. It appears to be unbiased but with high variance. There is a lot of data so the confidence interval on the mean response might be relatively narrow but the prediction interval would be much wider.

Learn it once, use it forever!