topic Re: How to track model performance ? in Discussions

How to track model performance ?

LogitTurtle576 — Thu, 08 Jun 2023 16:27:45 GMT

I developed a viscosity model on one product based on hundreds of lab data using few process parameters.

Now that the model is built, what is the best approach to track its performance over time ? Control chart ?

We continue to get lab analysis on a frequent basis so we can look at the analysis vs predicted value.

Thanks in advance for your advice.

Re: How to track model performance ?

statman — Fri, 17 Mar 2023 14:48:00 GMT

Interesting...Here are my initial thoughts:

First, how to you build the model (e.g., GLM, stepwise, PLS, neural)? Second, what is the purpose of your tracking the performance over time? Are you planning on using this to react to deviations from the prediction? How much deviation would cause you to react? What would your reaction be? Wouldn't it be better to chart the X's that are in your model (more predictive)?

If the model building was based on observational data, have you tried experimentation to confirm the model is causal vs. correlation?

Do you want to improve the model (e.g., identify additional variables or higher order terms to include in the model ) or just assess consistency of the model? Do you want to expand the inference space to include other products? I suppose you could plot the residuals over time (e.g., X, MR charts). What and how you plot is a function of what you want to accomplish with the "tracking".

Out of curiosity, have you assessed the measurement system? This is a great opportunity to use control chart method (and may provide better determination of calibration frequency).

Re: How to track model performance ?

Mark_Bailey — Fri, 17 Mar 2023 15:27:42 GMT

The obvious comparison is a scatter plot of actual by predicted. You could add the Y=X line for reference. @statman's suggestion goes to the stability of the prediction. You could also use process capability (prediction is a process) to help assess the suitability.