Random effect mixed model

[#1]

Hello,

I am building a DoE model. However, I see that the R2 for my model fit is not good. This could be due to the analytical sequence of sample measurement affecting the measured data. I added 'Analytical sequence- Series' as a random effect factor in the model. I see that the % from this factor is quite high in the model. In other models, they are lower. At what value should we start considering the random effect as a main factor in the model?

JMP prompts that the Press R2 will not be calculated for the mixed model. I could not find information on why this is the case?

Adding @martindemel as I had discussed one of the models with him. Now, I received more questions on this topic.

Thank you.

Kind regards,

Chethana

[#2]

Hi @Coverbird30 : The answer to your first question "At what value should we start considering the random effect as a main factor in the model?" is...it depends.  Can you provide more detail about exactly what your data are, how it was collected, goals, etc. 

Your second question is about Predicted R^2 and it not being available for mixed models (if your "Method" is REML); the leave-one-out residuals (and therefore Predicted R^2) are calculated from the diagonal elements of the Hat matrix via least squares.  If least sqaures (and EMS for variance components) in not used in favor of REML then there is no Hat matrix....and therefore no Press etc.  

More about Hat matrix below.

https://stats.stackexchange.com/questions/410443/how-to-use-press-statistic-for-model-selection

https://www.statology.org/press-statistic/

https://en.wikipedia.org/wiki/Projection_matrix#Ordinary_least_squares

https://math.stackexchange.com/questions/3895869/standardized-residuals

[#3]

Hi @Coverbird30 ,

To clarify, are you asking at what "Pct of total" do you want to include your factor as a fixed effect in your model? Or as you asking at what "Pct of total" should the random effect be considered to be a significant random effect?

In the first case, the decision to model an effect as random or fixed is really context dependent. In the second case, there is really no formal value that answers this question.

[#4]

Hi @Coverbird30,

What do you mean by " In other models, they are lower. " when talking about the random effects? Could you provide the dataset to take a further look? The Pct total will always add up to 100%, so that isn't always a good marker. The Wald p-value is a good indicator of if the random effect is significant (which it doesn't look to be), although you do have quite a difference between the associated percent between the Series random effect and the residual.

One thing you can do is compare a fixed effect and random effect model by comparing the BIC/AIC scores to see what is more of an appropriate fit. 

I would also think about if you can attribute the 'Series' further to the part, analyst or measurement system, as this will allow you to do more tests with the measurement system analysis and attribute the true random effects that are present in your study.

Thanks,

Ben

[#5]

Sorry, I'm a bit confused by some of your terminology and your situation.  Are you doing explanatory investigation or predictive modeling?What do you mean "I am building a DOE model"?  Do you mean you are building a model using data from an experiment?  Because, if you are running an experiment, you should have a hypothesized model already in mind.  This is how you determine what experiments need to be run. R^2 is only one of the many statistics used to help refine the model.  The more important statistic is the delta between the R^2 and R^2 Adjusted.  This will provide insight to over-specifying the model (e.g., including insignificant terms in the model.). 

Do you have measurement system issues or does your experimental unit change over time.  It goes back to my first question.  If you are developing an explanatory model, then having the random effect in the model may be useful.  It is much less useful for predictive models.