Why is my lack of fit p-value very small?

[#1]

Hello, JMP community. I have recently finished my experiments following an RSM design. I have inputted the data in the software and ran the model (accounting for interactions and RS). I have excluded the factors and interactions that were irrelevant based on the Effect Summary table. However, the lack of fit p-value remains very small (see screenshot below). I'm afraid the model is overfitting because while this is the case, my RSq value is high at 0.97-0.98 (I also know for a fact that this may not be a good indicator since there are nonlinear interactions among my factors). 

I have also tried the Stepwise modelling, and the lack of fit p-value did not improve. May I ask for help on what to do from here or what I should try to find the model that fits my data? I'm stuck at the moment and can't find the optimal conditions I'm looking for. Thank you.

 

[#2]

Hi @Chemist9,

Welcome in the Community !

The Lack of Fit test is helpful to evaluate if your model fits the data well.
It's very difficult to help you only with a screenshot of this model report. Can you share more screenshots from the different part of the report (actual by predicted plot, residuals, etc...) or even better, the anonymized dataset ? A statistically significant p-value for this test indicates that the model error is largely bigger than the pure error, and that your model may not be adequate.

This situation may happen in your example for several reasons:

• If your model is "too simple" for the results/data you have collected, for example if there are some missing terms to model curvature in your data (through 2nd order terms, interaction, quadratic effects, or higher order effects). Based on which info did you remove some terms from the model (p-values only) ? What are the behaviour and performances of the full assumed model ? Since you have done an RSM model, where the target is optimization and prediction, I would recommend using several metrics to refine your model, such as model RMSE (precision of the model) and an information criterion (AICc/BIC) to help you evaluate and balance the complexity vs. accuracy trade-off of your model.
• If your replicated points have extremely low variance compared to the model. Are your replicates experiments independent from the other experiments ? How did you run these experiments ? Are they "true" replicates : you have created AND measured these experiments independently (not just measured a second/third time the same experiment) ?

I would also recommend to check the adequacy of your regression model through residuals analysis, to check if regression assumptions are met and if any patterns are detected in the residuals that could indicate model's inadequacy. If a pattern is visible, that might confirm that either some terms are missing from the model, or that a transformation may be needed to fit your data correctly.

Hope this answer will help you,

[#3]

Thank you for your helpful response, @Victor_G ! 

To give you a preview of what I have already explored so far:

• I tried running the model where all possible two-way interactions and quadratic terms were included. I then excluded some terms based on their p-values indicated in the Effect summary table. I removed terms with p-value > 0.05 (except for Temperature since it has other important interactions with other factors).
• For the model where I removed the "insignificant" interactions, I got a normalized RMSE of 5.66%, which I believe is still acceptable for metallurgical processes that I am currently exploring. But let me know if we have a different standard on gauging a good or bad RMSE.
• I have not yet explored the information criterion (AICc/BIC) you mentioned. Can I get this from modelling using the Stepwise personality? 
• Yes, my replicates were performed and measured independently. Is it possible that the model is already nitpicking because of the small variance? 

To further aid us in this discussion, here are the relevant plots you mentioned:

• Actual by predicted: I personally do not see a pattern (S, etc.) but it looks like most points are concentrated at the upper right region
• 
• Residual by predicted: I am unsure if there is a pattern on the plot, but I don't think it is randomly distributed.
• Studentized residuals: Based on my current understanding, since all the points lie inside the green and red boundaries, there are no outliers in my data that I need to remove.
• 

My problem now is I do not know where to go from here since I have very limited knowledge on these tools.

As a new member of the community and JMP user, your response was really helpful. Looking forward to our discussion. Thank you!

[#4]

Ok, this is helpful.
Looking at the actual vs. predicted plot, it seems that some low values may be overestimated, and some high values may be underestimated. I'm also very curious about some "line of points" in these plots: are they replicates, or different points where a term is missing in the model to differentiate them ?

The information criterion is an additional metric available in Fit Least Squares report in the Summary of Fit panel.

Yes, in cases where variance from replicated points is very small, you may get statistically significant lack-of-fit test, with an appropriate model (hence my question about "true" replicates, to make sure you have done them correctly and not underestimate natural experimental variation). You may also have statistically significant lack of fit test more easily with high number of centre points : https://statease.com/blog/importance-center-points-central-composite-designs/#:~:text=The%206%20cent....

Please check the Box-cox transformation panel accessible in the Factor Profiling options, to make sure you don't need a small transformation to better fit your data. Percentage values responses usually need a LogitPct or LogisticPct transformation to make sure predicted values stay in the range from 0 to 100. These transformations should help reduce S patterns in the residuals and actual by predicted plots.

Hope this follow-up will help you,

[#5]

Hi @Victor_G !

I tried doing your recommendations, and I think there has been some improvements in the model. However, I am still unsure if this is the best-fit model for my data.

Those "line of points" you are referring to are true replicates. 

I did the LogitPct and LogisticPct, and it seems that LogitPct works better for my set of data. I had a more realistic prediction profiler and a lower value for AICc compared to the LogisticPct. Doing this also limited the maximum Y response to 100%, which I was encountering previously. However, my lack of fit p-value is still very small, although the RMSE improved. Is it possible that I am still getting low LOF p-value because of my center points? I designed my experiments to follow Central composite design-RSM.

I am also thinking of applying a higher-order model. Is there a way that JMP can suggest the possible higher-order interactions? Or is it done in the Fit model window manually?

Thank you very much for your helpful inputs!

[#6]

Hi @Chemist9,

Ok, the model looks indeed much better from the screenshots you send.

How many centre points do you have in your RSM ? If you look at my previous link from StatEase, common number of centre points is usually around 3 to 5. If you have a higher number of centre points, it's possible that you have increased drastically the sensitivity of the lack-of-fit test, and that the small p-value found is a "false positive". You can try to "hide and exclude" some centre points to see if the test diagnostic is still the same or not.

You can also try to add other terms in your model manually in the Fit Model window. Which type of Central Composite RSM design have you chosen ? Face-centered ? If this is the case, that means you have three different levels per factor, so the order of your model will be limited to 2. You can still try to add partial cubic terms in the model, which are interactions between main effects and quadratic effects. These terms are usually rare, but if one of them is active, it could significantly improve your model.

Finally, think about how you validate your model:

• What is the purpose of this model ? Optimization, prediction, knowledge building, etc... ? What are your criteria for success ?
• Does the model behaviour match your domain knowledge and expectations ?
• Are the precision of the predictions practically acceptable ?
• Are you planning to validate your model over the entire experimental space (by testing new experiments in untested locations of the experimental space) ? Or are you interested only in the optimum found by the model ?

These non exhaustive questions can help you define when to stop improving your model and start using it.
Hope this answer will help you,

[#7]

I agree with Victor that maybe your quadratic model is not able to explain the full response variation; your model might be more compex so checking the effect of higher order terms is interesting. I assume that your RSM is generated with a CCD? Augmenting your DOE to get a 3rd order model could be interesting to check effect on lack of fit p-value.

[#8]

Hi @Chemist9 : Are the "line of points" nested within something that is not included in the model? i.e., is each "line of points" from a "run" or some other such blocking factor or experimental unit?