Checking Lack of Fit Test Power

[#1]

Dear JMP Community,

Let's say JMP returned me a LOF test results.

If I wanted to know how strong (or power) of that test to detect LOF, are there any indicator to this?

Thanks.

B.r,

Chris

[#2]

Interesting question.

Just to clarify: are you wanting to understand the power to detect LoF when in the design phase? Do you want to know, for a given design that you are considering, the power to detect LoF?

LoF is used to determine if the model that you have fitted is adequate. If there is significant LoF it would suggest an alternative model is required. This is typically a higher order model.

I think you could use standard power analysis (in the JMP Evaluate Design platform or in the design dialogue) to get an indication of LoF fit power. For example, you might design a 2-level experiment for a main effects model, with multiple centre points for LoF detection. (BTW, I am not recommending this kind of design). In the power analysis you could then look at the power to detect a higher order term, such as one of the quadratics. This would tell you about the power of the experiment to detect a higher order term. You might then decide to have more or fewer centre points.

Another solution to consider is the simulation capabiltiy in JMP Pro. This would be a more direct answer to the question about LoF power because you can directly look at the LoF p-value. Again, this would require you to specify a model that represents the higher order behaviours that you want to be able to detect with the LoF test. That is, you need to specify the "alternative" model -- not the model that you have designed your experiment to estimate -- for the simulation.

Either way you need to think about what it is that you want your LoF test to be capable of finding. How different does the "true" model have to be from the model that you have designed for before you want the LoF test to be able to detect it.

I hope that helps.

Phil

[#3]

I don't know of a direct way to compute the power of the lack of fit test in JMP. This test requires certain conditions (e.g., replicate runs) and is based on the analysis of variance from the error sum of squares and the pure error sum of squares. You would have to specify the minimum lack of fit in terms of these sums of squares (I think) and degrees of freedom available to calculate the power of the F statistic. The 1-F Distribution( F, nDF, dDF ) function would be the way to go if choosing this direction.

It might be possible to use DOE > Design Diagnostics > Sample Size and Power > One Sample Standard Deviation with some thought about the parameters.

Phil's suggestion does not directly address the power of the lack of fit test but it might be more practical. You could think about the model inadequacy in terms of specific effects of a given minimal size and use simulation to determine power to detect its omission.

Instead of simulation, have you thought about defining a model in Custom Design that includes all possible terms and deciding the Estimability (Necessary or If Possible) for each term as a way to assess the power of detecting a significant effect?