topic Re: Test whether two predictions (via a multiple regression model) are significantly different in Discussions

Test whether two predictions (via a multiple regression model) are significantly different

broccoli — Wed, 05 Jun 2024 20:12:15 GMT

I'm working on a multiple linear regression model in JMP 16 that attempts to predict clinical outcomes (in the form of a numerical scale called "discharge mRS") for stroke patients based on the following:

1 continuous predictor (NIHSS on admission, which is a measurement of stroke severity on admission to the hospital)
1 categorical predictor (EVT timing, which is a treatment the stroke patient may receive), which can have any of the 3 following values:
- 1 = EVT (the treatment) was administered immediately upon arrival to the hospital
- 2 = EVT was administered after a delay
- 0 = EVT was NOT administered
an interaction term (NIHSS on admission * EVT timing)

This is what the model looks like in JMP right now:

My question / problem is:

For any one given value of NIHSS on admission, I want to calculate whether the predicted mean outcome (based on this model) will be significantly different (with p-values) between if the patient receives EVT immediately (i.e. EVT timing = 1) vs. no EVT (i.e. EVT timing = 0) - that is to say, between 2 different values of the categorical predictor.

Is there some way to do this analysis in JMP?

(Conceptually, my goal is to be able to utilize the 2 predictors in this model in a clinically useful way, e.g. in the hopes that the model can help us decide which patients - in this case, based on the severity of their stroke - should be given a treatment or not, having predicted the outcome with said treatment.)

Thanks for your help!

Re: Test whether two predictions (via a multiple regression model) are significantly different

MRB3855 — Thu, 06 Jun 2024 08:53:03 GMT

Hi @broccoli : Yes, this can be done. Click on the red triangle at the top of the output and choose Estimates/Custom Test (pic shown below). Here is an example.

Example of a Custom Test (jmp.com)

It can be tricky though...take a look, then come back if you need help. You'll need to add two columns. Then put 1 for the EVT timing (and interaction) of interest, and choose whatever value of NIHSS on admission you are interested in for the first two columns. Then the third column is the difference between the first two rows.

Here is an example comparing the prediction for batch 1_11 at time=3.14 (first column 101.2) and batch 3_13 at time=3.14 (second column 101.4). The third column is the difference in predictions (-0.202, with p-value 0.6758)..

Re: Test whether two predictions (via a multiple regression model) are significantly different

broccoli — Thu, 06 Jun 2024 09:48:32 GMT

Hi @MRB3855, I took a shot at setting up custom tests for this purpose but can't seem to get it right.

For example, in the picture below I'm attempting to compare EVT Timing = 0 (in column 1) vs. EVT Timing = 1 (in column 2), with NIHSS on admission = 0.

The numbers listed in the "Value" row in the first two columns of the test output (1.467 and 2.100) aren't equal to the values that my model predicts in those two conditions (0.0563 and 3.568, respectively - see prediction profiler below).

Similarly, when I attempt to repeat this custom test with NIHSS on admission = 20 instead (below), the first two columns of Value outputted (2.329 and 2.962) again do not match my model's predictions (5.105 and 5.089, respectively).

What am I missing here? Obviously it's something crucial, since the test "results" - which, if I'm understanding correctly, are supposed to be listed in the 3rd outputted column (difference in Value predictions, Std Error, t Ratio, p-value, and SS) - seem to be the exact same regardless of what I set NIHSS on admission to (which definitely shouldn't be the case).

Sorry for the trouble and thanks again!

p.s. as a side note, what does "WARNING: Non-Testable Contrast" mean?

Re: Test whether two predictions (via a multiple regression model) are significantly different

MRB3855 — Thu, 06 Jun 2024 14:12:24 GMT

Hi @broccoli You forgot the intercept (put 1 there in the first two columns).

And are you sure it doesn’t matter what NIHSS on admission is?

Edit: Hmmm, I'm not so sure it's working as i expected when there are interaction terms involving the covariate.

Stay tuned.

Re: Test whether two predictions (via a multiple regression model) are significantly different

MRB3855 — Thu, 06 Jun 2024 14:51:51 GMT

Hi @broccoli : Apologies, I made a mistake. See below for predicting at time=24 (for my data). For the interaction term you need to enter (NIHSS on admission - 14.7059). So, if you want to predict at NIHSS on admission=20, then 20-14.7059 = 5.2941 must be entered there for the interaction.

And for the batch that is not listed (I have four batches), it must be parameterized as follows.

Looks weird, I know; that's because of the way JMP parameterizes the model.

https://www.jmp.com/support/help/en/18.0/index.shtml#page/jmp/nominal-factors.shtml#

And, a contrast being non-testable refers to the sums of squares at the bottom of the output that provides a combined test of all contrasts (columns). It is meaningless in this case because the contrasts are not orthogonal (informally, this can be thought of as independent) . However, the tests of the each of the three contrasts individually are valid and correct.

More here about orthogonality. https://stats.stackexchange.com/questions/171324/what-is-the-relationship-between-orthogonal-correlation-and-independence

Re: Test whether two predictions (via a multiple regression model) are significantly different

broccoli — Thu, 06 Jun 2024 19:56:32 GMT

Hi @MRB3855,

Thank you so much! I think it worked. Obviously I didn't (don't know how to) do the math to confirm the tests' outputs myself, but the custom tests' predicted values now match my model's, and the test outputs make sense.

Really appreciate the help!