topic Re: Fit Model Custom Test Coefficients in Discussions

Fit Model Custom Test Coefficients

Stephen2020 — Sun, 26 Apr 2020 15:45:58 GMT

I am trying to jury-rig a post-hoc pairwise slope comparison with a custom test. I read through the documentation on custom tests but still don't understand how I am supposed to determine what coefficients to use to set up the tests. I have three treatments [PT CS OF] and want to do a pairwise test of the interaction between these treatments and a continuous variable [Duration]. If I understand the Custom test correctly each test will fill it's own column.

I want to test:

PT*Duration-CS*Duration=0;

PT*Duration-OF*Duration=0;

CS*Duration-OF*Duration=0;

Below I've attached a screen grab of the test coefficients and the results. I get a warning which I assume means I'm doing something wrong.

Thanks in advance for the help.

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Thu, 16 Apr 2020 11:31:40 GMT

These custom tests are also known as contrasts elsewhere.

The miss-specification is because you entered +1 for both levels in the first test. Make sure that the coefficients sum to zero. Then make sure that the value after "=" is the null hypothesis. Entering zero is usually appropriate if you are testing for a significant difference but might be otherwise in some situations.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Sun, 26 Apr 2020 15:46:39 GMT

Thanks for the clarification. I've attached another window below. If I understand correctly I have set up the first contrast (CS*Duration-OF*Duration=0) correctly. I thought the negative coefficients were supposed to let me test against the third categorical effect (PT*duration), you can see that in the expanded estimates but not in the set up for the custom test. How do I test against PT*Duration?

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Thu, 16 Apr 2020 13:29:58 GMT

I believe that you set the coefficient for the first estimate equal to 1 and leave the rest 0. You can check the result. The Value should be the difference between the estimates that you are comparing to null of zero difference.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Sun, 26 Apr 2020 15:47:12 GMT

Still don't think I'm getting it right.

First, I'm still getting a non-contrastable warning (see below).

Second, It appears that by making the second coefficient -1 for the CS*Duration -(-OF*Duration)=0 cross adds the value of the two which is not what I want. It ends up being -0.0447.

Third, if I am trying to compare PT*Duration to OF*Duration (OF*Duration-PT*Duration=0) the Value should be 0.00777 instead it is just the value of OF*Duration (0.0174926). So leaving the coefficient at 0 does not seem to work.

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Thu, 16 Apr 2020 19:59:04 GMT

I apologize for the confusion. I am not sure about all the confusion, so let's start at the beginning.

You started by saying that you want a post hoc pairwise slope comparison with a custom test. So the trend line for each group appears to be different and you want to test the difference. Let's try.

(NOTE: I received fantastic help from a colleague in technical support. My reply is based on his explanation.)

You can perform the custom tests in one step, but I am going to break it into two steps so you can see the development of the correct coefficients.

Step 1: Use custom tests of the slope of the line for each treatment. The first column in the custom test should be [ 0, 0, 0, 1, 1, 0, 0 ]. That is, the only parameters with a 1 are Pulse Duration and Treatment[PT]*Pulse Duration. The reason is because the slope of the line is the common slope, Pulse Duration, plus the differential slope for Treatment[PT]. Their sum is the slope you want to test against 0. The second column should be [ 0, 0, 0, 1, 0, 1, 0 ] because you are adding the Pulse Duration and the Treatment[CS]*Pulse Duration. Finally, add the differential slope for the third treatment but it is not explicitly reported. But we know that it is the negative of the sum of the other two parameters because of the way that the JMP parameterizes a categorical factor. So the third column should be [ 0, 0, 0, 1, -1, -1, 0 ].

Again, you do not need this step for the actual custom test that you want. It only helps my explanation of the actual tests. At this point we have determined the correct coefficients for the custom tests of the individual slopes.

Step 2: You want to compare the individual slopes, not to zero, but to each other, two at a time. We simply subtract the columns used in step 1 two at a time! To compare the slope of PT to CS, we subtract the column of coefficients in step one for CS from those for PT. So [ 0, 0, 0, 1, 1, 0, 0 ] - [ 0, 0, 0, 1, 0, 1, 0 ] = [ 0, 0, 0, 0, 1, -1, 0 ]. The column to test PT to OF is [ 0, 0, 0, 1, 1, 0, 0 ] - [ 0, 0, 0, 1, -1, -1, 0 ] = [ 0, 0, 0, 0, 2, 1, 0 ]. Finally, the column to test CS to OF is [ 0, 0, 0, 1, 0, 1, 0 ] - [ 0, 0, 0, 1, -1, -1, 0 ] = [ 0, 0, 0, 0, 1, 2, 0 ].

As before, you can always check the coefficients by comparing to the Value that JMP reports for the test with the hand calculations of the sums and differences of the parameter estimates.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Thu, 16 Apr 2020 19:53:39 GMT

Thanks, this is the solution I'm looking for and the results makes sense. The only final hiccup is I am still getting the "Warning: Non-Testable Contrast". Is that saying the joint test is Non-Testable or a one of the contrasts is not testable? When I run the pairwise contrasts independently I don't get the same response.

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Fri, 17 Apr 2020 14:23:46 GMT

The warning is about the joint test. There are not enough degrees of freedom available for the number of joint tests.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Fri, 17 Apr 2020 14:28:33 GMT

Thanks, I don't think I need the joint test. Just the independent contrasts.

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Fri, 17 Apr 2020 14:33:49 GMT

The joint test is used by some analysts to protect against a high false positive rate. You might perform a lot of tests. They are not adjusted individually for the number of tests.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Fri, 17 Apr 2020 14:41:28 GMT

If I am Bonferroni correcting the tests will that be enough to reduce false positives?

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Fri, 17 Apr 2020 14:53:25 GMT

I interpret what you ask to mean if you use the Bonferroni adjustment to your choice of alpha. Yes, that is one way to control the type I error rate with multiple comparisons.

Re: Fit Model Custom Test Coefficients

Stephen2020 — Fri, 17 Apr 2020 14:54:56 GMT

That's exactly what I'm planning to do. Thanks

Re: Fit Model Custom Test Coefficients

Stephen2020 — Mon, 20 Apr 2020 14:45:00 GMT

One follow-up question:

I'm fitting the same basic model to a bunch of different factors. And I end up testing the same interactions in different circumstances.

Do I Bonferroni correct within the joint test within the model (so three levels), should Bonferroni correct all the tests within a single model, or should I pool all of these custom tests across all models and do one big Bonferroni correction?

Thanks

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Mon, 20 Apr 2020 15:21:03 GMT

I understand that the issue of multiple comparisons is within a study and a single data set. If I repeat a study, then I do not have to count the comparisons made in the first study in my adjustment. It sounds, though, as if you have multiple factors in the same study, so you would adjust based on the cumulative number of tests.

Is there a reason that you would not include all the factors at the same time in a single model instead of separate analyses for each factor?

Re: Fit Model Custom Test Coefficients

Stephen2020 — Mon, 20 Apr 2020 15:41:12 GMT

Just wanting to be clear, I am including all of the independent factors and significant interaction terms in the single model. I have run a MANOVA that collects all the dependent factors in the same model, and the whole model is significant. Beyond that, I was told to parse out the behavior of the dependent factors in the MANOVA by breaking it back down into individual standard least squares analyses. If you have a link to a video or slide deck explaining how to interpret an work with MANOVA's I would appreciate that.

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Mon, 20 Apr 2020 16:07:00 GMT

I think that you are on the right course!

Did you look at the Effect Tests? There is more than one way to analyze and model data, but I like to start with the Whole Model, proceed with the Effect Tests and other information to reduce the model, and then use Custom Tests to with interesting comparisons within a significant effect.

There is a lot of helpful material on the JMP site. I do not pretend to know about everything that is available. It is searchable, though. For example, I found this page with tutorials and videos for academic users (but anybody can use it).

Re: Fit Model Custom Test Coefficients

Stephen2020 — Sun, 26 Apr 2020 15:45:03 GMT

Okay, Custom test and MANOVA questions.

When I would do the three column pairwise test for the interaction slope in the ANOVA it would give me all the test statistics and the warning about unable to make the contrast, but at least I would get all the estimates and test statistics. I tried the same thing in the MANOVA. I fit the model as I would with the factors and their interactions against all four response variables. (I removed some of the higher order non-significant interactions) I selected "test each column also" and "identity" as my response option. For a given column, say "total heading change" I tried a custom test the same way I would have in an ANOVA but the result was a single F-statistic and the same warning about being unable to test the contrast. When I do single pairwise tests I get results I would expect. Do I just do a billion pairwise tests in each of the Column tests I'm interested in and Bonferroni correct?

Re: Fit Model Custom Test Coefficients

Mark_Bailey — Sun, 26 Apr 2020 11:50:09 GMT

I suspect that it is an issue with degrees of freedom, but I am not familiar with the tests in the MANOVA procedure. Perhaps another member is able to advise you. I recommend that you consult the JMP or SAS software documentation for help. You can also contact JMP Technical Support (support@jmp.com) with your question.