topic Re: Huge DF denominators for repeated measures ANOVAs with JMP 14 in Discussions

Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Wed, 23 Jan 2019 06:57:01 GMT

I used to run repeated measures ANOVAs with JMP 13 with no proglems. But now I got JMP 14 and ran an ANOVA with the same data and somehow got "4326" as a DF Den under "Test Denominator Synthesis". This used to be "42" with JMP 13, and ovbiously the new F ratio got so much larger. I do not think there is anything different I did (data, how to set up), other than I am now using JMP 14 instead of JMP 13, and I really do not know what to do.

Could anyone please let me know what I am missing?

Thank you!

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Wed, 23 Jan 2019 11:44:03 GMT

We do not know much about the details of your data or analysis but the increase in the DF for the denominator suggests that you have fewer DF for the model. Are there either fewer terms or fewer levels of your predictors?

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Thu, 24 Jan 2019 04:31:12 GMT

Thank you very much for your response.

I attach a file with some images. Hopefully they will give you more information.

The image 1 shows how I specified the model with JMP 14, and the image 2 shows the output I get now. I am 100% sure that I used the same data set and 99% sure (I can no longer run JMP 13 to verify) that I used the same model specifiction, but I used to get an output like the image 3.

I do not know why DF Den are so different between the two versions, and would like to know how I could revert back to the output I used to get with JMP 13. I would really appreciate your help. Thank you!

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Thu, 24 Jan 2019 12:57:17 GMT

The fact that your results with JMP 13 exhibit fractional degrees of freedom for the denominator of the F-test suggests that you used REML to estimate the parameters and variances. REML is the default (and recommended) procedure so that makes sense. On the other hand, EMS is selected for the analysis done with JMP 14. Try using REML with JMP 14.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Thu, 24 Jan 2019 14:23:03 GMT

Thank you very much for your response.

I tried using REML, but the results are the same as the attached image.

I am wondering if you could try running my data. It would really help me if you could. I will attach my data set with the following message. Thank you!

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Thu, 24 Jan 2019 14:24:47 GMT

Here is the data file attached.

>Thank you very much for your response.

>I tried using REML, but the results are the same as the attached image.

>I am wondering if you could try running my data. It would really help me if you could. I will attach >my data set with the following message. Thank you!

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Thu, 24 Jan 2019 14:51:22 GMT

I verified your results with JMP 14.

It appears to me that almost half of the terms, all higher-order interactions, may be removed from the model.

I am not surprised that there are 4326 degrees of freedom for the denominator (MSE) of the F-tests. You start with 4576 observations and your model contains 208 parameters. I do not understand how the DF could have been only 42 in the original analysis.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Thu, 24 Jan 2019 14:54:33 GMT

It would have been helpful if you had suppled the JMP data table. I had to restore the meta-data and the model in Fit Least Squares by hand.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Thu, 24 Jan 2019 14:55:14 GMT

I don't see the need to nest Subject within Disease. I verified that the results without nesting are identical.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Thu, 24 Jan 2019 16:18:48 GMT

Thank you again for the quick reply!

I am sorry about the Excel vs. JMP formats, and I can see that for REML (although it makes a difference for EMS), there is no need for nesting Subject within Disease.

But my original questions still stay:

1. Why are the outputs so different between JMP 13 and JMP 14? If you look at the Test Denominator Synthesis sections for both, the JMP 13 secion lists different DF Den's for different factors, while the JMP 14 section list just one number (4326) for all the factors except Disease (42). Also, the Denom MS Synthesis column lists different factor combinations for different factors for JMP 13 but just "Residual" for all the factors except Disease for JMP 14.

2. How could I revert back to the output I used to get with JMP 13? I would like to use the JMP 13 version, as it matches the results from SPSS and Unix-ranova.

Thank you again.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Thu, 24 Jan 2019 17:06:37 GMT

They are not different. Not at all.

I copied the data from the Excel workbook to a JMP data table. I set up the same full factorial linear model with Subject as the random effect in Fit Model dialog. I used REML to fit the data with JMP 14. I saved the script for the Fit Least Squares analysis and then saved the data table. I closed the data table and JMP 14. I launched JMP 13, opened the data table and ran the script. The results are identical.

I then submitted the Fit Model job to SAS, which uses PROC MIXED. The results are identical. (Except for the different parametrization in the SAS procedure.)

So the DF for the denominators in question are the same (4326) by all three analyses (JMP 13, JMP 14, and SAS 9.4).

Also, as I mentioned before, these DF seem reasonable given the number of observations and the number of parameter estimates.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Fri, 25 Jan 2019 07:09:26 GMT

Thank you again for your quick response.

I now have to appologize. I am sorry I made a mistake in remembering the model I used to use. I found that my old laptop still had JMP 13 (which will be good until this March), started playing with it, and finally remembered that I used to put in all possible combinations of the factors including Subject as the attached image shows. In this way, I can get the output I showed you as Image 3 of my RepeatedMeasures.docx file. If I try to use REML instaed of EMS, JMP 13 complains that there is not enough memory. I also tried to run the same model with JMP 14 on my new laptop, but then I get the "Unable to allocate enough memory" error message for both EMS and REML.

I like this output as it gives the same results as Unix-ranova and SPSS (General Linea Model>Repeated Measures>Full factorial, SS Type III). I could try to do all the analyses with JMP 13 on my old laptop for these couple of months as a quick fix, but I do still have altered versions of my original questions as below:

1') Why putting in all possible factors change DF Den's so much (different DF Den's for different factors vs. just one number for all the factors except Disease, and for the Denom MS Synthesis column, different factor combinations for different factors vs. just "Residual" for all the factors except Disease)?

2') Ideally I would like to continue running my ANOVAs with JMP 14 on my new laptop, but would there be any way to allocate more memory?

Thank you.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Fri, 25 Jan 2019 14:50:47 GMT

All that I know about you (i.e., your background) and your purpose is based on this discussion, this data set, and the modeling attempts, so please forgive me if my response covers anything that you already know.

This question suggests that you do not have a strong background in linear models.
1. The DF for the denominator are a function of the number of observations and the number of parameters in the model for a given experimental unit. The parameterization in the JMP Fit Least Squares platform is such that the number of parameters for a categorical factor is one less than the number of levels. So including Channel costs 25 DF alone. Crossing it with other factors quickly adds up to a lot of parameters and, therefore, a large change in the DF.
2. The choice of the estimation method (EMS or REML) can also affect the DF for the denominator. That difference is the reason that REML is recommended.
3. Note, too, that the estimation depends on the design of the data. How did you design this study?
The amount of memory used by JMP cannot be altered from within JMP other than to use the 64-bit version instead of the 32-bit version. I am not an expert in Windows so I don't know how to allocate more memory to an application from the operating system.

You say that you like this output because it agrees with SPSS and Unix-ranova. I suggest that a model should be based on its own merits and not the software used to estimate the parameters. It seems to me that your model is over-specified.

The structure of your study indicates no nesting whatsoever. All of the factors are crossed. Why add this complication to the model specification?
You specified a full factorial model (i.e., all possible interaction effects). Are those terms reasonable?
You crossed Subject with many of these interactions. Are those terms reasonable?
You select EMS for estimating the model and variance components. Why use this inferior method?

How did you select the model that you like? What is the basis for selecting this model? What is the model selection process and criterion?

For example, I started with a full factorial model for the fixed factors and add Subject as a random effect. I was able to reduce the model using reasonable model selection criteria (e.g., parsimony, AICc),

Residual analysis indicates no pattern or heteroscedasticity or lack of fit. The residuals indicate some right skew in the errors. This model seems reasonable to me. Also, I would expect to get the same results with this model using JMP, SAS, SPSS, or Unix-ranova with REML procedure.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Mon, 28 Jan 2019 04:31:00 GMT

Thank you very much for your very detailed response and trying to run a more appropriate model.

I do agree that I have been trying to blindly follow what has been prevously done in the field and use a full factorial model, and that there is no theoretical reason for the experiment to cross Subject and all the interactions.

But actually I was asking something else in my quetion 1', so please let me ask this again with some more explanations:

Please compare Image 1 and Images 2ab of the attached file. Image 1 shows that when all possible factors are crossed, the DF Den column lists different numbers for different factors (from the top: 42, 42, 42, 1050, 42, 42 ...), which are also what I get with Unix-ranova and SPSS (General Linea Model>Repeated Measures>Full factorial, SS Type III). However, when Subject is not crossed with other factors as in Images 2ab, the DF Den column lists 42 for Disease but 4326 for all the other factors. Also, the Denom MS Synthesis column of Image 1 lists different factor combinations for different factors, while the Denom MS Synthesis columns of Images 2ab just list "Residual" for all the factors except Disease. Why would this be the case?

Thank you again.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

Mark_Bailey — Mon, 28 Jan 2019 14:40:23 GMT

Please see from within JMP Help > Books > Fitting Linear Models then the chapter at the end of the book with Statistical Details. The origin of the degrees of freedom is explained in that chapter.

Re: Huge DF denominators for repeated measures ANOVAs with JMP 14

mu — Fri, 01 Feb 2019 16:45:18 GMT

Thank you for your response and the reference to the JMP book.

I have looked at the book and found the following on pp. 197-198 of Chapter 3:

“Test Denominator Synthesis

For each effect to be tested, an F statistic is constructed. The denominator for this statistic is the mean square whose expectation is that of the numerator mean square under the null hypothesis. This denominator is constructed, or synthesized, from variance components and values associated with fixed effects.

Source Shows the effect to be tested.

MS Den Gives the estimated mean square for the denominator of the F test.

DF Den Gives the degrees of freedom for the synthesized denominator. These are constructed using Satterthwaite’s method (Satterthwaite 1946).

Denom MS Synthesis Gives the variance components used in the denominator synthesis. The residual error variance is always part of this synthesis.”

I also looked at Satterthwaite (1946), but it was too technical for me to follow.

In the meantime, I have also tried more factorial combinations. If you could look at Images 1-3 of the attached file, it appears that when a certain factor is crossed with Subject, the corresponding DF Den stays independent while the DF Dens for the other facters are put together as “Residual”. Could you kindly explain why it becomes like this in my example?

Thank you again.