Can't replicate repeated measures anova results for tukey test... feel like I'm doing something wrong.

[#1]

Hello everyone! This is my first post here. I'm a master's student doing research about remediating water pollution using wetland plants. I've been going through my data with JMP Student Edition to do tukey tests and find out which results are statistically significant with a statistician who has been tutoring me. But now he is on vacation and I've been trying to replicate the work we did together by myself, but I'm not getting the same results! So I've run into a wall. 

This is what I thought I remember us doing: 

1. importing the excel sheet pictured below to JMP. 

2. click Analyze --> Fit Model

3. in the dialogue box that pops open, put all the "Mean(pH, T1), Mean(pH, T2)," etc columns into the Y part under "Pick Role Variables."

4. put Species, Treatment, and Block into "Construct Model Effects" and doing "factorial to degree" to put them into relations with each other.

5. Running it as a manova, and choosing "repeated measures" and "time" to the boxes that show up. 

6. if the results are statistically significant, do a regular anova ("Standard Least Squares" personality) with each "Mean(pH, T1)" column individually with "Effect Leverage" emphasis. 

7. In the window that shows the results of that anova to the box with the little red arrow that says "Species" and do a tukey test, and see what different letters each plant species has. 

This is the part where I feel like something is wrong. We did this pH spreadsheet together, and I put the results into a word document pictured here with highlighted purple and yellow. The ones I did with the statistician guiding me are on the left, the ones I did by myself are on the right. As you can see, when I repeated it on my own to make sure I got the process right, the tukey test is slightly different... when I look at the "Least square mean," only the one on the Control is a different number. So I think that must be the key to it. 

I haven't gotten much education on statistics except for a class I took 12 years ago in undergrad. I feel like I just understand it at a caveman level of "different letter in tukey test means we can count them as different." But I don't have a deep understanding of how this stuff really works. 

I feel like I must have missed a step or something, which is why my results are slightly different from his. Also according to my notes in the Fixed Effect Tests for the pH weeks only the "Species" category had a Prob > F that was significant (red) when we did it together, but when I tried it by myself just now Species AND Treatment are red... so I know something funny is going on.

If it helps, the experiment was to have five different species of plants in microcosms (buckets) which we watered with three different levels of pollution (0%, 30%, and 70%) and monitored their health and the conditions of the water they were in for 12 weeks, including pH. 

I hope I've explained my problem well! Please let me know if there's any other information you need from me. that could help clarify things.

Thank you so much for your help! 

[#2]

Hi @PersuasionRam10 ,

  Welcome to the community pages. Hopefully we can help you out. But, one thing that makes it hard is not having your data and being able to reproduce both instances of what you are experiencing.

  Are you able to share the data -- even anonymized would be helpful.

  As you are learning how to use JMP in your analysis, one thing I HIGHLY recommend is that whenever you perform any kind of analysis and you might not remember all the little steps, you can always got to the red-hot button (triangle) next to the top-most level analysis (e.g. Fit Y by X, or Multivariate) and select Save Script > To Data Table. You can then go back into that script and see what all the settings were that got you that analysis. It's extremely helpful and saves you time.

  Anyway, if you can share the data in an anonymized table (Menu: Tables > Anonymize), this could be a very good place to start so that others can try and repeat what you are getting. If you do anonymize it, it might be helpful to let the community know which columns to use for which variable, e.g. Y or X. You can keep the important details hidden, but we would still need to which columns need to be used where. Below is a screenshot of the Big Class.jmp sample data as original and as anonymized so you know what it does to the data table.

Hope this helps!,

DS

[#3]

Hi there, thanks for your help! Sure, I can share my data. I've attached a .jmp file here, is that what you meant by sharing the data? (I just made sure it didn't have any names or anything in it, I don't mind having the column names and stuff out there since this is just one part of a larger project.)

I need to find out if there is a significant difference between Species on pH, what effect the Treatment has, and if there is a Species*Treatment effect, and I've been using tukey tests for it. 

Also thanks for the tip about sharing the script. I went ahead and did so.

[#4]

Hi @PersuasionRam10 ,

  Thanks for uploading your data table. After taking a look at what you have, as well as reading through what you've posted, I have a couple observations/questions.

1. What version of JMP are you using? I know you're using a student edition, but what's the version number? I ask because in some updates, JMP will correct certain calculations, which might lead to slightly different values (like with Prob > F), but they usually differ in higher decimal places -- like 5th or 6th. You might also want to find out what version the stats professor is using. For example, I'm using JMP Pro 19.0.3, and I'm getting 6.9819 for CONTROL with the Least Sq Mean when doing the Tukey HSD test on Species, and your screenshot shows 6.9810 for that same value.

2. Based on one of your screenshots, it looks like you've put one column in as a Random Effect, is this true? You show a snippet of the report that reads "Fixed Effect Tests", which means one variable must be included as a Random Effect. Which one are you using? Is it Block? Or is the column you're using given a different Attribute, like Mixture Effect or Excluded Effect?

3. Was this a designed experiment?  Was it a DOE? If so, what model did you start out with in the DOE? This is an important starting point as well -- if it was really a DOE, which it kind of looks like because if you look at distributions of Species, Treatment, and Block, it does look like things are distributed across the different factors evenly.

4. I can reproduce your numbers in your screenshots where you compare your results with the others that you got with the stats professor, but I can't reproduce the left side of the screenshots. I'm guessing this is where something related to the random effects is coming in, but we don't know what's been used as a random effect.

5. Related to 2 and 4: I am not reproducing your DFDen values in your Fixed Effect report that you shared, and this is likely due to what you're defining as Random and the other model effects your defining. I am seeing that Species*Treatment is having a large Prob > F value and very low Logworth value, so it's likely not an important factor (at least for Mean(pH,T1) -- I haven't looked at other Mean(ph,T*) columns yet because I can't reproduce both instances that you're showing.

6. Are any rows being excluded from the analysis? This could impact some of the calculations.

7. Are you performing any transformations on your columns, like Log for your Y or Model Effects?

Thanks!,

DS

[#5]

1. Let's see... looks like I'm using JMP Student Edition 19.0.1. I'm not sure what version my mentor uses, but we did it all on my computer so it was all 19.0.1.

2. Yes, that's right. I made all the ones in Construct Model Effects with Block random effects. So Block, Species*Block, and Treatment*Block were all random. 

3. I think so... I'm not sure exactly what a "designed experiment"/DOE means here. It was an experiment that I set up where we had four blocks of 18 buckets that had different plants in them, or no plants, and gave them different levels of pollution. 

4. To my knowledge, the left side of the screenshot is also from the same excel spreadsheet as the one I attached, and we made the Block effects random in that one too. 

6. I don't think so... all 72 rows should be included. 

7. Just checked all my notes... looks like we didn't logarithmically transform anything this time. 

Thanks! :smiling_face_with_smiling_eyes:

[#6]

Hi @PersuasionRam10 ,

  Thanks for the additional information. Also, just an FYI, if you use the @ symbol, you can tag specific people in the community forums. This helps as it notifies that person of a communication.

  As for your data, The only thing that I can come up with is there is a typo somewhere in the Treatment column for the CONTROL Species. If you look at the distributions of Species, Treatment and Block, there is one extra 30% treatment in Block 2 for CONTROL than in all the other blocks. Was that by design, or is that a typo? I've tried duplicating the Treatment column and changing either row 30 or 34 to the alternate value, but this still doesn't allow for me to reproduce your results that are on the left of your original post. I can perfectly reproduce the results shown on the right side of your post, though.

  Maybe go back to the original raw data and make sure there aren't any typos or mistakes that were analyzed incorrectly.

  A DOE is a design of experiment and they're used to test out many factors that might influence a response, and to do it in an efficient manner. In fact, when I create my own DOE based on your 3 factors: 6-level Species, 3-level Treatment and 4-level Block, and force 72 runs, JMP generates a DOE that is evenly distributed across all three factors. That's one reason why I'm thinking there is a typo, or more than one.

  Did you generate the data table that you shared by summarizing a different data table? Going back to the original raw data can sometimes help determine where something was done differently.

  Otherwise, I'd say your approach sounds right. Just double check using the raw data that what you are using for your analysis is the correct data set.

Hope this helps!,

DS

[#7]

@SDF1 

Hi there! Just wanted to say thanks for helping me a couple weeks ago. And I found the answer for why my stuff was slightly different! I needed to change the Block from "Continuous" to "Nominal" in the dialogue window. When I did this, the Control's least square mean in the tukey test went back to the number it was supposed to be. 

[#8]

Hi @PersuasionRam10 ,

  Glad to hear that you were able to reproduce the results from before. Something to be aware of, though is that I believe JMP does restrict random variables in models to continuous (maybe also ordinal), but I don't think it allows for nominal variables to be random. But, glad to hear that you 've figures some things out and are able to move forward with your work!

DS