topic Re: ANOVA vs. GLM for ecological field experiment in Discussions

ANOVA vs. GLM for ecological field experiment

cbhalpern — Sat, 10 Jun 2023 20:39:49 GMT

I am analyzing data from an ecological field experiment, designed to test the effects of herbivore (limpet) removal and moisture addition on change in cover of a rocky intertidal seaweed. The experimental design is a randomized block, two-factor full-factorial design. I have attached a document that details the experimental design and lists the scripts and model outputs relating to three questions for the Community Discussion:

Which modeling approach—Standard Least Squares ANOVA or GLM—is preferable for modeling results of an ecological field experiment? See Background.
Are the scripts I used to run each type of model constructed properly? See Scripts.
Given that the GLM (normal, identity) and ANOVA approaches both assume normal distributions, why does the GLM yield consistently smaller p values than the ANOVA? See Model output.

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Mon, 05 Oct 2020 21:34:12 GMT

The GLM is a generalization of the Ordinary Least Squares Regression. They should be essentially equivalent if your case meets the assumptions of ordinary regression. Otherwise, use the GLM with a different error distribution and link function.

If you get different p-values, then there is a difference in the model specifications.

Re: ANOVA vs. GLM for ecological field experiment

statman — Tue, 06 Oct 2020 14:29:27 GMT

My thoughts:

1. There is no right way to analyze the data. There are pros/cons to any analysis method. To over-simplify, ANOVA basically analyzes the magnitude of the effect while GLM analyzes both the magnitude and the direction of the effect.

2. I did not evaluate your scripts as I did not have the data set.

3. I'm not sure what you mean by "Given that the GLM (normal, identity) and ANOVA approaches both assume normal distributions". There is an assumption of the distribution of residuals, but not of the actual data set. ANOVA is fairly robust to non-normal distributions. First, don't fall in love with the p-value statistic. It is only an estimate. It is a result of a comparison of mean squares. Mean square of the treatment to the mean square error (both of which are also estimates). If you change the comparisons or estimates, the p-value will change.

Something you might try since you are treating block as a fixed effect...saturate the model with Block, L, M, L*M, Block*L, Block*M, Block*L*M. Use normal and Pareto plots to look for significant effects. his removes any mean square error bias.

Re: ANOVA vs. GLM for ecological field experiment

P_Bartell — Tue, 06 Oct 2020 14:54:11 GMT

To add to @Mark_Bailey and @statman 's comments/counsel have you plotted the data in a meaningful fashion BEFORE modeling? Since you conducted an experimental design, Factor and block vs. Response plots, simple distribution of the responses, and a response vs. experimental execution order are a minimum set of plots I'd suggest you create. With an eye towards answering the following questions, which ultimately are used to build process understanding AND help insure we've got clean data for modeling:

1. Do the factor/block plots suggest something that flies in the face of known biological/physical phenomena? Back in the day, I'd suggest to the engineers we look at these plots. If they show the moral equivalent of 'the plots suggest water runs up hill, well, we've got a problem'.

2. Are there any response outliers that suggest unanticipated lurking variables that may have crept into the conduct of the experiment?

3. Does the experimental execution order plot of responses suggest a trend or suspicious pattern that again there may have been a lurking variable within experimental conduct?

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Tue, 06 Oct 2020 20:18:53 GMT

Thanks for your reply, Mark. Unfortunately, selecting a different error/link option isn’t possible, because the data are change values (non-integer values, including numerous observations less than zero).

I'm pasting in the scripts we constructed for ANOVA and GLM. Could you tell us from these scripts how the model specifications might differ?

ANOVA

Fit Model(

Y( :My87Ja88Mo ),

Effects( :Blk#, :Limp, :Moist, :Limp * :Moist ),

Personality( "Standard Least Squares" ),

Emphasis( "Effect Leverage" ),

Run(

                              :My87Ja88Mo << {Summary of Fit( 1 ), Analysis of Variance( 1 ),
                              Parameter Estimates( 1 ), Lack of Fit( 0 ), Expanded Estimates( 1 ),
                              Scaled Estimates( 0 ), Plot Actual by Predicted( 1 ),
                              Plot Regression( 0 ), Plot Residual by Predicted( 1 ),
                              Plot Studentized Residuals( 0 ), Plot Effect Leverage( 1 ),
                              Plot Residual by Normal Quantiles( 0 ),
                              Box Cox Y Transformation( 0 )}

)

);

GLM

Fit Model(

Y( :My87Ja88Mo ),

Effects( :Blk#, :Limp, :Moist, :Limp * :Moist ),

Personality( "Generalized Linear Model" ),

GLM Distribution( "Normal" ),

Link Function( "Identity" ),

Overdispersion Tests and Intervals( 0 ),

Name( "Firth Bias-Adjusted Estimates" )(0),

Run

);

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Tue, 06 Oct 2020 20:24:21 GMT

The models might differ by the parameterization of the categorical factors. Please see these details.

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Tue, 06 Oct 2020 20:28:37 GMT

Thanks for your reply. Responding to your numbered statements:

1. Thanks for the clarification of the differences between ANOVA and GLM. GLM is relatively new territory for us.

2. We are enclosing one complete column of response data from the dataset, below.

3. Both our ANOVA and GLM models use the same treatments and error terms in the Fit Model dialog. If you time to test our scripts, perhaps you can explain why the p-values differ in the outputs.

4. Thanks for the introduction to the Effects Screening module, with its normal and Pareto plot options. Not sure how to interpret significant effects, but this may not be relevant to our case.

Block	Limpets	Moisture	Change/mo
1	+L	+M	0.5
1	+L	-M	1.2
1	-L	+M	0.9
1	-L	-M	0.7
2	+L	+M	0.5
2	+L	-M	0.2
2	-L	+M	1.7
2	-L	-M	0.1
3	+L	+M	-0.2
3	+L	-M	0.2
3	-L	+M	-0.3
3	-L	-M	-0.2
4	+L	+M	-0.2
4	+L	-M	1.1
4	-L	+M	0.5
4	-L	-M	0.0

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Tue, 06 Oct 2020 20:34:55 GMT

Also, the estimation procedure differs. The least squares solution is a closed form that is computed directly. The GLM solution is an iterative method for maximum likelihood estimates. Here is the same model estimated first with Fit Least Squares platform:

Here is the same model estimated with GLM using the normal distribution for the response with the identity link:

The parameterization is the same for the categorical predicts age and sex but the estimation routines differ. The former is based on sums of squares with t and F tests, and the latter is based on likelihood with chi square ratio tests.

The two models produce identical predictions of the response:

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Tue, 06 Oct 2020 20:37:13 GMT

In answer to your question, yes, we used JMP tools to look for outliers and heterogeneity of variances among the 2x2 treatment combinations. Responses vary among blocks (spatial units), as expected, hence the motivation for blocking. There is no replication of plots (tmt combinations) within blocks.Order of execution is not relevant, because treatments were applied at one time to all four plots within each block.

We also plotted means, sems, and individual block values for all response interval x block x treatment combinations (2 factors at x 2 levels each). Our goal with these models was to determine the underpinnings of patterns readily visible in the data.

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Tue, 06 Oct 2020 20:44:17 GMT

With regard to point 3, OLS is solving the normal equations directly. The results are based on sums of squares. This leads to the F test for the whole model. The normally distributed parameter estimates leads to t tests. On the other hand, the GLM is using MLE. The results are based on -2LogLikelihood. This leads to chi square tests (likelihood ratio tests).

So the estimation and tests use different methods. They are not expected to give identical p-values. Note that these values generally agree if not exactly.

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Tue, 06 Oct 2020 20:45:31 GMT

Regarding point 1, the main benefit of the GLM is that you are not restricted to a normal distribution model of the response. If your response is (conditionally) normally distributed, then just use OLS.

Re: ANOVA vs. GLM for ecological field experiment

statman — Tue, 06 Oct 2020 20:56:55 GMT

Box also liked to do the Bayes plots. Cuthbert Daniel was the first to use the Half Normal plots for the analysis of experiments. Essentially, if the null is true for all treatment effects, then their effects should be normally distributed (randomly) with a mean of zero. The Normal plot depicts the effects transformed so that effects that appear on a straight line are random effects and points that depart from the line are assignable. JMP adds Lenth's PSE line to the plot, but you must be careful with the interpretation.

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Tue, 06 Oct 2020 21:20:38 GMT

Thank you. This is very helpful. Will GLM always yield greater
significance (lower p) than ANOVA or would this depend on the structure of
the data (e.g., deviance from the assumption of normality)?

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Wed, 07 Oct 2020 11:59:05 GMT

No, GLM will not always produce smaller p-values than OLS for the same model.

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Sun, 11 Oct 2020 18:17:59 GMT

We have a follow-up question regarding whether one tests the same assumptions regarding normality in GLM as in ANOVA.

Our understanding is that to test the assumption of normality in ANOVA, one tests the normality of residuals. To do this, we would save the residuals to the data table, then use the Distribution tool to test normality.

Is there a corresponding procedure for GLM? If so, which residual output to use--Deviance residuals, Pearson residuals, Studentized Deviance residuals, or Studentized Pearson residuals?

Re: ANOVA vs. GLM for ecological field experiment

statman — Sun, 11 Oct 2020 18:51:57 GMT

I'm a bit confused by this "Our understanding is that to test the assumption of normality in ANOVA". There is NO assumption of normality of the raw data analyzed via ANOVA. The only assumptions are NID(0, variance). That is normally and independently distributed residuals with a mean of 0 and a constant variance. There are a number of ways to look at residuals:

1. in a time series,

2. vs. predicted

3. as a distribution,

4. leverage plots

There are a number of types of residuals as you have listed some. There is no right one to use. Easy enough to look at them all.

Re: ANOVA vs. GLM for ecological field experiment

Mark_Bailey — Mon, 12 Oct 2020 21:02:52 GMT

@statman describes the assumption for the linear regression model well. GLMs do not have any such assumption about the errors, so your concept of residual analysis does not carry over to the GLM. We can use the various residuals from the fitted GLM to check for outliers, influential observations, and bias, though. The GLM assumes that the response follows the conditional distribution that you selected (e.g., Poisson, binomial, normal) and maximizes the likelihood of the model parameter estimates.

It's different.

Re: ANOVA vs. GLM for ecological field experiment

cbhalpern — Tue, 13 Oct 2020 01:28:19 GMT

What would be the best way to learn about using the various residuals from the fitted GLM to check for outliers, influential observations, and bias?

Re: ANOVA vs. GLM for ecological field experiment

statman — Tue, 13 Oct 2020 14:45:12 GMT

Again, I'm not sure I understand the question? If it is about learning how to use residuals to determine adequacy of your model, there are many books and papers on the subject. First, a residual is the difference between the prediction from the model and the actual value. Regardless of any assumptions required or implied, in order for the model to be useful, you would hope that it does a fairly good job of predicting the actual results. From a model development perspective:

1. you would hope the model isn't biased high or low (the residuals would be distributed around 0),

2. you would hope the residuals have about the same variation around the model and this variation isn't fluctuating greatly ( the residuals have a constant variance)

3. you would hope there aren't any unusual data points not explained by the model (absence of outliers in the residuals)

4. You would hope the residuals didn't form some pattern or were related to each other (independently distributed)

If some of these hopes are not satisfied, you should seek to understand why. You should challenge the effectiveness of your model and how you can modify it to make it more useful (and better yet arrive at a better understanding of what is actually going on).