topic Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares? in Discussions

Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Thu, 03 Apr 2025 16:53:34 GMT

Hi all,

I fitted a simple model with one continuous and one nominal variable using Standard Least Squares. Now I am wondering why the sums of squares seem to mismatch under Analysis of Variance and Effect Tests. I am on JMP 18.0.1.

3.16 and 0.01 do not give the overall model's SS of 3.37. It's like some 0.2 are missing. Could anyone please explain why that is?

This particularly puzzles me because I am looking at analysis of the same data in minitab provided by colleagues, and there the factor "Month" indeed gets the increased SS of 0.209. All the other stuff like DF and SS of everything else matches. In the end, this has implications for the p-value to judge the significance of "month" for the fit, which is why this bothered me in the first place.

Regards,

Konstantin

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

statman — Thu, 03 Apr 2025 17:38:37 GMT

It would really help to look over your results if you attach the actual data table instead of just the outputs. A little bit like "show your work" in school.

One observation I can instantly make in your comparison with Minitab is Minitab is reporting sequential SS (type 1) not partial SS (type 3). Sequential SS are order dependent, while partial SS are more robust to model order (and are preferred and usually the default). JMP will use type 3 as the default and also uses REML (I think Minitab is still on the EMS estimation platform).

I'm not sure what your factors are, and which are continuous and which are nominal. Neither batch nor month seem continuous? You have 9 batches and 2 months. but I can't tell where your DF for MSE is from?

If you look at the parameter estimates table and add the VIF column, what are those values?

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Fri, 04 Apr 2025 06:55:45 GMT

Thank you very much, it is much clearer already!

I have attached the data to this reply. The VIFs are fairly small.

Your observation is very enlightening to me. Yesterday I was actually wondering if the order of the model plays a role, and switched month and batch around, but JMP did not care. Perfectly makes sense with your explanation with Type 3. I now also understand that the effects in JMP are adjusted for the presence of other effects, while minitab does not do this, which explains the difference. After removing "batch" from the model, JMP gives an SS of 0.2 for "month", which is what minitab outputs from the model with both factors.

I believe this question is resolved.

However, I have also been wondering similarly as you about the DFs. Month is a continuous factor. So why does it end up with a DF of 1? Maybe you get an idea after looking at the data.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

MRB3855 — Fri, 04 Apr 2025 10:46:15 GMT

Hi @Bashburz1 : "However, I have also been wondering similarly as you about the DFs. Month is a continuous factor. So why does it end up with a DF of 1?"

That's because as a continuous factor, it only introduces one parameter to the model; the "slope". i.e., the term that is multiplied by Month.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Fri, 04 Apr 2025 11:16:43 GMT

Got it, thank you!

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

statman — Fri, 04 Apr 2025 18:06:37 GMT

BTW, if you change the data type for month to nominal you will get 9 DF's for month. I'm not sure why you consider month a continuous variable?

Also if you click on the red triangle next to the response in the Fit Model output, select Estimates>Sequential Tests, you will get type 1 SS.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

MRB3855 — Fri, 04 Apr 2025 20:01:02 GMT

Hi @statman My educated guess is that “month” is indeed continuous; it is actually time since the sample was put on stability…I.e., in a chamber at a given temperature and relative humidity. But its units are months (could be seconds, minutes, days, weeks, etc). I’m guessing this is stability analysis (shelf-life) in the pharmaceutical industry. Am I correct @Bashburz1? if so, “month” is continuous.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

statman — Fri, 04 Apr 2025 20:10:35 GMT

Hmmmm, interesting, if you are concerned with stability, I wouldn't model month with a linear model. I would use sampling ideas and control charts for stability assessment. What would it mean if month was significant? Was there an unusual month (special using Deming's terminology) or is there just a lot of variation month to month?

You certainly can't complete the study and say month is significant and month 1 is better, let's run there. Now if you were to ask the question in a baking process does time have an effect, you would certainly consider that "time" a continuous factor.

But admittedly I am not a "shelf-life" SME.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

MRB3855 — Fri, 04 Apr 2025 20:35:20 GMT

@statman The appropriate analogy is your baking process; the linear model for “stability” (that’s the term used in the industry for what is commonly referred to as shelf-life…not to be confused with a “stable process” per QC lingo) is an ANCOVA model with time as a covariate.

Probably more than you asked for, but here ya go!
https://database.ich.org/sites/default/files/Q1E_Guideline.pdf#page13

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

statman — Sat, 05 Apr 2025 16:11:47 GMT

Thanks for the light reading this morning. Wow that is obviously guidance written by committee.

"The purpose of a stability study is to establish, based

on testing a minimum of three batches of the drug substance or product, a retest

period or shelf life and label storage instructions applicable to all future batches

manufactured and packaged under similar circumstances".

Where did 3 come from? 3 batches is representative of all batches...in what world?

As you note, the use of the word stability is confusing to me and not defined in the document. It appears to be directed towards the deterioration of the drug efficacy?

I like this section:

"Data not amenable to statistical analysis"

In any case, to the OP, if you are following this guidance, month should be the covariate and you should use sequential tests on the covariate, then partial SS (type 3) for the remaining factors in your study.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Mon, 07 Apr 2025 06:18:10 GMT

That's right, month is simply the unit of time.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Mon, 07 Apr 2025 06:36:20 GMT

Yeah, "stability" is to monitor any decline or change of any attribute of the drug over time. Thanks for the tip about using the sequential tests in JMP. In this context, you usually have the three (maybe historically for practical reasons) batches and are interested whether they behave the same. So, you start off with batch, time, and their cross-term. You first test if you can throw out the cross-term, meaning all slopes are equal. Then, you would like to throw out "batch", i.e., show that the intercepts are equal. You throw the datapoints all in one plot and fit a line. Sometimes you can have more factors, which you would ideally like to have no influence. Thus, month/time is usually the last man standing here.

I've seen a great course here on the JMP site about mixed models and how they can be applied to this topic, but it comes with a disclaimer that such an approach is simply not compliant with the mentioned guideline, so there we go :)

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

MRB3855 — Mon, 07 Apr 2025 09:50:45 GMT

Hi @Bashburz1 : not sure if you are aware, by there is a module for stability analysis in JMP.

https://www.jmp.com/support/help/en/18.1/index.shtml#page/jmp/stability-analysis-in-the-degradation-platform.shtml

It does the same thing as in the fit model platform, but it has a few options that are specific to stability analysis per ICH Q1E.

Re: Why does the Sum of Squares under "Effect Tests" not add up to the overall model's Sum of Squares?

Bashburz1 — Mon, 07 Apr 2025 10:45:54 GMT

I wasn't aware, thank you! It's still neat to understand the steps and do it manually, but that module literally does this job for oneself.