The Effect test

[#1]

Hi All,

I have been wondering if anyone could help me to read below results from the Fit Model/ the effect Test

Basically, I have a set of data from the last 3 trials and I would like to find out if there is any difference between method of weighing. Currently I use Auto weighers & Manual weighers at different ages. Therefore, I have used age & type of weigher as a factor affecting the bodyweight results. I would like to find out if the differences seen between those two weighers are due to type of weigher being used or not.

Thank you!

Trial 1

Effect Test	Nparm	DF	Sum of Squares	F Ratio	Prob > F
Type of weigher	1	1	0.5328	1.3884	0.2399
Age (weeks)	6	6	5364.0025	2329.370	<.0001*
Type of weigher*Age (weeks)	6	6	5.6173	2.4394	0.0264*

Trial 2:

Effect Test	Nparm	DF	Sum of Squares	F Ratio	Prob > F
Type of weigher	1	1	0.691	6.0384	0.0145*
Age (weeks)	8	8	11195.978	12237.44	<.0001*
Type of weigher*Age (weeks)	8	8	5.641	6.1657	<.0001*

Trial 3:

Effect Test	Nparm	DF	Sum of Squares	F Ratio	Prob > F
Type of weigher	1	1	0.0290	0.2056	0.6506
Age (weeks)	7	7	8635.3296	8733.892	<.0001*
Type of weigher*Age (weeks)	7	7	4.7245	4.7785	<.0001*

[#2]

What is a 'trial?' Why do you analyze each trial separately instead of combining the trials into one data set for analysis?

The effect tests are hypothesis tests of the null hypothesis, which is there is no effect. That is, the parameter estimates equal zero. The test is performed by estimating the mean sum of squares twice. The first estimate (mean sum of squares error) is independent of any effects. The second estimate (mean sum of squares model) includes a contribution of the effects. The ratio of these two estimates (F ratio) is near 1 when there are no effects. The ratio is greater than 1 when there are effects. The conclusion about significant effects is usually accomplished by determining ɑ prior to the test and then comparing the p-value for the F ratio to this level. A common, but not always appropriate, ɑ = 0.05 and JMP assumes that level for the format of p-values.

If I use ɑ = 0.05, I conclude that:

• there is no effect of Type of Weigher,
• there is an effect of Age,
• but the effect of Age is not the same for the two types of weighers.

Please see Help > JMP Documentation Library > Fitting Linear Models guide > Chapter 3: Standard Least Squares Report and Options.

[#3 · Accepted solution]

Mila,

If you attach your data table, we could have a look at writing the correct model for analysis.

[#4]

Hi,

I have attached the data in excel file.

I have gather all the body weight data from the last 4 trials (Auto & Manual weighing).

I would like to find out if the differences seen between Auto & Manual at certain ages are due to method of weighing or number of birds being weighted or maybe other factor. 

Also, I would like to find out if Auto weights are more accurate than Manual, as more birds have been weighted, but at the end of the trial (16 -20 wks) less birds are using Auto scales or it is the other way round.

I have been wondering, if pen number should be used in the model as a Random factor (?)

Background:

• up to 6 wks of age 140 birds/pen and after 8 wks of age 50 birds/pen are weighted, using manual weigher. Whilst, Auto weigher record more weights than the Manual (data in the excel spreadsheet)
• up to 6 wks of age 700 birds per pen  is present; from 8 to 14 wks: 230 birds/pen and from 14 to 20 wks: 176 birds/pen
• overall there are 20 pens

I would appreciate any suggestions or solutions.

Thank you very much in advance.

[#5]

Why do you not provide your data in a JMP data table?

What was your analysis plan at the start of this study?

Please see Help > JMP Documentation Library > Quality and Productivity guide > Measurement System Analysis chapter for lessons about determining accuracy of measurements with JMP.

[#6]

Hi,

Thank you for your suggestions.

To be honest I do not know how to upload the JMP table. Therefore, I have uploaded excel spreadsheet.

I have been planning to check when the differences between  Auto & Manual weights become significant (if it at certain ages or maybe across the board).

Also, I have wanted to determine if the differences are due to method of weighing or not (?) or maybe due to the number of birds being weighted (Auto will record more birds, but I am aware that the same bird might be weighted several times, when with Manual weighing every bird is weighted once. What is more, less birds is keen use the Auto scales in the later stage of life)

As mentioned there is a variance in the birds numbers. At the moment our trials are based on manual weights (20% of pen is being weighted), but we would like to move away from this method and be able to relay on Auto. Therefore, I try to figure out how to present the data which would answer the question if we can rely on Auto or not.

[#7]

In reply to this thread:  

"When I have looked at the data by each trial the “pen” seems to have significant effect, but when I combined all the data there is no significant effect of pen (I do not understand why it would be like that?)

One of the member from JMP community (@mark_baily) kindly did some analysis for me (please find attached – word document), but again I do not fully understand it.

I have been wondering if you could look into it and let me know what would be the best way to handle these data."

First of all, it is best not to start a new thread on the same topic. It just gets confusing for everyone that wants to help you. And for anyone trying to make sense of this to help them with their problem.

Secondly, please attach the JMP data table. You can attach a jmp file in exactly the same way as you would a .xslx file.

I have attached the data that you shared as a JMP table.

In the table there are 2 model scripts: 1 for the model on all trials (trial is a random effect) and 1 for a separate model for each trial.

Yes, different things are significant in these 2 ways of modelling the data. That is not unusual. A simple explanation is that for a specific trial there might be an important difference that is averaged out when you look over all trials.

Personally, I think it is a REALLY good idea to first of all plot your data, if only to check the quality. Have you done this?

I found this plot informative (script also in table). For example, Trial 5 and Age 16-17 data looks worth investigating.

 

Regards,

Phil

[#8]

I imported your workbook and changed some data column attributes in JMP.

I used a Partition model to assess important variables. Age is by far the most important variable.

I used all the variables as predictors in a linear regression model for body weight. I treated trial and pen as a random effects and all the rest as fixed effects. Here is the initial analysis:

The variance components analysis of the random effects suggests that there is no significant effect (difference) between trials or pens. Age exhibits a non-linear effect. There is a difference between the type of weigher and a dependence on the number of birds weighed.

This plot shows the predictive performance of the model:

I also examined the residuals to look for issues with the data or model assumptions.

You can see in the detail that there is still some bias in the model for the lowest body weights. There is also a cluster of weights in the middle of the range that are distinct from the other neighboring weights.

I assumed that the data were recorded in the order they were generated. If so, then there are still patterns to be investigated. I colored by Trial and marked by type of weigher.

[#9]

You have not identified "trials" in your excel sheet? What is a trial?  I tried to figure out what a trial was based on the repeated measures of age, but when I looked at that pattern, I found 7 trials?

Accuracy is  "the average deviation from the true value".  This is typically done for calibration purposes.  You cannot answer the question of accuracy with this data set.  Perhaps you are interested in precision?  You don't have any record of wether the same bird is weighed more than once.  In order to assess precision (repeatability) of the devices, you will need to have the same bird measured twice on the same device (also in a short time period, not over different ages).  In order to assess precision reproducibility, you will need to have the same bird measured on both devices in a short time period.  There is no record of this in the data set, so you cannot answer questions about measurement system precision.

The only recorded Y is my guess, average weight.  Whenever you are taking an average of multiple measures, it is important to understand the variation and distribution of the individual values.  If there are "special cause" data points, they can distort the averages.  If the distributions are grossly not normal, then an average might be an inappropriate measure of central tendency.

You don't have any data for the automated weigher at 6 weeks. Number of birds weighed is confounded with Type of weigher (Manual is always 50 and Automated is some random #).  Your background information is confusing:

"up to 6 wks of age 140 birds/pen and after 8 wks of age 50 birds/pen are weighted"

"up to 6 wks of age 700 birds per pen  is present"

Perhaps you should normalize the data based on population of birds?

As Mark suggests, you should get a better understanding of measurement system studies.  I recommend reading:

Wheeler, Donald and Lyday, Richard “Evaluating the Measurement Process” SPC Press (ISBN 0-945320-06-0)

I've attached a simple graph of the data.  Additional analysis without further understanding would be premature.

[#10]

Hi Milena,

Just adding from our email exchange for the discussion on here:

Basically, the aim of my project is to establish whether the Auto weighing system can replace the Manual weighing, completely or maybe at certain ages (?)

• Therefore, you’d then need to specify that you are interested in the effect of “Type of weigher” and the interaction effect between “Type of weigher” and “Age”.  
• if the “Type of weigher” factor is significant then there’s differences between the measurement systems
• if additionally the interaction factor is significant then difference changes with the age of the birds.
• Also take into account @Mark_Bailey 's important point about the "age" by "age" quadratic effect.

and also which system is more accurate Auto or Manual (?). How the “noise” in the data can be removed.

• This is what you can’t answer as you don’t have a gold standard measure, that is, a exact value of the weight of the birds in the pen. You have three measurement systems but each on of them is an approximation of the true weight of all the birds (the gold standard measurement).
• If you had the gold standard measure, you could calculate the difference between that and the measurement systems and use this as the outcome.

There are areas which I think play a key role:

Number of birds (at 6 weeks: 140 birds/pen are weighted manually, and from 8 to 20 weeks only 50 birds per pen). Whereas Auto gives an average over 24h period and there is a chance that the same birds were weighted several times

• You can’t test this as a factor, “No. of birds weighed” for Auto is random and Manual it’s controlled over the trial. Unfortunately, it also means that the “Type of weigher” and “Age” factor is going to be describing the variability caused by this change and not the validity of the measurement systems over the age of the birds.

Accuracy of weighers & calibration, although both are calibrated with calibration weights there are some discrepancies. For Example, before weighing I do 10kg check, so I record 10 readings of 10 kg and some of those readings fell outside the acceptable range of -/+ 10g

• You’d need to ensure that all weighers are accurate before using them in this study. This may be why some pens (2,6,9,13 and 19) are significantly different to others? Do they all have separate weighers? See @statman 's response regarding Measurement Systems Analysis.

I would like to find out what analysis can be done to answer a question: why in some Trials those factors at certain ages are crucial and in others not (?) (please find attached JMP table and power point slides with results).

• General rule of thumb, more data means more power to detect an effect, always use as much data as possible.

Also, I have been trying to carry out similar analysis for my colleague based on single trial, where pen was treated as random effect and age & type of the weigher as fixed effect. However, the outcome of analysis is different to Phils’, so I assume that I have done something wrong. Please find attached file called “broiler Trial 54”

• I don’t think Pen is a random effect, you seem to have 20 pens that you use for all trials? If so, then keep this as a fixed effect. If you're selecting 20 out of a possible 100 pens at random for example, then it's random effect.

With these comments and the thorough comments from @Mark_Bailey , @Phil_Kay and @statman above, this isn’t an analysis you can inference from currently.