topic Re: Repeated measurements...sort of... in Discussions

Repeated measurements...sort of...

gchesterton — Thu, 08 Jun 2023 20:58:50 GMT

Suppose I have a 2^3 full factorial design with replicate, so 2x8=16 runs. Each run lasts 30 minutes, where participants are detecting and prosecuting 8 'targets' over the course of that 30 minutes. Each of the 8 targets can conceivably have its own response time. Since the target profiles vary, I am randomizing their order for each run. I don't care about a target type factor, so it is not a design factor for which I am trying to measure an effect. I am trying to detect main and 2-way effects.

Does it make sense to treat those 8 response times per run as independent observations, as if I was getting 8 response time data points for each factor level, for a total of 16x8 observations? Or instead average the 8 response times for a single observation per run? If I can use 8 observations per run, are they best treated as repeat measurements for a run? Because they're not repeated measurements -- each target is unique and I know in advance they will not produce the same response time. All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance.

Re: Repeated measurements...sort of...

Byron_JMP — Fri, 14 Feb 2020 03:32:47 GMT

Trying to understand your experiment.

It sounds like you have 16 experimental units.

Within each unit some action is repeated 8 times.

1. All 8 actions are dependent on the conditions of the experimental unit.

The goal of the experiment is to measure the effect of the experimental conditions on the measured actions.

or...

2. The experiment results in a response curve with 8 points over 30 min.

The goal is to understand how the experimental conditions affect the response curve.

I think you might be working with case 1. Stacking all the rows for the 16x8 data points is fraught with danger.

I suspect you have your data table set up one column for each experimental variable, and one column for each of the 8 measurements. One option is to treat this as a mixed model and analyze the between and within variation in the 8 times. Another option is to use the mean and standard deviation of the 8 measurements as the responses.

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 03:39:29 GMT

It's definitely case 1 - The goal of the experiment is to measure the effect of the experimental conditions on the measured actions. I too thought it would be wrong to stack the observations as if I had 8 observations for each factor's level.

If I were to take the mean and s.d. of the 8 observations, then my response variable becomes...the mean? So would I then have 8 means for both levels of a factor, and testing for a significant difference of the means of those means?

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 03:46:40 GMT

I should have avoided causing confusion in the example I provided by instead supposing that I have 10 targets per run. If those were somehow expressed as a single summary value (mean), I would have 1 mean for each of the 16 runs. If I treated them as separate values, I would have 10 values for each of the 16 runs. I'm trying to think ahead for how I would be analyzing the results in both those cases if my goal is simply to verify a factor effect (for each of the 3 factors).

Re: Repeated measurements...sort of...

Byron_JMP — Fri, 14 Feb 2020 04:26:08 GMT

Taking the mean of the (10) replicates for each run (1 of 16) will give you a better estimate of the mean time to acquire a target, or time between acquisition, however you are scoring the events...

If you wanted to also account for the order of the targets, in the case of the targets not all being identical, there is a little different experimental design for that. (experimenting order factors).

At the 2019 discover conference, Kevin Gallagher, a Scientist at PPG Industries, presented a really nice case study on this topic. https://community.jmp.com/t5/Discovery-Summit-Tucson-2019/The-Design-and-Analysis-of-Experiments-With-Order-Factors-2019/ta-p/223351

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 06:48:28 GMT

Thanks very much @Byron_JMP , I'll check out Kevin's presentation. I think the mean will be sufficient. Another approach would be to take the max or 90th percentile (of the 10 targets) rather than the means. The thinking here is that the largest of the detection times are the more worrisome values, and a desired effect should be a reduction in those more critical values. Using the max (of the 10) would be a risky value -- outliers will be troublesome. Maybe the upper quartile value or 90th percentile would be a good compromise.

Re: Repeated measurements...sort of...

Mark_Bailey — Fri, 14 Feb 2020 13:33:52 GMT

Alternatively, you might consider target profile a factor. That is, it has an effect on the response. It sounds like you do not consider target profile to have a fixed effect. While each treatment includes the same eight profiles, the profiles represent a random sample from a population of possible profiles. They introduce additional variation to the response, a random effect. This approach would give you additional information: variation across and within profiles.

Re: Repeated measurements...sort of...

Byron_JMP — Fri, 14 Feb 2020 13:46:14 GMT

@Mark_Bailey How would you structure the data to include target type as a random effect?

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 14:53:50 GMT

Thanks @Mark_Bailey , I think that's the right way to think of it. I am re-using the same 10 target profiles, just randomly varying their order to reduce operators' predictability. I want to be able to remove the target profile source of variance, but I don't need to measure the target profile effect. So, does that mean I simply have a profile variable (column), and produce 10 observations (rows) per run...one observation for each target profile, for each of the 16 runs? That would mean a data table of 10x16 = 160 rows if I'm following your logic. Do I need to tell JMP to treat the target profile variable any differently either in building the design or in the ANOVA?

Re: Repeated measurements...sort of...

Mark_Bailey — Fri, 14 Feb 2020 18:43:23 GMT

Yes, enter the response for each target as a separate observation (row). Add a Profile column with the nominal modeling type. Add the Random Effect to this term in the Effect list. Do not add any term for the replicate - that will be used for the error estimate (repeatability).

How does that work?

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 20:06:05 GMT

Thanks again @Mark_Bailey I asked JMP to create the table with some fake data, then modeled the results just to implement your recommendation. With the output table, I ran two models -- one model with the three main effects terms and three 2-way terms, and the target profile term as a random effect, and one model where I simply excluded the target profile term from the model. In the first model dialog, JMP recommended REML. I am using the SLS personality and effect leverage emphasis. In the second model, without the random effect term, JMP did not prompt me for a method.

In any case, the results in both models are quite similar -- perhaps because the data were meaningless since they were synthetic. All main effects and interactions were significant at 0.05 in both models. That seems counterintuitive -- its seems like treating the profile as a random effect term should have yielded much different results. But perhaps I am comparing apples to oranges in the way that I am modeling the data with these two models. In model 2, I am probably making bad conclusions since there's an effect I didn't account for. But I would have thought that would have resulted in fewer significant effects.

Re: Repeated measurements...sort of...

statman — Fri, 14 Feb 2020 20:12:00 GMT

Late to the party, but there are a number of things that come to mind, though I can't say I understand what exactly you want out of the experiment.

But this sounds interesting...so if you'll indulge me I have some questions (of course if you are happy with the responses you have received, I understand):

What are responses are trying to understand? Response time? Rate of response (from detecting to prosecuting)?

What is a "target profile"? And why don't you care about the interaction between the design factors and the target profile?

Aren't you trying to choose design factors, that affect response time, robust to target profile?

It seems you're only interested in the mean? Why not the variation? It is impossible for me to think about means without some idea of the variance...it is like giving the score for one team and not the other in a sporting event.

If you are using the mean, shouldn't you first test to see if that is the appropriate statistic? What if there are unusual data points in the 8 target profiles? Then the mean might be a poor summary statistic.

You are replicating the design, why? Do you know what noise is changing between replicates? If so, wouldn't you be interested in knowing the effect of that noise and possibly noise-by-factor interactions (think robust design). Treat the replicate as a block and a fixed effect. If not, then I understand the use of the replicate to get an "unbiased" estimate of the MSE.

A further thought is to treat the design structure as the whole plot and the profile factor as the subplot of a split-plot design. I would think you would want to know more about this source based on the statement. "All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance"

Re: Repeated measurements...sort of...

Mark_Bailey — Fri, 14 Feb 2020 20:24:23 GMT

It should not affect the estimates of the fixed effects that much. Your design is orthogonal. It is like having two error terms, within and between targets. It is just more detailed information about the random variation.

Re: Repeated measurements...sort of...

gchesterton — Fri, 14 Feb 2020 21:01:15 GMT

Hi @statman jump on in.

@statman wrote:
What are responses are trying to understand? Response time? Rate of response (from detecting to prosecuting)?

It's a command & control scenario where detectionTime is one (of many) measures of the effect of several design factors.

What is a "target profile"? And why don't you care about the interaction between the design factors and the target profile?

The target profiles add variety and reasonable error to the experiment, but they are simply a small sample from a large number of potential profiles. I don't really need to know whether there is a target profile effect.

Aren't you trying to choose design factors, that affect response time, robust to target profile?

well, yes

It seems you're only interested in the mean? Why not the variation? It is impossible for me to think about means without some idea of the variance...it is like giving the score for one team and not the other in a sporting event.

Hence my earlier concern that collapsing a bunch of values to a mean does not seem like a good idea. Perhaps a 90th percentile is more approp if I am trying to characterize a sample with a single value.

You are replicating the design, why? Do you know what noise is changing between replicates?

It's just the randomness associated with human operator actions. Two runs with the same factor levels will definitely yield different results, because the response is affected by human actions and decisions.

A further thought is to treat the design structure as the whole plot and the profile factor as the subplot of a split-plot design. I would think you would want to know more about this source based on the statement. "All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance"

I tend to think of split-plot designs for cases where I have a hard-to-change factor, which is not the case here. This is a simulation where we can control the factors fairly easily from one run to the next.

Re: Repeated measurements...sort of...

statman — Sat, 15 Feb 2020 15:52:55 GMT

You need to read "Split-plot designs for robust product experimentation" Box and Jones, Journal of Applied Statistics, Vol. 19, No.1, 1992. Hard to change factors is only one reason to do split-plots (and not the best reason). Restrictions on randomization can be used effectively to partition the noise in an experiment to simultaneously increase precision for detecting factor effects while not negatively affecting inference space.

When you are calculating the mean to use as a response variable, also calculate the standard deviation (or variance) and us eat as a response variable.

It seems you ultimately want to understand human actions and decisions. Randomizing those will not allow for estimating those effects nor will it allow understanding of those effects.

Re: Repeated measurements...sort of...

gchesterton — Sat, 15 Feb 2020 18:52:06 GMT

@statman

Ok, so in the split-plot option, I suppose I'd develop say, 6 scenarios that involve various target profiles. Those scenarios would be the main plots and would occupy a day. Then within those main plots, I would have split plots for levels of the system design under investigation: system A, system B.

day	1	2	3	4	5	6
am	A	A	B	A	B	B
pm	B	B	A	B	A	A
Scenario	5	2	3	1	4	6

What are the real benefits of this approach, if as is my case that changing the scenarios is not difficult?

Re: Repeated measurements...sort of...

statman — Sun, 16 Feb 2020 15:31:54 GMT

It sounds like you are changing your original description of the situation. For your proposal, you are confounding the "scenarios" with Day (and the associated noise) which would not be a benefit. What I am suggesting is for each whole plot treatment combination (the factorial stated in your original post, I believe 2^3), run a set of treatments based on a factorial of target profiles (not sure what changes profile -to-profile, but I would think you could identify what those x's are and experiment on them). Instead of repeats, the factorial of the profile factors would create the sub plot. Since the whole plot and the sub plot degrees of freedom are analyzed separately (e.g., each with their own normal plot), you have increased the precision of the whole plot (less noise in just the whole plot) and increased the precision of the sub plot as well. Also you will get WP by SP interactions, so if the factors in the whole plots effects depend on which target profile was being used, you would be able to identify and quantify this.

Re: Repeated measurements...sort of...

gchesterton — Sun, 16 Feb 2020 18:28:44 GMT

I think I see what you mean, but I may be misinterpreting your proposal. It would help to see your design in a graphic if you wouldn’t mind.
To be clear, our 30-minute runs consist of multiple targets and the only real reason we’re doing multiple ‘profiles‘ is so that there is operational variety and to help avoid a learning effect. If there’s a more common way to handle such issues, then I’d like to hear about those rather than unnecessarily force the experiment into a particular design.

Re: Repeated measurements...sort of...

statman — Mon, 17 Feb 2020 15:40:35 GMT

I have attached a picture of the Factor Relationship Diagram. This includes the block, X1-X3 design factors in the whole plot and one factor for the Target Profile (TP) in the subplot (TP1 could be TP1-TP3 in a factorial if you want 8 different configurations of the target profiles, e.g, target size, target distance, target color). Or since you don't care about resolution of the TP factors (you're just trying to create a large inference space) you could do this in 4 treatments (half fraction of the 3 TP factors) per 30 minute run. Or you could take advantage of split-plot confounding to further reduce the number of treatments...with very little sacrifice of resolution. I would need to know more specifics of your situation.