Repeated measurements...sort of...

[#1]

Suppose I have a 2^3 full factorial design with replicate, so 2x8=16 runs. Each run lasts 30 minutes, where participants are detecting and prosecuting 8 'targets' over the course of that 30 minutes. Each of the 8 targets can conceivably have its own response time. Since the target profiles vary, I am randomizing their order for each run. I don't care about a target type factor, so it is not a design factor for which I am trying to measure an effect. I am trying to detect main and 2-way effects. 

Does it make sense to treat those 8 response times per run as independent observations, as if I was getting 8 response time data points for each factor level, for a total of 16x8 observations? Or instead average the 8 response times for a single observation per run? If I can use 8 observations per run, are they best treated as repeat measurements for a run? Because they're not repeated measurements -- each target is unique and I know in advance they will not produce the same response time. All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance. 

[#2]

Trying to understand your experiment.

It sounds like you have 16 experimental units.  

Within each unit some action is repeated 8 times. 

    1. All 8 actions are dependent on the conditions of the experimental unit.

    The goal of the experiment is to measure the effect of the experimental conditions on the measured actions. 

or...

   2. The experiment results in a response curve with 8 points over 30 min.

   The goal is to understand how the experimental conditions affect the response curve.

I think you might be working with case 1. Stacking all the rows for the 16x8 data points is fraught with danger.

I suspect you have your data table set up one column for each experimental variable, and one column for each of the 8 measurements.  One option is to treat this as a mixed model and analyze the between and within variation in the 8 times.  Another option is to use the mean and standard deviation of the 8 measurements as the responses.

[#3]

It's definitely case 1 - The goal of the experiment is to measure the effect of the experimental conditions on the measured actions. I too thought it would be wrong to stack the observations as if I had 8 observations for each factor's level. 

If I were to take the mean and s.d. of the 8 observations, then my response variable becomes...the mean? So would I then have 8 means for both levels of a factor, and testing for a significant difference of the means of those means?

[#4]

I should have avoided causing confusion in the example I provided by instead supposing that I have 10 targets per run. If those were somehow expressed as a single summary value (mean), I would have 1 mean for each of the 16 runs. If I treated them as separate values, I would have 10 values for each of the 16 runs. I'm trying to think ahead for how I would be analyzing the results in both those cases if my goal is simply to verify a factor effect (for each of the 3 factors). 

[#5]

Taking the mean of the (10) replicates for each run (1 of 16) will give you a better estimate of the mean time to acquire a target, or time between acquisition, however you are scoring the events...   

If you wanted to also account for the order of the targets, in the case of the targets not all being identical, there is a little different experimental design for that.  (experimenting order factors). 

At the 2019 discover conference, Kevin Gallagher, a Scientist at PPG Industries, presented a really nice case study on this topic.  https://community.jmp.com/t5/Discovery-Summit-Tucson-2019/The-Design-and-Analysis-of-Experiments-Wit...

[#6]

Thanks very much @Byron_JMP , I'll check out Kevin's presentation. I think the mean will be sufficient. Another approach would be to take the max or 90th percentile (of the 10 targets) rather than the means. The thinking here is that the largest of the detection times are the more worrisome values, and a desired effect should be a reduction in those more critical values. Using the max (of the 10) would be a risky value -- outliers will be troublesome. Maybe the upper quartile value or 90th percentile would be a good compromise. 

[#7]

Alternatively, you might consider target profile a factor. That is, it has an effect on the response. It sounds like you do not consider target profile to have a fixed effect. While each treatment includes the same eight profiles, the profiles represent a random sample from a population of possible profiles. They introduce additional variation to the response, a random effect. This approach would give you additional information: variation across and within profiles.

[#8]

@Mark_Bailey  How would you structure the data to include target type as a random effect?

[#9]

Thanks @Mark_Bailey , I think that's the right way to think of it. I am re-using the same 10 target profiles, just randomly varying their order to reduce operators' predictability. I want to be able to remove the target profile source of variance, but I don't need to measure the target profile effect. So, does that mean I simply have a profile variable (column), and produce 10 observations (rows) per run...one observation for each target profile, for each of the 16 runs? That would mean a data table of 10x16 = 160 rows if I'm following your logic. Do I need to tell JMP to treat the target profile variable any differently either in building the design or in the ANOVA?

[#10 · Accepted solution]

Yes, enter the response for each target as a separate observation (row). Add a Profile column with the nominal modeling type. Add the Random Effect to this term in the Effect list. Do not add any term for the replicate - that will be used for the error estimate (repeatability).

How does that work?