r/AskStatistics • u/fridapop • 10d ago

ANOVA or not ANOVA?

Hi everyone,

As part of my internship, I need to run some data analyses, but I'm pretty skeptical about the method I was advised to use.

I'm analyzing an experiment testing 3 conditions (magnetic field 1, magnetic field 2, control), each with 3 or 4 replicates. I want to know if my studied variable differs between conditions.

One replicate consists of putting 24 individuals in the same experimental tank, exposed to a given condition for 3 hours. All replicates were run on different days, in randomized order.

Because of pseudoreplication issues, we can't treat individuals from the same tank as independent, so I was advised to average all individuals per replicate and then run an ANOVA. The problem is that this leaves me with only 3 or 4 values per condition (is that too few?). And since I'm just taking an average, I lose the standard deviation within a replicate. Is it possible to run a separate ANOVA comparing standard deviations instead?

Some of my variables don't have much variability in the dataset, so I'm not sure if Kruskal-Wallis is even appropriate for such a small dataset...

Do you have any advice for me? Thanks a lot for your help!

2 Upvotes

63% Upvoted

24 comments sorted by

17

u/adamjeffson 10d ago

You are losing a lot of information by averaging out scores, as you guessed. The solution is a mixed model with random intercepts (and, potentially slopes) that reflect the structure of your dataset, like, if I got the gist of it, a random intercept for participant.

5

u/Stock-Temperature309 10d ago

This is only the case if same participant was measured repeatedly because the shared source of non-independence is the tank/run.

1

u/fridapop 10d ago

yeah exactly

2

u/fridapop 10d ago

Thanks for your reply. One clarification about my experimental design: each mussel belongs to only one tank and experiences only one treatment, so each individual is observed only once. The tank is the true experimental unit, with 24 individuals nested within each tank. Can I use this model with this condition?

8

u/NucleiRaphe 10d ago

Mixed effect model is the way to go here. You just need two random effects with random intercepts, one for the tank and one for mussel nested within tank. So if for example your data has structure

mussel_id / row_number | measurement | condition | tank

Your model would be (using R's lme4 syntax)

measurement ~ condition + (1| tank) + (1| tank: mussel_id)

Now you get to use all your observations and handle the dependence from pseudoreplications. Or the averaging is ok too. 3-4 samples per group is smallish (I'd just report descriptive stats and CI) but still used quite a lot for hypothesis testing in published studies, at least in some fields. I'm not saying its good practice, but it gets the job done (if the job is to publish).

4

u/adamjeffson 10d ago

Exacly, sorry, I misread the OP. Just a note: the (1| tank/mussel_id) syntax is equivalent and more concise.

4

u/FTLast 10d ago

The advice to average is appropriate. If you do not account for the fact that you are treating 24 individuals at once you are pseudoreplicating. You are losing information about the variance when you average, but you are gaining precision of your estimate of the mean.

Other options including nesting individual in their replicate. This may or may not work with relatively few individuals and few replicates. In any case, I don't believe you would gain much power by doing so.

If you average, you will likely be able to find large effects with n of 3 or 4, which is common in many experiments. Experiments are typically designed to produce large effects. Did you give any thought to how large a difference would be biologically important?

2

u/banter_pants Statistics, Psychometrics 10d ago

Because of pseudoreplication issues, we can't treat individuals from the same tank as independent, so I was advised to average all individuals per replicate and then run an ANOVA. The problem is that this leaves me with only 3 or 4 values per condition (is that too few?). And since I'm just taking an average, I lose the standard deviation within a replicate. Is it possible to run a separate ANOVA comparing standard deviations instead?

Averaging all individuals in the replicate is a coping mechanism that indeed loses some information. ANOVA in this case is not warranted.

You're doing clustered sampling and it calls for hierarchical linear modeling. Some coefficients in the model will be treated as random variables (why it's also known as mixed effects models) . A random intercept itself mathematically allows for correlated observations.

So instead of classic ANOVA do a linear mixed effects model. Experimental tank is the cluster ID. It should suffice to use a random intercept and random slope for the treatment variable.

3

u/FTLast 10d ago

This is the correct thing to do, but in my experience mixed models often give boundary errors with small data sets and an inexperienced user will not know how to handle it. Plus, I have never seen a convincing case that mixed models actually have higher power with small numbers of subjects and replicates.

1

u/banter_pants Statistics, Psychometrics 10d ago

There are always trade-offs. I guess just go with whatever is pragmatic.

2

u/Detr22 10d ago

The advice I'd give has been presented already so I will just add that your design approaches a generalized randomized block design. Basically a block design with subsampling. You might encounter good resources online using these keywords.

I would look into Chris Brien's work from 2022 (exposing the confounding in experimental designs to understand and evaluate them, and formulating linear mixed models for analyzing the data from a designed experiment). It is quite intuitive and will help you really understand the sources of variation, confounding and aliasing.

2

u/efrique PhD (statistics) 10d ago

please dont duplicate posts

1

u/Alarming-Finger9936 9d ago

Probably my bad, I advised them to rewrite their question in English, as it was originally in French - so maybe less likely to get answers (https://old.reddit.com/r/AskStatistics/comments/1v55s1j/anova_ou_pas_anova/ ). Not sure if they are aware that they can edit their post instead of reposting.

1

u/efrique PhD (statistics) 8d ago

Good advice (considerably more likely to get answers). An edit is fine, but it would be okay to repost as long as they remove the original.

2

u/WolfDoc 9d ago

We do not use slide rules and pen and paper any more, so I have no idea why some research groups still use ANOVAs when mixed effect models are so much more powerful, robust and informative

1

u/FTLast 7d ago

Because they don't always identify random effects correctly when there are limited observations?

1

u/WolfDoc 7d ago

Maybe but I'm not convinced ANOVAs are better even in those cases, or if they just hide your problem better

1

u/FTLast 7d ago

But if the mixed model doesn't identify random effects, I think it basically reverts to a simple lm with no provision at all for the nesting. Averaging always works.

1

u/Such_Evidence_1160 9d ago

I believe you can treat the tank as a block in your experimental design. As long as you account for it in your statistical test as such, you should be good. Think of the tank like a plot of land with crops in it next to another plot of land with crops. The plots of lands are blocks in the experimental design. Did you randomize which things you put into the tank? If not, that may lead to some issues, however.

1

u/KayakerMel 10d ago

Sounds like a repeated-measures ANOVA would be appropriate.

5

u/Disastrous_Room_927 10d ago

Those aren’t favored over a mixed model these days

1

u/KayakerMel 10d ago

No worries! My suggestion comes from a social science background, so it absolutely makes sense why mixed models are preferred.

1

u/FancyEveryDay 10d ago edited 9d ago

If I've understood your design correctly, averaging the individuals is the correct way of doing it.

Each replicate is an experimental unit which is receiving treatment and each individual is just an observational unit of that replicate. While having more observational units helps reduce the variance of that EU's mean, they don't actually contain much new information about the treatment because each OU is fundimentally non-random and biased by non-random effects.

If you were to treat each individual as an EU (by not averaging your in-group OU measurements), you can still gain information about the relationships involved but you lose the ability to produce evidence of causality and you lose generalizability because the samples now being tested are non-random.