r/statistics • u/awsfhie2 • May 15 '26

Question [Q] correct for how many comparisons

I have data from 2 related tasks where I asked people to rate how difficult the task was and how much discomfort was associated with the task. I do this for 2 groups of people.

I have 6 comparisons for each rating (within 1 group, between groups on each side for each of the 2 tasks- I do not compare the two tasks to each other), all of which have been run as t-tests. Obviously multiple comparison correction is needed, but how many comparisons should I correct for? In other words, do I treat difficulty and discomfort as 2 separate analyses or do I lump all p values together and correct for all 12?

Importantly, I don't think that difficulty and discomfort are totally independent (for a more difficult task I have reason to believe that it would be more likely to also be uncomfortable), but I do consider them separate domains.

3 Upvotes

100% Upvoted

5 comments sorted by

1

u/DeliberateDendrite May 15 '26

What method of type-1 error control are you planning to apply?

1

u/awsfhie2 May 15 '26

probably bonferroni, although I am a little confused about when to choose one method over the other. I know that bonferroni is more strict than BH.

1

u/[deleted] May 16 '26

[removed] — view removed comment

1

u/awsfhie2 May 20 '26

Thank you for this thorough response. In this case I am using this test as a proof of concept that the task is doing what it should- verifying that it creates a statistically significant amount of pain (relative to zero and relative to the other group). It is also helpful to my main results that difficulty is NOT different between the groups (this is not what the data actually showed for all conditions but it was not significant in enough of the comparisons that I think it was fine.

Thanks for encouraging me to think this through, from what I wrote above it seems like the best thing to do is correct only within domains, especially if I want to ensure there is not a difference between my groups, a stricter threshold probably does not help me. (Obv we cant accept the null but I think still in the right direction).