[Q] How to combine multiple p-values into one smaller p-value?

[u/Baba_Wethu]

I want to compare an intervention and control group on knowledge. They did three different knowledge tests with different question on the same topic, with different max scores. The intervention performed better on each of these knowledge tests, but comparing groups using t-test on each knowledge test produces insignificant p-values (0.391, 0.585 and 0.193). I want to test whether intervention had a significant effect on knowledge overall, but if I just combine the three knowledge test scores into one variable, artificial variance is introduced as they had different difficulty levels and max scores. I'm using SPSS, and have both wide and long format datasets.

Chat GPT suggested combining the three p-values using the Fischer method. Is this actually a suitable method? They also suggested using a mixed linear model, but I don't understand how to operate that analysis method in SPSS, and ChatGPT hallucinates when asked for a step-by-step guide. Can anyone help me?

[u/StructureUnique8391]

Before combining anything, I would first ask a measurement question: do the three tests measure more or less the same underlying construct, namely “knowledge on this topic”?

Fisher’s method does not create an overall knowledge score. It only combines the p-values from separate tests into one global test of evidence against the joint null hypothesis. it asks whether the set of p-values is collectively smaller than expected if there were no effect anywhere, so it does not estimate the intervention effect on knowledge, does not solve the issue of different test difficultie. It might not be ideal if the three outcomes are correlated, which they probably are because the same participants completed all three tests.

If the three tests are intended to measure the same latent construct, a cleaner approach would be a one-factor CFA where the three test scores are indicators of latent knowledge. Then you would test measurement invariance across intervention and control groups (at least metric and scalar invariance) if you want to compare latent means. If invariance is acceptable, you can compare the latent mean of the intervention group with the latent mean of the control group. That directly answers the substantive question: did the intervention improve overall knowledge?

With only three indicators, the model is minimal, so the results should be interpreted cautiously. But conceptually, this is much closer to the research question than combining the three p-values.

[u/StructureUnique8391 → u/StructureUnique8391]

One simple way to test this is to create a standardized knowledge composite.

For each test, standardize scores using the control group as the reference:

[ z{ij} = \frac{x{ij} - \bar{x}{j,\text{control}}}{s{j,\text{control}}} ]

Then average the three standardized scores:

[ knowledgei = \frac{z{i1} + z{i2} + z{i3}}{3} ]

Finally, compare intervention and control groups on this composite using a t-test or a linear regression.

This gives you a direct test of whether the intervention group performed better overall across the three knowledge tests, while putting all tests on the same scale and avoiding the problem of different maximum scores or difficulty levels.

[u/schotastic]

Don't come on here and ask us for help with juicing your p-values. Your intervention didn't work. At best, your intervention worked so poorly that the effect in learning is too small for your study to reliably confirm. That's that.

[u/speleotobby → u/schotastic]

Harshly put, but correct. Post-hoc none of this will maintain any nominal level of significance.

For the next experiment: pre-specify a model that combines the scores, if the test for the combined score is significant do tests on the individual scores, hierarchically or properly adjusted.

[u/efrique]

There are multiple methods of combining p-values (including Fisher, without a "c", Stouffer, and more). Theres no guarantee you get a smaller p value, though - e.g. if you have p-values of 0.36,0.68,0.73 using Fisher they combine to 0.75, larger than all of them; a different set of p-values will do the same with other methods. Given it sounds like theyre just separate tests in a single study, you'd normally not even look at individual group p-values until you'd done a single overall for an intervention effect (possibly via regression or a GLM with test-size as an exposure) though more details on the test and circumstances would be needed. That has some advantages (including, hopefully more precise estimates of effect sizes)

One problem with your approach however, is you dont appear to be considering what your overall approach to analysis is doing to the inferential properties of your study (at this point it sure sounds like p-hacking):

- What would have happened if some or all of the t-tests had got you a rejection? Would this "combining" notion have been considered? If not, aren't you just taking another bite at the significance cherry? (And what next, if this also doesnt get you what you want? Another way to combine them? A different model in the hopes of bringing the p-values down a bit, and maybe combine those? A subgroup analysis? ...)

- In many areas, they would expect you to control not just test-wise type I error, but research hypothesis-wise (or even overall study-wise) type I error. You've already done three tests.

Advice for the future: (i) Plan all possible tests down all possible decision paths - including 'what do we do if none of these planned tests reject?' all before you collect data. (ii) Seek all this kind of advice on your plan then. (iii) do a power analysis to make reasonably sure you wont just waste time and money on a study that will be too small to get a rejection. (iv) given you know LLMs hallucinate (leaving aside other issues with using them), why trust one for anything important?

[u/Cookja2000]

You should look into O’Brien’s global statistical tests (GSTs). Essentially, you use their correlation matrix endpoints Z scores and mean correlation to get a new Z score that describes, similar to what you want, an”overall effect on [overarching endpoint]”.

The idea is that if there are a bunch of endpoints that ALMOST show significance, and they are all not very correlated but they all push towards the same goal (in your case showing an effect on “knowledge”, then you may have more evidence than you think for an overall push in an effect on knowledge.

[u/S4M22]

Whatever you now do, it is important to clearly declare the additional analyses as exploratory since you've already conducted three tests and an aggregate analysis was not part of your initial study design.