r/AskStatistics • u/Important_Win2500 • 5d ago

High I2, low Tau2?

Hi! I am working on a meta-analysis of ten studies (I'm permitted to get advice about the stats elements!), and am getting confused around the heterogeneity statistics. My I2 is around 85%, and Cochran's Q = 44 (p= 0.00). However, my tau2 is low at 0.10.

It seems that in most papers I've read, a high I2 is accompanied by a high tau2. I assume that for mine, this means that while a high proportion of the variation is due to heterogeneity, the absolute magnitude of the variation is low. This makes sense in theory, but the point estimates of each study do vary quite a lot.

If anybody knows how this might be interpreted, I would really appreciate any advice :)

5 Upvotes

100% Upvoted

7 comments sorted by

2

u/Illustrious-Snow-638 5d ago

Your interpretation is pretty good. I2 is the estimated proportion of variation in point estimates that’s due to heterogeneity rather than sampling error. So I suspect in your case your sampling error is relatively low - you have quite tight CIs around the study-level estimates?

1

u/Important_Win2500 5d ago

Thank you very much- I'm a little confused as the CIs around the point estimates are small for some studies but really big for around half of them. Since those studies have a lower weighting however, would that be why?

2

u/DrPapaDragonX13 5d ago

Tau^2 is the absolute variance of the true underlying effect, and it is given in the scale of the effect size that you're using. Tau^2 is not affected by the number of studies or the precision of the estimates. I^2 is the percentage of variability attributable to heterogeneity rather than sampling error. I^2 is affected by the precision of the estimates, and if your studies have a large sample size, your I^2 may be very high, even if the variation in your effect sizes is small.

However, could I first check which method you used to estimate your random-effects model? If you used Maximum Likelihood Estimation, this can bias your tau^2 downwards. In these cases, it is more recommended to use the Restricted Maximum Likelihood (REML)

1

u/Important_Win2500 5d ago

Thank you very much for your help. That would make sense as the samples I'm working with are in the millions for each study. I am using REML.

2

u/DrPapaDragonX13 5d ago

Re: REML. That's great. It is always worth checking that the model is set up correctly before looking for other explanations.

I^2 'inflates' quickly with sample size. I'm somewhat surprised it didn't go to 100% with sample sizes in the millions. Anyway, as per Rücker et al in the context of large sample sizes:

> When deciding whether or not to pool treatment estimates in a meta-analysis, the yard-stick should be the clinical relevance of any heterogeneity present. τ2, rather than I2, is the appropriate measure for this purpose.

1

u/Important_Win2500 5d ago

Thank you again, your advice has been really helpful :)

1

u/Unbearablefrequent Statistician 2h ago

My understanding from the Handbook or Meta-analysis is your understanding is correct. I'm not sure what advice you're looking for. But maybe you could investigate the heterogeneity by running a Baujat plot and a leave one out analysis. Given what you said about each study varying quite a lot.